Bulk Runner
Async processing of big arrays in bulks!
- split a big array into bulks.
- for each bulk, process each of its items.
- only when bulk processing is done, continue to the next bulk.
- return a result array of all bulk results concatenated.
Install
npm install async-bulk-map
Example
import { asyncBulkMap } from 'async-bulk-map';
const bigArray = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const bulkSize = 2;
const res = await asyncBulkMap(bigArray, bulkSize, {
beforeAll: originArray => {
console.log(`starting map for ${originArray.length} items...`);
},
beforeBulk: (bulk, bulkIndex) => {
console.log(`editing each bulk #${bulkIndex} before process`);
return Promise.all(bulk.map(item => Promise.resolve(item * 2)));
},
map: item => {
console.log(`processing a single item in bulk`);
return item + 5;
},
afterBulk: bulkRes => {
console.log(`done processing bulk`);
return bulkRes.map(item => (item - 5) / 2);
},
afterAll: resultArray => {
return resultArray.filter(item => item > 0);
},
});
// res: [1,2,3,4,5,6,7,8,9,10]Usage
asyncBulkMap<T, R>(bigArray: T[], bulkSize: number, actions: IBulkActions<T, R>): Promise<R[]>;Bulk Actions
Notice that:
- All events are optional except the actual
mapof the items. - Each events can return either:
- no value, so the array/bulk won't be effected
- a new array or a
Promisefor it, that will replace the array/bulk
export type MaybePromiseOf<T> = Maybe<T | Promise<T>>;
export interface IBulkActions<T, R> {
beforeAll?: (arr: T[]) => MaybePromiseOf<T[]>; // for modifying the source array before starting
beforeBulk?: (bulk: T[], bulkIndex: number) => MaybePromiseOf<T[]>; // before processing of each bulk.
map: (item: T) => MaybePromiseOf<R>; // processing of each item
afterBulk?: (bulkRes: R[]) => MaybePromiseOf<R[]>; // after processing all items in the bulk and before continuing to the next one
afterAll?: (res: R[]) => MaybePromiseOf<R[]>; // after processing all bulks
}Utilities
- logging before starting a bulk
- delay before starting the next bulk
import { asyncBulkMap, createPipe, logBulk, createDelay } from 'async-bulk-map';
const bigArray = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
const bulkSize = 2;
const res = await asyncBulkMap(bigArray, bulkSize, {
beforeBulk: createPipe(logBulk).pipe((bulk, bulkIndex) => {
// it will: console.log(`bulk index: ${bulkIndex} (${bulk.length} entries)`);
// then arrive for processing here
}),
map: item => item,
afterBulk: createPipe(createDelay(2000)).pipe(bulkRes => {
// it will create a 2sec delay after processing the bulk
// then arrive for processing here
}),
});