Tokenize Monster
Yet another powerful tokenizer in JS.
Installation
npm install --save tokenize-monsterGetting Started
const tokenizer = require('tokenize-monster');
const tokens = tokenizer.matchAndMap(
'i\'d like to <drink> some <milk>',
/<[^><]+>/,
text => [[text]],
text => [text]
);
console.log(tokens);
// => [ 'i\'d like to ', [ '<drink>' ], ' some ', [ '<milk>' ], '' ]
const nTokens = tokenizer.matchAndMap(
'i\'d like to <drink> some <milk>',
/<[^><]+>/,
text => [text],
text => []
);
console.log(nTokens);
// => [ '<drink>', '<milk>' ]Example
Documentation
Our docs are available at Tokenize Monster Wiki.
Changelog
Visit Github Releases page for more information.
