link.parallel
link.parallel is a database agnostic query system.
See documentation for more informations.
Installation
pip install link.parallel
Features
- parallel loops interface with IPython and multiprocessing drivers
- Map/Reduce middleware
Examples
Create your map/reduce functions:
from b3j0f.task import register_task
@register_task('mymap')
def mymap(mapper, item):
if item['i'] < 5:
mapper.emit('i_lt_5', item)
elif item['i'] > 5:
mapper.emit('i_gt_5', item)
else:
mapper.emit('i_eq_5', item)
@register_task('myreduce')
def myreduce(key, values):
return (key, len(values))Get input data and pass it to the middleware:
from link.middleware.core import Middleware
mapreduce = Middleware.get_middleware_by_uri(
'mapreduce+ipython:///test/classify?mapcb=mymap&reducecb=myreduce'
)
items = # load items
result = dict(mapreduce(items))
print(result)Donating
