Classes to run producers (iterators) and consumers (coroutines) in a background thread/process.

concurrency parallelism iterator iterable pipeline
pip install concurrent-iterator==0.2.6



Run Status Coverage Badge Supported Python versions License


Classes to run producers (iterators) and consumers (coroutines) in a background thread/process.

There are many libraries to create pipelines with stages running in separate processes, a nice one is parallelpipe, but this library does something different. It will lift the entire pipeline up to the point of the Producer into a separate process or thread. It's a more coarse library but easier to integrate since things keep looking as normal generators.


There are currently 3 implementations:

  • dummy.Producer: non-concurrent implementation
  • thread.Producer: uses a background thread to run the generator
  • process.Producer: uses a background process to run the generator

dummy.Producer is useless in practice.

thread.Producer is useful for IO bound generators.

process.Producer is useful for CPU or IO bound generators. It has the complications of dealing with processes (different memory spaces, logging, etc). For logging, module multiprocessing-logging can be used.


Basic example:

from concurrent_iterator.thread import Producer


items = Producer(slow_generator, maxsize=5)

for item in items:
    [Do some time consuming task]

In the previous example, while doing some time consuming task, the slow_generator will continue running in a background thread and will pre-calculate up to 5 values.