Controlled multi-threading

thread, multi, processing
pip install rethreader==1.1.3



Intro & Examples

Rethreader is born as a thread pool for a function:

rt = Rethreader(sum)
rt.start()              # Start the thread pool
rt.add([1, 2, 3])       # Add sum([1, 2, 3]) to the queue
rt.add(range(10))       # Add sum(range(10)) to the queue
rt.quit()               # Quit the thread pool
rt.results              # Return the results
>> [6, 45]

But it could be used as a generic thread pool:

rt = Rethreader()
rt.add(sum, [1, 2, 3])                      # Add sum([1, 2, 3]) to the queue
rt.add(sum, range(10))                      # Add sum(range(10)) to the queue
rt.add(sorted, [3, 9, 6], reverse=True)     # Add sorted([3, 9, 6], reverse=True) to the queue
>> [6, 45, [9, 6, 3]]

It also supports the "with" statement:

with Rethreader(sum) as rt:         # Implicit rt.start()
    rt.add([1, 2, 3])
                                    # Implict rt.quit() at exit
>> [6, 45] 

It can aslo recieve the list of the different arguments at the initialization:

rt = Rethreader(sum, [[1, 2, 3], range(10)])
rt.add([2, 4, 8])
rt.add([8, 9, 10])
>> [6, 45, 14, 27] 

If all the queue is passed before the thread pool starts, the rethreader can simply be run:

list_of_args = [
    [1, 2, 3], 
    [2, 4, 8]

rt = Rethreader(sum, list_of_args)                                        # rt.start() & rt.quit()
>> [6, 45, 14] 

Most of the class functions return the rethread itself, so they can be chained:

Rethreader(sum, [[1, 2, 3], range(10)]).add([2, 4, 8]).run().results
>> [6, 45, 14] 

It has a lot of keyword arguments to be personalized with:

from urllib.request import urlretrieve

with Rethreader(urlretrieve, max_threads=4, clock_delay=2, save_results=False) as rt:
    for url, file in [...]:
        rt.add(url, file)

Note that it is much slower than mapping for fast offline functions:

squares = [x*x for x in range(999)]                             # ~0.5 milliseconds
squares = list(map(lambda x: x*x, range(999)))                  # ~1 millisecond
squares = Rethreader(lambda x: x*x, range(999)).run().results   # ~1 second
squares = Rethreader(lambda x: x*x, range(999), 
           clock_delay=0).run().results                         # ~120 milliseconds

However, it's more userful with internet requests

# Headlines from newspapers example
from requests_html import HTMLSession

session = HTMLSession()

def get_headlines(url):
    request = session.get(url)
    h3_tags = request.html.find('h3')
    h3_texts = [h3.text for h3 in h3_tags]
    return h3_texts
news = ['',     '',
        '', '',
        '',         '']

headlines = [get_headlines(url) for url in news]           # ~4.5 seconds
headlines = Rethreader(get_headlines, news).run().results  # ~0.75 seconds



Every parameter is optional at the initialization.

Rethreader(target, queue, max_threads, clock_delay, auto_quit, daemon)

  • target (function): the target of the treads

  • queue (iterable): list of the arguments of the threads

  • max_threads (integer): the maximum number of running threads (default: 16)

  • clock_delay (float): the speed of the clock (default: 0.01)

  • auto_quit: whether the rethreader stops after completing the queue (default: True if there's a queue, else False)

  • save_results: whether the rethreader has to save the returns of the functions (default: True)

  • daemon: whether the threads are daemonic (default: None, handled by the threading library)

Compatible with the "with" statement


  • finished: (int) number of finished threads

  • in_queue: (int) number of threads in the queue

  • remaining: (int) number of remaining threads (active and in queue)

  • results: (list) list of the returns of the functions (for not finished threads, inserts a None)


If not specified, the function returns the rethreader.

Basic functions:

  • add(args): appends to the queue

  • extend(iterable: args): extends the queue

  • remove(args): removes from the queue or the running threads

  • auto_quit(bool): whether the rethreader stops after finishing the queue (True if not specified)

  • run: runs the rethreader (if auto_quit is not enabled, this will run until killed)

  • start: starts the rethreader in a thread

  • quit: finish the queue, then stops the rethreader

  • kill: empties the queue and stops immediatly the rethreader

  • is_alive: (bool) whether the rethreader is running

  • is_empty: (bool) whether there are not remaining threads

Advanced functions:

  • postpone(delay: float, args): removes args, then appends to the queue after the delay

  • insert(args): insterts at the start of the queue

  • prioritize(iterable: args): extends the start the queue

  • find(args): (Key/KeyThread) returns the thread or its info

  • task(args): appends to the queue and waits until returned

  • execute(args): inserts into the start of the queue and waits until returned

Argument input

The list of arguments can be passed both initially and after the declaration of the rethreader

rt = Rethreader(print, range(10))
with Rethreader(print) as rt:
with Rethreader(print) as rt:
    for i in range(10):
rt = Rethreader(print)
# rt.start()
for i in range(10):
# rt.quit()

Keywords arguments could be passed also through a dictionary.

with Rethreader(print) as rt:
    for i in range(10):
        rt.add(i, end='')
with Rethreader(print) as rt:
    for i in range(10):
        rt.add(i, {"end": ''})