Quick Start

A simple setup looks like:

from littleworkers import Pool

# Define your commands.
commands = [
    'ls -al',
    'cd /tmp && mkdir foo',
    'echo "Hello There."',
    'sleep 2 && echo "Done."'

# Setup a pool. Since I have two cores, I'll use two workers.
lil = Pool(workers=2)

# Run!


littleworkers shines when you just want to parallelize something without a lot of fuss & when you care more about the data/commands to be run.

  • Tiny source
  • Easy to queue a set of actions
  • Works with any runnable commands
  • Uses processes
  • Non-blocking

Seriously, it’s not a replacement for threading or multiprocessing if your application needs to share a ton of data with the children.


littleworkers was designed to be extended, so most customizations should be possible without forking the code. Instead, you should simple subclass Pool & extend/override the method. You can find the details of each method in the API docs.

Example Customizations

You want the stdout back:

import subprocess
from littleworkers import Pool

class MyPool(Pool):
    def __init__(self, *args, **kwargs):
        super(MyPool, self).__init__(*args, **kwargs)
        self.collected_output = []

    def create_process(self, command):
        logging.debug("Starting process to handle command '%s'." % command)
        return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

    def remove_from_pool(self, pid):
        return super(MyPool, self).remove_from_pool(pid)

You want to use a Queue instead of the default list:

from Queue import Queue, Empty
from littleworkers import Pool

class QueuePool(Pool):
    def __init__(self, *args, **kwargs):
        super(QueuePool, self).__init__(*args, **kwargs)
        self.commands = Queue()

    def prepare_commands(self, commands):
        for command in commands:

    def command_count(self):
        return self.commands.qsize()

    def next_command(self):
            return self.commands.get()
        except Empty:
            return None

You want to setup a callback:

from littleworkers import Pool

codes = []

def track(proc):
    codes.append("%s returned status %s" % (proc.pid, proc.returncode))

commands = [
    'sleep 1',
    'busted_command --here',
    'sleep 1',
lil.run(commands, callback=track)