Proc-tored-Pool

managed work pool with Proc::tored and Parallel::ForkManager


License
Artistic-1.0-Perl

Contributors

Jeff Ober


See all contributors


Documentation

NAME

Proc::tored::Pool - managed work pool with Proc::tored and Parallel::ForkManager

VERSION

version 0.07

SYNOPSIS

  use Proc::tored::Pool;

  # Create a worker pool service
  my $pool = pool 'thing-doer', in '/var/run', capacity 10,
    on success, call {
      my ($me, $id, @results) = @_;
      print "thing $id complete: @results";
    },
    on failure, call {
      my ($me, $id, $message) = @_;
      warn "thing $id failed: $message";
    };

  # Do things with the pool
  run {
    my ($thing_id, $thing) = get_next_thing();
    process { do_things($thing) } $pool, $thing_id;
  } $pool;

  # Control the pool as a Proc::tored service
  pause $pool;
  resume $pool;
  stop $pool;
  zap $pool, 15 or die "timed out after 15 seconds waiting for pool to stop";
  start $pool;

DESCRIPTION

Provides a simple and fast interfact to build and manage a pool of forked worker processes. The process is controlled using a pidfile and touch file.

EXPORTED SUBROUTINES

As a Proc::tored::Pool is a Proc::tored service, it by default exports the same functions as "EXPORTED SUBROUTINES" in Proc::tored.

In addition, the following subroutines are exported by default.

pool

Creates the pool (an instance of Proc::tored::Pool::Manager). Requires a $name as its first argument.

  my $pool = pool 'the-proletariat', ...;

capacity

Sets the max number of forked worker processes to be permitted at any given time.

  my $pool = pool 'the-proletariat', capacity 16, ...;

on

Builds an event callback with one of "assignment", "success", or "failure".

  my $pool = pool 'the-proletariat', capacity 16,
    on success, call { ... };

call

Defines the code to be called by an event callback. See "on".

pending

Returns the number of tasks that have been assigned to worker processes but have not yet completed.

process

Sends a task (a CODE ref) to the pool, optionally specifying a task id to identify the results in callbacks. The return value of the supplied code ref is passed as is to the "success" callback (if supplied).

  process { seize_the_means_of_production() } $pool;
  process { seize_the_means_of_production() } $pool, $task_id;

sync

For situations in which a task or tasks must be completed before program execution can continue, sync may be used to block until all pending tasks have completed. After calling sync, there will be no pending tasks and all callbacks for previously submitted tasks will have been called.

  process { seize_the_means_of_production() } $pool;
  sync $pool;

EVENTS

assignment

Triggered immediately after a task is assigned to a worker process. Receives the pool object and the task id (if provided when calling "pool").

  my $pool = pool 'thing-doer', ...,
    on assignment, call {
      my ($self, $task_id) = @_;
      $assigned{$task_id} = 1;
    };

  process { do_things() } $pool, $task_id;

success

Triggered after the completion of a task. Receives the pool object, task id (if provided when calling "pool"), and the return value of the code block.

  my $pool = pool 'thing-doer', ...,
    on success, call {
      my ($self, $task_id, @results) = @_;
      ...
    };

  process { do_things() } $pool, $task_id;

If "process" performs an exec, no data is returned from the worker process (because exec never returns). In this case, if the process had a zero exit status, the success callback is triggered with the results value of "zero but true". A non-zero exit status is handled by failure as in the general case.

failure

Triggered if the code block dies or the forked worker exits abnormally. Recieves the pool object, task id (if provided when calling "pool"), and the error message generated by the code ref.

  my $pool = pool 'thing-doer', ...,
    on failure, call {
      my ($self, $task_id, $error) = @_;
      warn "Error executing task $task_id: $error";
    };

  process { do_things() } $pool, $task_id;

BUGS AND LIMITATIONS

Proc::tored

Warnings and limitations for Proc::tored also apply to this module, especially the notes regarding service control from within a running service which also apply to code executing in forked worker processes. See "BUGS AND LIMITATIONS" in Proc::tored for details.

Parallel::ForkManager

Warnings and limitations for Parallel::ForkManager also apply to this module, including the injunction against using two pools simultaneously from the same process. See "BUGS AND LIMITATIONS" in Parallel::ForkManager and "SECURITY" in Parallel::ForkManager for details.

SEE ALSO

Proc::tored
Parallel::ForkManager

AUTHOR

Jeff Ober <[email protected]>

COPYRIGHT AND LICENSE

This software is copyright (c) 2017 by Jeff Ober.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.