ThreadedWorkQueue Step

From Okapi Framework
Jump to: navigation, search

Overview

Warning: EXPERIMENTAL. Use with caution.

This step creates a wrapper around the main step and processes text units concurrently. Example of use:

pipeline.addStep(new TextUnitWorkQueueStep(new LongRunningStep(), 4));

  • LongRunningStep is the step that will be processed concurrently and is wrapped by ThreadedWorkQueueStep
  • 4 is the number of threads or work-queues to create. In this case 4 text units will be processed concurrently. The optimal number of threads is dependent on the wrapped step.

Takes: Filter events. Sends: Filter events. (TextUnits and DocumentParts are buffered and sent as MultiEvents)

  • Does the same processing as the main step. Processes multiple text units concurrently.

See the Okapi Developer Wiki for more details.

Parameters

This step has no parameters.

Limitations

  • Will only process text units concurrently.
  • Steps must be essentially stateless: TextUnit in, TextUnit out
  • DocumentParts are passed through without processing of any kind. If DocumentPart processing is needed then a new implementation must be created
  or override the startConcurrentProcesing() method.