ThreadedWorkQueue Step

• Steps List and Overview
• Articles on Pipelines and Steps in the Knowledge Base
• All Rainbow Utilities

Overview

Warning: EXPERIMENTAL. Use with caution.

This step creates a wrapper around the main step and processes text units concurrently. Example of use:

pipeline.addStep(new TextUnitWorkQueueStep(new LongRunningStep(), 4));

• LongRunningStep is the step that will be processed concurrently and is wrapped by ThreadedWorkQueueStep
• 4 is the number of threads or work-queues to create. In this case 4 text units will be processed concurrently. The optimal number of threads is dependent on the wrapped step.

Takes: Filter events. Sends: Filter events. (TextUnits and DocumentParts are buffered and sent as MultiEvents)

• Does the same processing as the main step. Processes multiple text units concurrently.

See the Okapi Developer Wiki for more details.

Parameters

This step has no parameters.

Limitations

• Will only process text units concurrently.
• Steps must be essentially stateless: TextUnit in, TextUnit out
• DocumentParts are passed through without processing of any kind. If DocumentPart processing is needed then a new implementation must be created

  or override the startConcurrentProcesing() method.