Package net.sf.okapi.steps.tokenization
-
Interface Summary Interface Description ITokenizer -
Class Summary Class Description Parameters Tokenization step parametersRbbiTokenizer Token TokenType generated by theRbbiTokenizerTokenConfigs TokenType DefinitionsTokenConfigs.TokenConfig TokenizationStep Tokenizer Tokens List ofTokenand configuration for each token type.TokensAnnotation Holds a list ofToken -
Enum Summary Enum Description ITokenizer.TokenType