Uses of Class
net.sf.okapi.steps.tokenization.Token

Packages that use Token

Package	Description
net.sf.okapi.steps.tokenization

Uses of Token in net.sf.okapi.steps.tokenization

Methods in net.sf.okapi.steps.tokenization that return Token

Modifier and Type	Method	Description
Token	ITokenizer.next()
Token	RbbiTokenizer.next()

Methods in net.sf.okapi.steps.tokenization that return types with arguments of type Token

Modifier and Type	Method	Description
List<Token>	TokenizationStep.apostrophe(Token token, LocaleId locale)	Break French and Italian words with apostrophe into three tokens WORD, PUNCTUATION, WORD
Collection<? extends Token>	TokenizationStep.postProcess(Token t, LocaleId language)	Various rules to make corrections to RbbiTokenizer

Methods in net.sf.okapi.steps.tokenization with parameters of type Token

Modifier and Type	Method	Description
List<Token>	TokenizationStep.apostrophe(Token token, LocaleId locale)	Break French and Italian words with apostrophe into three tokens WORD, PUNCTUATION, WORD
Collection<? extends Token>	TokenizationStep.postProcess(Token t, LocaleId language)	Various rules to make corrections to RbbiTokenizer