Filters

Filters are the components that convert input documents from their native file format into a common internal set of resources that all Okapi components use. The extracted content can be re-written into the original file format. When using the steps, the extraction is done by the Raw Document to Filter Events Step and the re-writing by the Filter Events to Raw Document Step.

Note: The Okapi Filters Plugin for OmegaT allows you to use some of the filters directly from OmegaT.

List of the Filters

The framework distribution comes with the following filters:

Archive Filter DTD Filter Doxygen Filter DXF Filter EPUB Filter HTML Filter HTML5-ITS Filter ICML Filter IDML Filter JSON Filter Markdown Filter Message Format Filter MIF Filter Moses Text Filter Multi-Parsers Filter OpenOffice Filter OpenXML (MS Office) Filter

PDF Filter Pensieve TM Filter PHP Content Filter Plain Text Filter PO Filter Properties Filter Rainbow Translation Kit Filter Regex Filter SDL Trados Package Filter Simplification Filter Table Filter TMX Filter Trados-Tagged RTF Filter

Transifex Filter TS Filter TTX Filter TXML Filter Wiki Filter WSXZ Package Filter Vignette Filter XLIFF Filter XLIFF-2 Filter XML Filter XML Stream Filter YAML Filter

Supported File Formats

The following is a list of some of the file formats supported by the distribution through pre-defined configurations:

Format	Extensions	Pre-Defined Configuration	Filter	Notes
Android Strings	.xml	okf_xml-AndroidStrings	XML Filter
Apple Stringsdict	.stringsdict	okf_xml-AppleStringsdict	XML Filter
Archive	.zip	okf_archive	Archive Filter	Meta filter that processes zip files with various formats as one file.
Auto Xliff	.xlf, .xliff	okf_autoxliff	Auto Xliff Filter	Detects the version of an XLIFF file and then hands parsing off to the appropriate filter
AutoCAD DXF	.dxf	okf_dxf	DXF Filter	Only supports textual DXF, not binary DXF
CSV (Comma-separated values files)	.csv, .txt	okf_table_csv	Table Filter
CSV (Multiple complex sub-formats)	.csv	okf_multiparsers	Multi-Parsers Filter
DITA	.dita, .ditamap, .xml	okf_xmlstream-dita	XML Stream Filter
DocBook v5.0	.xml	okf_xml-docbook	XML Filter	Since Okapi 1.42. <footnote> is not handled properly.
DokuWiki pages	.txt	okf_wiki	Wiki Filter
Doxygen-commented files	.c, .h, cpp	okf_doxygen	Doxygen Filter
DTD	.dtd	okf_dtd	DTD Filter
EPUB	.epub	okf_epub	EPUB Filter
Fixed-Width Columns Table	.txt	okf_table_fwc	Table Filter
Idiom WorldServer XLIFF	.xlf	okf_xliff-iws	XLIFF Filter
InCopy ICML	.wcml	okf_icml	ICML Filter
InDesign IDML	.idml	okf_idml	IDML Filter
iOS/Mac Strings	.strings	okf_regex-macStrings	Regex Filter
Java Properties	.properties	okf_properties	Properties Filter
Java Properties (Output not escaped)	.properties	okf_properties-outputNotEscaped	Properties Filter
Java XML Properties	.xml	okf_xml-JavaProperties	XML Filter
Java XML Properties (HTML strings)	.xml	okf_xmlstream-JavaPropertiesHTML	XML Stream Filter
JSON	.json	okf_json	JSON Filter
Haiku CatKeys	.catkeys	okf_table_catkeys	Table Filter
HTML (any)	.html, .htm	okf_html	HTML Filter
HTML (Well-formed, and XHTML)	.html, .htm	okf_html-wellFormed	HTML Filter
HTML5 (and XHTML5)	.html, .htm	okf_itshtml5	HTML5-ITS Filter
Markdown	.md	okf_markdown	Markdown Filter
Microsoft Excel 2007/2010	.xlsx, .xlsm, .xltx, .xltm	okf_openxml	OpenXML Filter
Microsoft PowerPoint 2007/2010	.pptx, .pptm, .potx, .potm, .ppsx, .ppsm	okf_openxml	OpenXML Filter
Microsoft Visio	.vsdx, .vsdm	okf_openxml	OpenXML Filter
Microsoft Word 2007/2010	.docx, .docm, .dotx, .dotm	okf_openxml	OpenXML Filter
MIF	.mif	okf_mif	MIF Filter
Moses Text	.txt	okf_mosestext	Moses Text Filter
OpenOffice.org Calc	.ods, .ots	okf_odf	OpenOffice Filter
OpenOffice.org Draw	.odg, .otg	okf_odf	OpenOffice Filter
OpenOffice.org Impress	.odp, .otp	okf_odf	OpenOffice Filter
OpenOffice.org Writer	.odt, .ott	okf_odf	OpenOffice Filter
PDF	.pdf	okf_pdf	PDF Filter
Pensieve TM	.pentm	okf_pensieve	Pensieve TM Filter
PHP Content	.php	okf_phpcontent	PHP Content Filter	Can be used as a subfilter only
Plain Text (Line = text unit)	.txt	okf_plaintext	Plain Text Filter
Plain Text (Paragraph = text unit)	.txt	okf_plaintext_paragraphs	Plain Text Filter
PO	.po	okf_po	PO Filter
PO (Monolingual style)	.po	okf_po-monolingual	PO Filter
Rainbow Translation Kit manifests	.rkm	okf_rainbowkit	Rainbow Translation Kit Filter	Used as a tkit reader only
Regex (Any text-based format)	.txt	okf_regex	Regex Filter
RDF (Mozilla RDF)	.rdf	okf_xml-MozillaRDF	XML Filter
RESX	.resx	okf_xml-resx	XML Filter
SDLPPX	.sdlppx	okf_sdlpackage	SDL Trados Package Filter
SDLRPX	.sdlrpx	okf_sdlpackage	SDL Trados Package Filter
SDLXLIFF	.sdlxlf	okf_xliff-sdl	XLIFF Filter
Skype Language Files	.lang	okf_properties-skypeLang	Properties Filter
SRT (Sub-Rip Text, sub-titles files)	.srt	okf_regex-srt	Regex Filter
Tab-Delimiter files	.tsv, .txt	okf_table_tsv	Table Filter
Tex files	.tex	okf_tex	TEX Filter
TMX	.tmx	okf_tmx	TMX Filter
Transifex project	.txp	okf_transifex	Transifex Filter
Trados-Tagged RTF	.rtf	okf_tradosrtf	Trados-Tagged RTF Filter
TS - Qt TS files	.ts	okf_ts	TS Filter
TTX - Trados TagEditor TTX files	.ttx	okf_ttx	TTX Filter
TXML - Wordfast Pro TXML files	.txml	okf_txml	TXML Filter
Vignette Export/Import Content	.xml	okf_vignette	Vignette Filter
WSXZ Package Filter	.wsxz	okf_wsxzpackage	WSXZ Package Filter
XHTML	.html, .htm	okf_html-wellFormed	HTML Filter
WIX (Windows Installer XML) localization files	.wix	okf_xml-WixLocalization	XML Filter
XLIFF v1.2	.xlf, .xliff	okf_xliff	XLIFF Filter
XLIFF v2	.xlf	okf_xliff2	XLIFF-2 Filter
XML (Generic, using ITS defaults)	.xml	okf_xml	XML Filter
XML (Generic, using stream reader)	.xml	okf_xmlstream	XML Stream Filter
YAML (Generic YAML filter)	.yml, .yaml	okf_yaml	YAML Filter
Message Format (ICU Message Format Filter)	Any container format that supports subfilters	okf_messageformat	Message Format Filter

Note that most filters allow you to create your own configurations to support more file formats.

Code Simplification Rules

There are two levels of code simplification: filter and step (the Inline Codes Simplifier Step and the Post-segmentation Inline Codes Removal Step). And there are different ways of configuring it:

Firstly, the extraction pipeline can contain just:

• Raw Document to Filter Events Step

At the moment, only IDML Filter, XML Filter and Simplification Filter support this. It should be noted that the last one performs like a wrapper for another filter.

Secondly, the extraction pipeline can look like that:

• Raw Document to Filter Events Step
• Inline Codes Simplifier Step

This is the only way for filters that do not support their own code simplification, and it should be used with care because the final merge may not always handle this correctly. The aforementioned IDML Filter and XML Filter can perform their own simplification, and the added Inline Codes Simplifier Step should not affect the events produced.

Thirdly, the extraction pipeline can consist of:

• Raw Document to Filter Events Step
• Segmentation Step
• Post-segmentation Inline Codes Removal Step

Here, the Post-segmentation Inline Codes Removal Step performs code simplification after segmentation rules are applied, and it may be useful for skipping extra codes between segments.

By default, the Inline Codes Simplifier Step and Post-segmentation Inline Codes Removal Step maximise the trimming and merging (aka simplification) of inline codes. This can be tuned via the following string parameters:

• removeLeadingTrailingCodes - true by default
• mergeCodes - true by default
• rules - empty by default

Only the Inline Codes Simplifier Step configuration can be overridden by the optional filter ones via the following parameters:

• moveLeadingAndTrailingCodesToSkeleton - maps to the removeLeadingTrailingCodes parameter
• mergeAdjacentCodes - maps to the mergeCodes parameter
• simplifierRules - maps to the rules parameter

The simplification rules allow the prevention of specific codes trimming or merging. When a rule matches a code, it means: do not simplify this code.

General Syntax

The rules parser ignores irrelevant whitespace. Rules can be separated by spaces, newlines or nothing. This makes it easier to accommodate various container formats and their whitespace normalization rules. When a rule applies, it means "do not simplify the matched code". Uppercase tokens are constants and predefined by the rule parser. Multiple rules are always OR'ed together.

For more details, see the JavaCC grammar: ../okapi/core/src/main/javacc/SimplifierRules.jj

Each rule starts with if and ends with ;.

if TYPE = "bold";

Multiple rules are always OR'ed together. In other words, if any rule matches a code, that code is not simplified.

Within one rule, expressions can be combined with and, or and parentheses.

if TYPE = "bold" or (DATA = "<br/>" and TAG_TYPE = STANDALONE);

The parser supports comments:

# This is a line comment
if TYPE = "bold";

/*
This is a block comment
*/
if TYPE = "italic";

Available Fields and Literals

The following string fields can be used with quoted string values:

Field	Meaning
DATA	The raw/native data of the inline code. For XML/HTML-like tags this is normally the complete raw code, for example <a>, </a> or <br/>. Depending on the filter and the configuration format, the literal value may also contain escaped markup, for example &lt;/a&gt;. DATA is not just the tag name.
OUTER_DATA	The complete outer data of the inline code, if available. This is mainly relevant for formats that store inline codes as markup themselves, such as XLIFF or TMX. If no separate outer data is available, this returns the same value as DATA.
ORIGINAL_ID	The original inline-code ID from the filtered/input format, if available. If no original ID is available, it behaves like an empty string.
TYPE	The abstract type of the inline code. This is often the tag name or a semantic type, for example "a", "bold", "italic", "underline" or "lb". Opening and closing codes that belong together normally have the same TYPE.

The following field is used with unquoted tag-type literals:

Field	Allowed values
TAG_TYPE	OPENING, CLOSING or STANDALONE

Important: TAG_TYPE is written with an underscore. TAGTYPE is invalid.

The tag-type literals are not strings and must not be quoted:

if TAG_TYPE = OPENING;
if TAG_TYPE = CLOSING;
if TAG_TYPE = STANDALONE;

This is invalid:

if TAG_TYPE = "CLOSING";

The following boolean flag literals can be used without an operator:

Literal	Meaning
ADDABLE	Matches codes for which code.isAdded() is true. Despite the rule name, this means that the code was added after extraction and was not found in the original source.
DELETABLE	Matches codes for which code.isDeleteable() is true. Such a code may be removed from the text, for example a formatting code, unlike a required placeholder such as %s.
CLONEABLE	Matches codes for which code.isCloneable() is true. Such a code may be duplicated in the text, for example a formatting code, unlike a placeholder that must occur exactly once.

Since a matching rule means do not simplify this code, a rule such as the following protects all codes that match any of these three flags from trimming and merging:

if ADDABLE or DELETABLE or CLONEABLE;

This rule is useful if the information expressed by these flags should be preserved and the code should not be merged with neighbouring codes or moved out of the text unit as a leading/trailing code.

Operators

String fields such as DATA, OUTER_DATA, ORIGINAL_ID and TYPE can be matched with quoted string values:

Operator	Meaning	Example
=	Exact string match	if TYPE = "bold";
!=	Negated exact string match	if TYPE != "bold";
~	Regular expression match	if DATA ~ ".*meta-ref.*";
!~	Negated regular expression match	if DATA !~ ".*meta-ref.*";

TAG_TYPE is matched against the unquoted tag-type literals:

if TAG_TYPE = OPENING;
if TAG_TYPE != CLOSING;

Boolean flag literals are used directly:

if DELETABLE;
if DELETABLE or CLONEABLE;

Rule Examples

If a code has any of these flags, then do not simplify it:

if DELETABLE or ADDABLE or CLONEABLE;

Match an opening a code by abstract type:

if TYPE = "a" and TAG_TYPE = OPENING;

Match a closing a code by abstract type:

if TYPE = "a" and TAG_TYPE = CLOSING;

Match a standalone br code by abstract type:

if TYPE = "br" and TAG_TYPE = STANDALONE;

Match by raw data. For XML/HTML-like tags, DATA usually contains the complete raw tag, not just the tag name:

if DATA = "<a>" and TAG_TYPE = OPENING;
if DATA = "</a>" and TAG_TYPE = CLOSING;
if DATA = "<br/>" and TAG_TYPE = STANDALONE;

Depending on the filter and the configuration format in which the rule is stored, XML/HTML characters may need to be escaped. For example, if the literal DATA value is </a>, use:

if DATA = "</a>" and TAG_TYPE = CLOSING;

Use TYPE when you want to match the abstract code type or tag name. Use DATA when you want to match the actual raw/native inline-code content.

Regular expression match:

if DATA ~ ".*meta-ref.*";

Regular expression match for opening a tags where the raw tag may contain attributes:

if DATA ~ "<a[ >].*" and TAG_TYPE = OPENING;

Negated regular expression match:

if DATA !~ ".*meta-ref.*";

Match on code type; in this case, do not simplify line break codes:

if TYPE = "lb";

Do not simplify rich text types:

if TYPE = "bold" or TYPE = "italic" or TYPE = "underline";

Expressions can be recursive and support parentheses:

if TYPE = "bold" or (DATA = "bar" or (DATA = "foo" and TYPE = "underline"));

Match by original ID, if the filter provides one:

if ORIGINAL_ID = "1";
if ORIGINAL_ID ~ "b[0-9]+";

Match by outer data, for formats that store inline codes as markup themselves, such as XLIFF or TMX:

if OUTER_DATA ~ ".*<ph.*";

Match flag literals individually:

# Do not simplify codes added after extraction
if ADDABLE;

# Do not simplify codes that may be removed
if DELETABLE;

# Do not simplify codes that may be duplicated
if CLONEABLE;

Or combine them in one rule:

if ADDABLE or DELETABLE or CLONEABLE;

Filter Config Examples

Examples of using simplifier rules within the filter configuration formats used by Okapi.

YAML

simplifierRules: |
  if ADDABLE or DELETABLE or CLONEABLE;
  if TYPE = "a" and TAG_TYPE = OPENING;
  if TYPE = "a" and TAG_TYPE = CLOSING;
  if DATA = "<br/>" or DATA = "<font>" or DATA = "</font>" or DATA = "</a>";
  if DATA ~ "\\<font.+" or DATA ~ "\\<img.+" or DATA ~ "\\<a.+";

ITS

ITS rules are XML. Therefore, XML entity escaping happens before the simplifier rule parser sees the rules.

• To pass a literal < or > character to the rule parser from an ITS file, write < or > in the ITS file.
• To compare against a DATA value that itself contains the literal characters < and >, write < and > in the ITS file.
• The same escaping rules apply to OUTER_DATA. For example, to let the rule parser see a regular expression containing <ph, write <ph in the ITS file.

<?xml version="1.0" encoding="UTF-8"?>
<its:rules
  xmlns:its="http://www.w3.org/2005/11/its"
  version="1.0"
  xmlns:itsx="http://www.w3.org/2008/12/its-extensions"
  xmlns:okp="okapi-framework:xmlfilter-options">

  <its:translateRule selector="//*" translate="yes"/>
  <its:withinTextRule selector="//codeph" withinText="yes"/>
  <its:withinTextRule selector="//ph" withinText="yes"/>

  <okp:simplifierRules
    moveLeadingAndTrailingCodesToSkeleton="yes"
    mergeAdjacentCodes="yes">
    if ADDABLE or DELETABLE or CLONEABLE;
    if TYPE = "a" and TAG_TYPE = OPENING;
    if TYPE = "a" and TAG_TYPE = CLOSING;

    # DATA example: matches a code whose DATA value literally contains &lt;/a&gt;.
    # Because this rule is inside ITS/XML, the ampersands are escaped as &amp;.
    if DATA = "&amp;lt;/a&amp;gt;" and TAG_TYPE = CLOSING;

    # OUTER_DATA example: matches outer inline-code markup such as <ph id="1">code</ph>.
    # Because this rule is inside ITS/XML, the < and > characters in the regex are escaped.
    if OUTER_DATA ~ ".*&lt;ph[^&gt;]*&gt;.*&lt;/ph&gt;.*";
  </okp:simplifierRules>

</its:rules>

The DATA example above is for the case where the value stored in DATA is the literal string </a>. If the value stored in DATA is the raw string </a> instead, then the ITS file only needs one XML escaping level:

<okp:simplifierRules
  moveLeadingAndTrailingCodesToSkeleton="yes"
  mergeAdjacentCodes="yes">
  if DATA = "&lt;/a&gt;" and TAG_TYPE = CLOSING;
</okp:simplifierRules>

FPRM / Parameters

#v1
extractNotes.b=true
simplifierRules=if ADDABLE or DELETABLE or CLONEABLE; if TYPE = "a" and TAG_TYPE = OPENING; if TYPE = "a" and TAG_TYPE = CLOSING;

Font Mapping

The font mapping can be considered as a filter's ability to automatically substitute font information in the target document on the fly, according to a provided configuration - this helps to reduce the amount of reformatting and post-translation DTP. It is supported by IDML and OpenXML (DOCX, PPTX and XLSX documents) filters at the moment.

The following font mapping configuration options are available:

• The source locale regular expression pattern: .*, en.*, en-UK, etc. It can be ommited to apply the mapping to any source locale.
• The target locale regular expression pattern: .*, ru.*, ru-RU, etc. It can be ommited to apply the mapping to any target locale.
• The source font name regular expression pattern: .*, Arial.*, Times New Roman, etc. It can be ommited to apply the mapping to any source font name found.
• The target font name: Arial, Times New Roman, etc. It should not be empty. And if it is made so, the mapping configuration is ignored.

Also, the configured font mappings are applied in the order they are stated. And the final target font value is determined by a sequential substitution of the source font values. I.e. if there is more than one mapping:

1. Arial -> Times New Roman
2. Times New Roman -> Sans Serif

then the first mapping will produce Times New Roman replacement and the second one will be applied to this new value, thus, ending up with the Sans Serif.

The parameters serialisation format can look like that:

fontMappings.0.sourceLocalePattern=en.*
fontMappings.0.targetLocalePattern=ru.*
fontMappings.0.sourceFontPattern=Times.*
fontMappings.0.targetFont=Arial Unicode MS
fontMappings.1.sourceLocalePattern=ru
fontMappings.1.targetLocalePattern=fr
fontMappings.1.sourceFontPattern=The Sims Sans
fontMappings.1.targetFont=Arial Unicode MS
fontMappings.number.i=2

When source locale, target locale and source font are omitted:

fontMappings.0.targetFont=Arial Unicode MS
fontMappings.number.i=1

And this is the same as the abovementioned:

fontMappings.0.sourceLocalePattern=.*
fontMappings.0.targetLocalePattern=.*
fontMappings.0.sourceFontPattern=.*
fontMappings.0.targetFont=Arial Unicode MS
fontMappings.number.i=1