Filters - Revision history

Ctingley: /* List of the Filters */

2026-03-23T23:23:04Z

List of the Filters

← Older revision		Revision as of 19:23, 23 March 2026
Line 13:		Line 13:
	* [[DTD Filter]]		* [[DTD Filter]]
	* [[Doxygen Filter]]		* [[Doxygen Filter]]
	* DXF Filter		* [[DXF Filter]]
	* [[EPUB Filter]]		* [[EPUB Filter]]
	* [[HTML Filter]]		* [[HTML Filter]]

Ctingley: /* Supported File Formats */

2026-03-23T23:22:53Z

Supported File Formats

← Older revision		Revision as of 19:22, 23 March 2026
Line 71:		Line 71:
	\|- valign="top"		\|- valign="top"
	\| Auto Xliff \|\| .xlf, .xliff \|\| <code>okf_autoxliff</code> \|\| [[Auto Xliff Filter]] \|\| Detects the version of an XLIFF file and then hands parsing off to the appropriate filter		\| Auto Xliff \|\| .xlf, .xliff \|\| <code>okf_autoxliff</code> \|\| [[Auto Xliff Filter]] \|\| Detects the version of an XLIFF file and then hands parsing off to the appropriate filter
			\|- valign="top"
			\| AutoCAD DXF \|\| .dxf \|\| <code>okf_dxf</code> \|\| [[DXF Filter]] \|\| Only supports textual DXF, not binary DXF
	\|- valign="top"		\|- valign="top"
	\| CSV (Comma-separated values files) \|\| .csv, .txt \|\| <code>okf_table_csv</code> \|\| [[Table Filter]] \|\|		\| CSV (Comma-separated values files) \|\| .csv, .txt \|\| <code>okf_table_csv</code> \|\| [[Table Filter]] \|\|

Ctingley: Add stub entry for DXF filter

2026-03-23T23:21:42Z

Add stub entry for DXF filter

← Older revision		Revision as of 19:21, 23 March 2026
Line 13:		Line 13:
	* [[DTD Filter]]		* [[DTD Filter]]
	* [[Doxygen Filter]]		* [[Doxygen Filter]]
			* DXF Filter
	* [[EPUB Filter]]		* [[EPUB Filter]]
	* [[HTML Filter]]		* [[HTML Filter]]

Dkonovalyenko: /* Code Simplification Rules */

2025-11-14T09:42:57Z

Code Simplification Rules

← Older revision		Revision as of 05:42, 14 November 2025
Line 210:		Line 210:
	==Code Simplification Rules==		==Code Simplification Rules==

	There are two levels of ~~processing~~: filter and step. ~~If possible, please always prefer the filter level, as the final merge may not always handle the applied code simplification~~ of ~~the step level correctly.~~		There are two levels of code simplification: filter and step (the [[Inline Codes Simplifier Step]] and [[Post-segmentation Inline Codes Removal Step]]). And there are different ways of configuring it:

	~~By default~~, the [[~~Inline Codes Simplifier Step]], [[Simplification~~ Filter~~]] and [[Post-segmentation Inline Codes Removal~~ Step]] maximize the trimming and merging (aka simplification) of inline codes. In some cases, this may not be desired. So, the <code>moveLeadingAndTrailingCodesToSkeleton</code> boolean parameter value set to <code>false</code> can turn off the trimming and the <code>mergeAdjacentCodes</code> - the merging. They are made optional for filters. This means that if they are not present (not mentioned) at the filter level, the specified step will take on the responsibility to handle them.		Firstly, the extraction pipeline can contain just:
			: - [[Raw Document to Filter Events Step]]

	~~Besides~~ that, the simplification rules allow ~~you to prevent~~ specific codes ~~from being trimmed~~ or ~~merged~~.		At the moment, only [[IDML Filter]], [[XML Filter]] and [[Simplification Filter]] support this. It should be noted that the last one performs like a wrapper for another filter.

			Secondly, the extraction pipeline can look like that:
			: - [[Raw Document to Filter Events Step]]
			: - [[Inline Codes Simplifier Step]]

			This is the only way for filters that do not support their own code simplification, and it should be used with care because the final merge may not always handle this correctly. The aforementioned [[IDML Filter]] and [[XML Filter]] can perform their own simplification, and the added [[Inline Codes Simplifier Step]] should not affect the events produced.

			Thirdly, the extraction pipeline can consist of:
			: - [[Raw Document to Filter Events Step]]
			: - [[Segmentation Step]]
			: - [[Post-segmentation Inline Codes Removal Step]]

			Here, the [[Post-segmentation Inline Codes Removal Step]] performs code simplification after segmentation rules are applied, and it may be useful for skipping extra codes between segments.

			By default, the [[Inline Codes Simplifier Step]] and [[Post-segmentation Inline Codes Removal Step]] maximise the trimming and merging (aka simplification) of inline codes. This can be tuned via the following string parameters:
			: - <code>removeLeadingTrailingCodes</code> - <code>true</code> by default
			: - <code>mergeCodes</code> - <code>true</code> by default
			: - <code>rules</code> - empty by default

			Only the [[Inline Codes Simplifier Step]] configuration can be overridden by the optional filter ones via the following parameters:
			: - <code>moveLeadingAndTrailingCodesToSkeleton</code> - maps to the <code>removeLeadingTrailingCodes</code>
			: - <code>mergeAdjacentCodes</code> - maps to the <code>mergeCodes</code>
			: - <code>simplifierRules</code> - maps to the <code>rules</code>

			The simplification rules allow the prevention of specific codes trimming or merging.

	===General Syntax===		===General Syntax===
Line 220:		Line 246:
	The rules parser ignores irrelevant whitespace. Rules can be separated by spaces, newlines or nothing. This makes it easier to accommodate various container formats and their whitespace normalization rules. When a rule applies, it means "do not simplify the match code". Uppercase tokens are constants and predefined by the rule parser. Multiple rules are always OR'ed together.		The rules parser ignores irrelevant whitespace. Rules can be separated by spaces, newlines or nothing. This makes it easier to accommodate various container formats and their whitespace normalization rules. When a rule applies, it means "do not simplify the match code". Uppercase tokens are constants and predefined by the rule parser. Multiple rules are always OR'ed together.

	For more details see the JavaCC grammar: <code>../okapi/core/src/main/javacc/SimplifierRules.jj</code>		For more details, see the JavaCC grammar: <code>../okapi/core/src/main/javacc/SimplifierRules.jj</code>

	===Rule Examples===		===Rule Examples===

	If Code has any of these flags then don't simplify		If Code has any of these flags, then don't simplify

	<pre>if DELETABLE or ADDABLE or CLONEABLE;</pre>		<pre>if DELETABLE or ADDABLE or CLONEABLE;</pre>

Dkonovalyenko: /* Code Simplification Rules */

2025-09-23T13:48:35Z

Code Simplification Rules

← Older revision		Revision as of 09:48, 23 September 2025
Line 210:		Line 210:
	==Code Simplification Rules==		==Code Simplification Rules==

	~~All filters support~~ code simplification ~~rules~~. By default the [[Inline Codes Simplifier Step]], [[Simplification Filter]] and [[Post-segmentation Inline Codes Removal Step]] maximize the trimming and merging (aka simplification) of inline codes. In some cases this may not be desired. ~~The~~ simplification rules allow you to ~~override the default behavior and~~ prevent specific codes from being trimmed or merged.		There are two levels of processing: filter and step. If possible, please always prefer the filter level, as the final merge may not always handle the applied code simplification of the step level correctly.

			By default, the [[Inline Codes Simplifier Step]], [[Simplification Filter]] and [[Post-segmentation Inline Codes Removal Step]] maximize the trimming and merging (aka simplification) of inline codes. In some cases, this may not be desired. So, the <code>moveLeadingAndTrailingCodesToSkeleton</code> boolean parameter value set to <code>false</code> can turn off the trimming and the <code>mergeAdjacentCodes</code> - the merging. They are made optional for filters. This means that if they are not present (not mentioned) at the filter level, the specified step will take on the responsibility to handle them.

			Besides that, the simplification rules allow you to prevent specific codes from being trimmed or merged.

	===General Syntax===		===General Syntax===

	The rules parser ignores irrelevant whitespace. Rules can be separated by spaces, newlines or nothing. This makes it easier to accommodate various container formats and their whitespace normalization rules. When a rule applies it means "do not simplify the match code". Uppercase tokens are constants and predefined by the rule parser. Multiple rules are always OR'ed together.		The rules parser ignores irrelevant whitespace. Rules can be separated by spaces, newlines or nothing. This makes it easier to accommodate various container formats and their whitespace normalization rules. When a rule applies, it means "do not simplify the match code". Uppercase tokens are constants and predefined by the rule parser. Multiple rules are always OR'ed together.

	For more details see the JavaCC grammar: <code>../okapi/core/src/main/javacc/SimplifierRules.jj</code>		For more details see the JavaCC grammar: <code>../okapi/core/src/main/javacc/SimplifierRules.jj</code>
Line 273:		Line 277:
	<its:withinTextRule selector="//codeph" withinText="yes"/>		<its:withinTextRule selector="//codeph" withinText="yes"/>
	<its:withinTextRule selector="//ph" withinText="yes"/>		<its:withinTextRule selector="//ph" withinText="yes"/>
	<okp:simplifierRules>		<okp:simplifierRules moveLeadingAndTrailingCodesToSkeleton="yes" mergeAdjacentCodes="yes">
	if ADDABLE or DELETABLE or CLONEABLE; if DATA ~ ".+";		if ADDABLE or DELETABLE or CLONEABLE; if DATA ~ ".+";
	</okp:simplifierRules>		</okp:simplifierRules>

Jhargraveiii: /* List of the Filters */

2023-11-16T19:54:32Z

List of the Filters

← Older revision		Revision as of 15:54, 16 November 2023
Line 20:		Line 20:
	* [[JSON Filter]]		* [[JSON Filter]]
	* [[Markdown Filter]]		* [[Markdown Filter]]
			* [[Message Format Filter]]
	* [[MIF Filter]]		* [[MIF Filter]]
	* [[Moses Text Filter]]		* [[Moses Text Filter]]
Line 52:		Line 53:
	* [[XML Stream Filter]]		* [[XML Stream Filter]]
	* [[YAML Filter]]		* [[YAML Filter]]
	* [[Message Format Filter]]
	\|}		\|}

Jhargraveiii: /* General Syntax */

2023-11-13T21:54:48Z

General Syntax

← Older revision		Revision as of 17:54, 13 November 2023
Line 216:		Line 216:
	The rules parser ignores irrelevant whitespace. Rules can be separated by spaces, newlines or nothing. This makes it easier to accommodate various container formats and their whitespace normalization rules. When a rule applies it means "do not simplify the match code". Uppercase tokens are constants and predefined by the rule parser. Multiple rules are always OR'ed together.		The rules parser ignores irrelevant whitespace. Rules can be separated by spaces, newlines or nothing. This makes it easier to accommodate various container formats and their whitespace normalization rules. When a rule applies it means "do not simplify the match code". Uppercase tokens are constants and predefined by the rule parser. Multiple rules are always OR'ed together.

	For more details see the JavaCC grammar: <code>../okapi-core/src/main/~~java/net/sf/okapi/core/simplifierrules~~/SimplifierRules.jj</code>		For more details see the JavaCC grammar: <code>../okapi/core/src/main/javacc/SimplifierRules.jj</code>

	===Rule Examples===		===Rule Examples===

Jhargraveiii at 21:46, 12 October 2023

2023-10-12T21:46:46Z

← Older revision		Revision as of 17:46, 12 October 2023
Line 203:		Line 203:
	\| YAML (Generic YAML filter) \|\| .yml, .yaml \|\| <code>okf_yaml</code> \|\| [[YAML Filter]] \|\|		\| YAML (Generic YAML filter) \|\| .yml, .yaml \|\| <code>okf_yaml</code> \|\| [[YAML Filter]] \|\|
	\|- valign="top"		\|- valign="top"
	\| Message Format (ICU Message Format Filter) \|\| <code>okf_messageformat</code> \|\| [[Message Format Filter]] \|\|		\| Message Format (ICU Message Format Filter) \|\| Any container format that supports subfilters \|\| <code>okf_messageformat</code> \|\| [[Message Format Filter]] \|\|
	\|}		\|}

Jhargraveiii at 21:45, 12 October 2023

2023-10-12T21:45:57Z

← Older revision		Revision as of 17:45, 12 October 2023
Line 203:		Line 203:
	\| YAML (Generic YAML filter) \|\| .yml, .yaml \|\| <code>okf_yaml</code> \|\| [[YAML Filter]] \|\|		\| YAML (Generic YAML filter) \|\| .yml, .yaml \|\| <code>okf_yaml</code> \|\| [[YAML Filter]] \|\|
	\|- valign="top"		\|- valign="top"
	\| Message Format (ICU Message Format Filter) ~~\|\| .yml, .yaml~~ \|\| <code>okf_messageformat</code> \|\| [[Message Format Filter]] \|\|		\| Message Format (ICU Message Format Filter) \|\| <code>okf_messageformat</code> \|\| [[Message Format Filter]] \|\|
	\|}		\|}

Jhargraveiii at 21:36, 12 October 2023

2023-10-12T21:36:11Z

← Older revision		Revision as of 17:36, 12 October 2023
Line 52:		Line 52:
	* [[XML Stream Filter]]		* [[XML Stream Filter]]
	* [[YAML Filter]]		* [[YAML Filter]]
			* [[Message Format Filter]]
	\|}		\|}

Line 201:		Line 202:
	\|- valign="top"		\|- valign="top"
	\| YAML (Generic YAML filter) \|\| .yml, .yaml \|\| <code>okf_yaml</code> \|\| [[YAML Filter]] \|\|		\| YAML (Generic YAML filter) \|\| .yml, .yaml \|\| <code>okf_yaml</code> \|\| [[YAML Filter]] \|\|
			\|- valign="top"
			\| Message Format (ICU Message Format Filter) \|\| .yml, .yaml \|\| <code>okf_messageformat</code> \|\| [[Message Format Filter]] \|\|
	\|}		\|}