<?xml version="1.0"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
	<id>http://okapiframework.org/wiki/index.php?action=history&amp;feed=atom&amp;title=Encoding_Conversion_Step</id>
	<title>Encoding Conversion Step - Revision history</title>
	<link rel="self" type="application/atom+xml" href="http://okapiframework.org/wiki/index.php?action=history&amp;feed=atom&amp;title=Encoding_Conversion_Step"/>
	<link rel="alternate" type="text/html" href="http://okapiframework.org/wiki/index.php?title=Encoding_Conversion_Step&amp;action=history"/>
	<updated>2026-11-20T09:33:26Z</updated>
	<subtitle>Revision history for this page on the wiki</subtitle>
	<generator>MediaWiki 1.46.2</generator>
	<entry>
		<id>http://okapiframework.org/wiki/index.php?title=Encoding_Conversion_Step&amp;diff=379&amp;oldid=prev</id>
		<title>Ysavourel: 1 revision imported</title>
		<link rel="alternate" type="text/html" href="http://okapiframework.org/wiki/index.php?title=Encoding_Conversion_Step&amp;diff=379&amp;oldid=prev"/>
		<updated>2016-06-04T23:20:00Z</updated>

		<summary type="html">&lt;p&gt;1 revision imported&lt;/p&gt;
&lt;table style=&quot;background-color: #fff; color: #202122;&quot; data-mw-interface=&quot;&quot;&gt;
				&lt;tr class=&quot;diff-title&quot; lang=&quot;en&quot;&gt;
				&lt;td colspan=&quot;1&quot; style=&quot;background-color: #fff; color: #202122; text-align: center;&quot;&gt;← Older revision&lt;/td&gt;
				&lt;td colspan=&quot;1&quot; style=&quot;background-color: #fff; color: #202122; text-align: center;&quot;&gt;Revision as of 23:20, 4 June 2016&lt;/td&gt;
				&lt;/tr&gt;&lt;tr&gt;&lt;td colspan=&quot;2&quot; class=&quot;diff-notice&quot; lang=&quot;en&quot;&gt;&lt;div class=&quot;mw-diff-empty&quot;&gt;(No difference)&lt;/div&gt;
&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;</summary>
		<author><name>Ysavourel</name></author>
	</entry>
	<entry>
		<id>http://okapiframework.org/wiki/index.php?title=Encoding_Conversion_Step&amp;diff=378&amp;oldid=prev</id>
		<title>Ysavourel at 20:26, 13 September 2011</title>
		<link rel="alternate" type="text/html" href="http://okapiframework.org/wiki/index.php?title=Encoding_Conversion_Step&amp;diff=378&amp;oldid=prev"/>
		<updated>2011-09-13T20:26:11Z</updated>

		<summary type="html">&lt;p&gt;&lt;/p&gt;
&lt;p&gt;&lt;b&gt;New page&lt;/b&gt;&lt;/p&gt;&lt;div&gt;{{Steps Header}}&lt;br /&gt;
__TOC__&lt;br /&gt;
==Overview==&lt;br /&gt;
&lt;br /&gt;
This step converts the character set encoding of a input document, and in some cases, updates its encoding declaration.&lt;br /&gt;
&lt;br /&gt;
Takes: Raw document. Sends: Raw document.&lt;br /&gt;
&lt;br /&gt;
===Updating Encoding Declarations===&lt;br /&gt;
&lt;br /&gt;
The encoding declarations of XML and HTML documents, given a few conditions.&lt;br /&gt;
&lt;br /&gt;
The following algorithms are run on the first 1024 characters of each document. Note that the routine is not XML or HTML aware and cannot make distinction between normal XML/HTML code and XML/HTML comments.&lt;br /&gt;
&lt;br /&gt;
====Detection of XML Documents====&lt;br /&gt;
&lt;br /&gt;
* If an XML encoding declaration is detected:&lt;br /&gt;
** The encoding value is updated. &lt;br /&gt;
* Otherwise, if an XML declaration is detected:&lt;br /&gt;
** An encoding declaration is inserted inside the XML declaration just after the version. &lt;br /&gt;
* Otherwise, if the name of the file ends with a .xml extension:&lt;br /&gt;
** An XML declaration (with an encoding declaration of the output encoding) is added at the top of the document. &lt;br /&gt;
&lt;br /&gt;
====Detection of the HTML Documents====&lt;br /&gt;
&lt;br /&gt;
* If an HTML charset declaration is detected: &lt;br /&gt;
** The charset value is updated.&lt;br /&gt;
* Otherwise, if the name of the file ends with an extension that starts with &amp;lt;code&amp;gt;.htm&amp;lt;/code&amp;gt;:&lt;br /&gt;
** If a &amp;lt;code&amp;gt;&amp;amp;lt;head&amp;gt;&amp;lt;/code&amp;gt; element is found:&lt;br /&gt;
*** A charset declaration is added just after.&lt;br /&gt;
** Otherwise, if a &amp;lt;code&amp;gt;&amp;amp;lt;html&amp;lt;/code&amp;gt; string is detected:&lt;br /&gt;
*** A &amp;lt;code&amp;gt;&amp;amp;lt;head&amp;gt;&amp;lt;/code&amp;gt; and a charset declaration are added after the first &amp;#039;&amp;lt;code&amp;gt;&amp;gt;&amp;lt;/code&amp;gt;&amp;#039; after the string found.&lt;br /&gt;
&lt;br /&gt;
Note that a document can be both and XML and HTML document and have both types of encoding/charset declarations.&lt;br /&gt;
&lt;br /&gt;
==Parameters==&lt;br /&gt;
&lt;br /&gt;
===Input Tab===&lt;br /&gt;
&lt;br /&gt;
====Un-escape the following notations====&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Numeric character references&amp;lt;/cite&amp;gt; &amp;amp;mdash; Set this option to un-escape all types of numeric character references (NCRs) when reading the input document. For example: &amp;lt;code&amp;gt;&amp;amp;amp;#255;&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;&amp;amp;amp;#xE1;&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;&amp;amp;amp;#e1;&amp;lt;/code&amp;gt; will be un-escaped to &amp;#039;á&amp;#039;. If this option is not set, any NCR in the input document will remain in the exact same form in the output document. For more on NCR, see http://en.wikipedia.org/wiki/Numeric_character_reference.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Character entity references&amp;lt;/cite&amp;gt; &amp;amp;mdash; Set this option to un-escape the standard HTML character entity references (CERs) when reading the input document. For example: &amp;lt;code&amp;gt;&amp;amp;amp;aacute;&amp;lt;/code&amp;gt; will be un-escaped to &amp;#039;á&amp;#039;. If this option is not set, any CER in the input document will remain in the exact same form in the output document. For more on CER, see http://en.wikipedia.org/wiki/Character_entity_reference. &lt;br /&gt;
&lt;br /&gt;
Note that the character entity references &amp;lt;code&amp;gt;&amp;amp;amp;amp;&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;&amp;amp;amp;lt;&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;&amp;amp;amp;gt;&amp;lt;/code&amp;gt;, &amp;lt;code&amp;gt;&amp;amp;amp;apos;&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;&amp;amp;amp;quot;&amp;lt;/code&amp;gt; are not un-escaped as they may need to be preserved regardless of the encoding in HTML and XML document.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Java-style escape notation&amp;lt;/cite&amp;gt; &amp;amp;mdash; Set this option to un-escape the Java-style escaped characters when reading the input document. For example: &amp;lt;code&amp;gt;\u00e1&amp;lt;/code&amp;gt; and &amp;lt;code&amp;gt;\u00E1&amp;lt;/code&amp;gt; will be un-escaped to &amp;#039;á&amp;#039;. If this option is not set all Java-style escaped characters in the input document will remain in the exact same form in the output document.&lt;br /&gt;
&lt;br /&gt;
===Output Tab===&lt;br /&gt;
&lt;br /&gt;
====What characters should be escaped====&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Only the characters un-supported by the output encoding&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to escape only characters not supported by the output encoding.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;All extended characters&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to escape all extended characters.&lt;br /&gt;
&lt;br /&gt;
====Escape notation to use====&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Uppercase hexadecimal numeric character reference&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to use an uppercase hexadecimal NCR when escaping a character. For example &amp;#039;á&amp;#039; will become &amp;lt;code&amp;gt;&amp;amp;amp;#xE1;&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Lowercase hexadecimal numeric character reference&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to use an lowercase hexadecimal NCR when escaping a character. For example &amp;#039;á&amp;#039; will become &amp;lt;code&amp;gt;&amp;amp;amp;#xe1;&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Decimal numeric character reference&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to use a decimal NCR when escaping a character. For example &amp;#039;á&amp;#039; will become &amp;lt;code&amp;gt;&amp;amp;amp;#255;&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Character entity references&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to use a character entity reference when escaping a character. For example &amp;#039;á&amp;#039; will become &amp;lt;code&amp;gt;&amp;amp;amp;#aacute;&amp;lt;/code&amp;gt;. If there is no corresponding entity defined for the character to escape, the uppercase hexadecimal NCR form is used instead.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Uppercase Java-style notation&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to use an uppercase Java-style notation when escaping a character. For example &amp;#039;á&amp;#039; will become &amp;lt;code&amp;gt;\u00E1&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Lowercase Java-style notation&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to use a lowercase Java-style notation when escaping a character. For example &amp;#039;á&amp;#039; will become &amp;lt;code&amp;gt;\u00e1&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;User-defined notation&amp;lt;/cite&amp;gt; &amp;amp;mdash; Select this option to use a customized format when escaping a character. The user-defined format must be a Java format expression that take an integer as the value to display. For example the expression &amp;lt;code&amp;gt;&amp;lt;nowiki&amp;gt;[[%d]]&amp;lt;/nowiki&amp;gt;&amp;lt;/code&amp;gt; for &amp;#039;á&amp;#039; will give &amp;lt;code&amp;gt;&amp;lt;nowiki&amp;gt;[[255]]&amp;lt;/nowiki&amp;gt;&amp;lt;/code&amp;gt;, and the expression &amp;lt;code&amp;gt;\&amp;#039;%x&amp;lt;/code&amp;gt; for 日本語 will give &amp;lt;code&amp;gt;\&amp;#039;65e5\&amp;#039;672c\&amp;#039;8a9e&amp;lt;/code&amp;gt;. The value display in the Unicode code point of the character to escape.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Use the bytes values&amp;lt;/cite&amp;gt; &amp;amp;mdash; Set this option when you want the values applied to the user-defined expression to be the byte value of the output encoding rather than the Unicode code points. For example, the expression &amp;lt;code&amp;gt;\&amp;#039;%x&amp;lt;/code&amp;gt; for 日本語 will give &amp;lt;code&amp;gt;\&amp;#039;93\&amp;#039;fa\&amp;#039;96\&amp;#039;7b\&amp;#039;8c\&amp;#039;ea&amp;lt;/code&amp;gt; if this option is set and the output encoding is Shift-JIS.&lt;br /&gt;
&lt;br /&gt;
====Miscellaneous====&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;Use Byte-Order-Mark for UTF-8 output&amp;lt;/cite&amp;gt; &amp;amp;mdash; Set this option to add a Byte-Order-Mark (BOM) for UTF-8 output. For more information on the BOM see http://www.unicode.org/faq/utf_bom.html.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;cite&amp;gt;List characters not supported by the output encoding&amp;lt;/cite&amp;gt; &amp;amp;mdash; Set this option to list in the Log all characters not supported by the output encoding.&lt;br /&gt;
&lt;br /&gt;
[[Category:Steps]]&lt;/div&gt;</summary>
		<author><name>Ysavourel</name></author>
	</entry>
</feed>