Okapi Framework Changes Log - Jan-05-2014

Note this document is common to both the okapi-lib distribution and the okapi-apps distribution. The information pertaining to applications other than Tikal are relevant only for the okapi-apps distribution.

• Download page for latest STABLE RELEASE: http://code.google.com/p/okapi/downloads/list
• Download page for latest DEVELOPMENT SNAPSHOTS: http://okapi.opentag.com/snapshots
• Main Web site: http://www.opentag.com/okapi/wiki
• Google Code Project site: http://code.google.com/p/okapi/

Changes from M23 to M24

• Tikal:
• Changed default resource for -q command from OpenTran to Translate Toolkit
• Rainbow:
• Made usability improvements to the Testing Console for rapid iteration when creating custom filter configurations.
• Steps:
• Added the Copy Or Move Step: Copies or moves files to a specified location with the option to overwrite or backup existing files or skip copying files if there is an existing file.
• Rainbow Translation Kit Creation Step:
• Removed the experimental output to XLIFF 2.0 (too outdated to be useful) See the Okapi XLIFF Toolkit project for more up-to-date support for XLIFF 2.0.
• Added option to specify the post-processing hook for OmegaT tkits.
• Format Conversion Step:
• Added the Word Table output format.
• Filters:
• IDML Filter:
• Changed the default spread size threshold to 2000 Kb and updated the warning/error to show the spread size.
• XML Filter:
• Android Strings pre-defined settings: Exposed content of <item> elements for translation (used in <plurals>, <string-array> elements).
• HTML Filter:
• Added option to treat CDATA as an inline element.
• Content of excluded inline elements is exposed for, e.g., inclusion in XLIFF equiv-text attributes.
• Fixed issue #336: The filter will no longer produce translatable segments consisting only of tags.
• XML Stream Filter:
• Fixed issue #336: The filter will no longer produce translatable segments consisting only of tags.
• OpenXML Filter:
• Fixed issue #351: Improve filter performance.
• JSON Filter:
• Fixed issue #377: Support for subfiltering in JSON.
• Fixed issue #373: JSONFilter should use the JSONEncoder.
• Note: Changes in escaping/unescaping behavior in this filter break compatibility with files extracted by previous versions.
• Filters Plugin for OmegaT:
• Added capability to specify a custom filter parameters file for each Okapi filter in the plugin. This closes issue #376.
• Connectors:
• Added the Bilingual File Connector: Directly query a bilingual file format such as TMX, PO, etc., without importing to a TM first.
• Library:
• Important: Changed minimum requirement from Java 1.6 to Java 1.7.
• Fixed ITS content writer to output locQualityIssueProfileRef and not locQualityIssueProfile.
• Improved report output of quality checker.
• Updated and cleaned up the build files.
• Added HUMAN_RECOMMENDED type to the MatchType list.
• Modified the base implementation for IParameters, this may result in compilation errors in your code if you access directly some variables: you should now use the corresponding getter and setter methods.
• Added support for tuv-level attributes that were missing in TMXWriter.

Changes from M22 to M23

• Rainbow:
• Added the Inconsistency Check Step to the pre-defined Quality Check pipeline.
• CheckMate:
• Fixed issue #358: The Check Document button now works in all cases.
• Filters Plugin for OmegaT
• Added .mxliff as one of the default extensions for XLIFF.
• Fixed issue #364: .sdlxliff files with UTF-8 BOM open now.
• Steps:
• Added the Inconsistency Check Step: a way to flag entries with the same source that have different targets or the entries with the same target that have different source.
• Rainbow Translation Kit Creation Step:
• Added a libVersion attribute in the manifest indicating the version of the library used to create the manifest.
• Add option to use encapsulation notation (<bpt>/<ept>/<ph>/<it>) for inline codes in OmegaT tkits.
• LanguageTool Step:
• Updated the library to version 2.2.
• Encoding Conversion Step:
• Fixed issue #318: ASCII characters in NCR form are now un-escaped except for ", ', &, < and >.
• Search and Replace Step:
• Fixed issue #183: Added simple log of the replacements.
• Fixed issue #362: Step for Terminology fixes on translation candidates.
• Quality Check Step:
• Resolved issue #357: Added function to detect blacklisted terms.
• Improved ITS LQI support.
• Filters:
• IDML Filter:
• Implemented issue #356: By default spread above the threshold cause an error. The option allows to skip without error.
• XML Filter:
• Continued implementation of ITS 2.0.
• Fixed issue #361: MIME type can be different in sub-classes of XMLFilter.
• HTML5-ITS Filter:
• Continued implementation of ITS 2.0.
• JSON Filter:
• Resolved issue #360: The use of the key for the resname value is now optional.
• XLIFF Filter:
• Continued implementation of ITS 2.0.
• Fixed issue #364: Woodstox XML parser is now always used.
• Alt-trans with empty target are now skipped.
• Added support for the <tool> and <phase> elements as well as the state-qualifier attribute.
• OpenXML Filter:
• Fixed issue #291: Sub-documents are now processed in correct order.
• Fixed issue #319: 'squishable' tests has been changed.
• Simplification Filter:
• Fixed issue #355: parameters of sub-filter are properly read in the cases where the primary filter uses a sub-filter.
• XINI Filter:
• Fixed a case where placeholders were being renumbered incorrectly when reading a XINI file.
• Library:
• Verification Library:
• Fixed issue with non-initialized start/end variable when checking patterns from the target.
• Added support for sub-document in Quality Checker library.
• Continued implementation of ITS 2.0 in XLIFFWriter, XLIFFContent, etc.
• Fixed issue #352: XMLWriter now throw OkapiIOException if an error occurs.
• Updated XLIFF Writer to match ITS/XLIFF official mapping (http://www.w3.org/International/its/wiki/XLIFF_1.2_Mapping).
• Added the experimental lib-concurrent package to improve multi-threaded pipelines. See ThrededWorkQueue Step page for details.

Changes from M21 to M22

• Tikal:
  • Made it possible to run tikal.sh from another directory on Mac OS X.
  • Updated the way the application root folder was computed to allow call from Network share.
• Rainbow:
• Fixed -log option to allow it anywhere in the command-line.
• Filters:
  • Table Filter:
    • Fixed issue #300 (enhancement): Added a new Table Filter for 2-column (source + target), tab separated files.
  • OpenXML Filter:
    • Fixed issue #166: Text from mc:Fallback and mc:Choice Requites="wps", WordArt, TextArt, and Watermarks is handled properly now.
    • Fixed issue #169: Segmentation around inline codes seems to work properly.
    • Fixed issue #286: PPTX smart-tags are now imported.
    • Fixed issue #323: Files are not corrupted anymore when using text areas.
    • Fixed issue #324: Nested <w:p> merge properly now.
    • Fixed issue #325: The slides of PPTX documents are now extracted in order.
    • Fixed issue #329: Text from PPTX diagrams are now extracted.
    • Fixed issue #351: Creation of XLIFF work on documents with SmartArt graphics.
  • XML Stream Filter (Abstract Markup Filter):
    • Fixed issue #332: When using the global_cdata_subfilter option, the filter will no longer generate extra segments consisting only of placeholders.
    • Fixed issue #339: The filter was not grouping the tags back properly when merging back.
    • Added handling of variable placeholders for the pre-defined settings for RESX files.
  • ITS Filters (XML Filter and HTML5+ITS Filter):
    • Continued implementation of ITS 2.0 (See http://www.opentag.com/okapi/wiki/index.php?title=ITS_Components for details)
  • TMX Filter:
  • Fixed the issue where <it> codes where mapped to placeholder rather than opening/closing internal codes.
  • XLIFF Filter:
    • Continued implementation of ITS 2.0: Improved support for LQI, added support for Provenance.
  • Simplification Filter:
    • Added new filter for resource simplification (removal of references and/or reducing the number of inline codes). See wiki for details: http://www.opentag.com/okapi/wiki/index.php?title=Simplification_Filter.
  • General:
    • Filters that update language properties (like xml:lang) during merging will now be region-insensitive when doing so.
• Steps:
  • Term Extraction Step:
    • Added support for Text Analysis annotations.
    • Made the three extraction methods options, and attached the relevant options to the statistical method.
  • Full-Width Conversion Step:
    • Added log message if at least one character was modified (per input file). This resolves issue #327.
  • Enrycher Step:
    • Improved hanlding of nested annotations.
  • Batch TM Leveraging Step:
    • Fixed issue #331: Entries with no text are now not sent for translation.
  • Format Conversion Step:
  • Fixed the issue where the "Output generic inline codes" was not recognized for the Tab-delimited table output.
  • MS Batch Translation Step:
    • MT candidate with a very low score (e.g. from error) are not output in the TMX.
  • Space Checker Step:
  • Improved reporting of errors and changes.
  • Fixed issue #346: Iterating through text fragments ran out of bounds. Indexing error was fixed.
  • Fixed issue #348: inline code index marker broken as a result of spacing changes. Index marker error was fixed.
  • Translation Comparison Step:
    • improved output statistics for the report. There is now more details and an edit effort score.
      See wiki for details: http://www.opentag.com/okapi/wiki/index.php?title=Translation_Comparison_Step.
  • Consolidated Paragraph Alignment and Sentence Alignment steps
• Connectors:
  • Microsoft MT Connector:
    • Improved error handling (e.g. problem with inline codes in result).
• Filters Plugin for OmegaT:
  • Added *.xliff and *.sdlxliff as default extensions.
  • Changed default for isFileSupported() to return true (this allows user-defined extensions).
• Libraries:
  • Updated XLIFF Writer to match ITS/XLIFF official mapping (http://www.w3.org/International/its/wiki/XLIFF_1.2_Mapping)
  • Fixed QueryUtil HTML-coded conversion methods to allow proper round-trip of <it> codes.
  • Added a new library okapi-lib-preprocessing, contains Preprocessing Filter (available only programmatically), and Simplification Filter (see wiki for details: http://www.opentag.com/okapi/wiki/index.php?title=Simplification_Filter).
  • Fixed issue #347: FilterConfigurationMapper.editConfiguration() now uses Class.getName() like addConfiguration().

Changes from M20 to M21

• All applications:
  • Applications now launch correctly on Mac OS X when they are located in a path containing a space.
• Rainbow:
  • Added the -log option to specify result log file. By default the log file is {user.home}\rainbowBatchLog.txt
• Tikal:
  • Added the -safe option to prompt user when overriding a directory when extracting.
• Filters:
  • ITS Filters (XML Filter and HTML5+ITS Filter):
    • Continued implementation of ITS 2.0 (See http://www.opentag.com/okapi/wiki/index.php?title=ITS_Components for details)
    • Added support for HTML5 translate="" (same as translate="yes").
  • XLIFF Filter:
    • Continued implementation of ITS 2.0.
    • Added support for okp:engine attribute in <alttrans>.
  • Wiki Filter:
    • Fixed issue #315: WikiFilter didn't work with preserve_whitespace: true.
  • Regex Filter:
    • Improved the macStrings default settings to include slash+star comments with next extracted string.
  • IDML Filter:
    • Fixed issue #316: Added default to not extract hidden layers and added the option "Extract hidden layers".
    • Enabled the option "Create new paragraphs on hard returns".
      Important: This option is still BETA and may prevent you to merge back the extracted file. Make sure to test the round-trip before using this option for real projects.
  • TMX Filter:
    • Fixed a bug where attribute values on <tuv> elements were being written back to the skeleton without proper escaping.
    • Improved filter performance.
  • Properties Filter:
    • Fixed issue #313 where the extended characters were not escaped when using the sub-filter.
  • TS Filter:
    • Changed the instantiation of the XML parser to use Woodstox.
  • XML Stream Filter (Abstract Markup Filter):
    • Fixed issue #303: When using the global_pcdata_subfilter option, the filter will no longer generate extra segments consisting only of placeholders.
• Steps:
  • Microsoft Batch Translation Step:
    • Added support for the ${domain} variable for the category.
    • Added support for ${rootDir} and other variables for the path of the Engine mapping file.
  • Quality Check Step:
    • Added option to save or not the session. This option is not accessible when editing the parameters from CheckMate, but only when editing a step's parameters.
  • Rainbow Translation Kit Creation Step:
    • Added to XLIFF outputs the option of outputting ctype and equiv-text attributes in inline codes.
    • Added to OmegaT output the output of ctype and equiv-text attributes in inline codes.
    • Added the option to merge a new OmegaT translation kit with an existing one, rather than overwriting it.
  • Enrycher Step:
    • Added support for segmented text units and implemented handling of inline codes.
    • Added parameter for number of events to process on each call to the service.
  • Translation Comparison Step:
    • Added option to log the average scores per documents in a tab-delimited file.
    • Added the output of a new tab-delimited file with all scores, along with the HTML report.
    • Extended the repartition table to use 11 brackets instead of 3, and include the two scores.
  • Segmentation and Desegmentation Steps:
    • Added the option to renumber code IDs after segmentation so that they are 1-indexed as much as possible. A corresponding option on the desegmentation step reverses the process. This option will not work correctly with formats that use non-consecutive or non-numeric code IDs, such as XLIFF.
• Connectors:
  • Microsoft Translator Connector:
    • Added information about the engine in the query results.
• Libraries:
  • Continued implementation of ITS 2.0 in XLIFF Writer.
  • Changed options settings for the XLIFFWriter class to use an object rather than multiple setters.
• Filters Plugin for OmegaT:
  • Fixed issue #322: Updated the TS filter to use the Woodstox parser, and added the dependencies.
  • Added the XLIFF Filter to the plug-in.
  • Added basic support for some ITS data categories in the Comments pane (Text Analysis, Terminology).

Changes from M19 to M20

• Rainbow:
  • Improved the logging output and UI responsivness during lengthy processes.
  • Updated the user's preference dialog to allow the selection of the log levels as defined by SLF4J (Normal, Debug, Trace)
  • Rainbow's input root directory now supports expansion of system environment variables.
• Tikal:
  • Fixed -lfc command output.
  • Use the -continue option to specify that batch operations should continue processing even if one or more files in the batch fail to process.
  • Summary information will be included at the end of batch commands.
  • Timing information is included for each file processed, and total elapsed time is included in the batch summary.
  • Added the -pd option to specify a directory to search for custom filter configurations.
  • Fixed a crash when merging (-m) a file with no extension.
• Filters:
  • ITS Filters (XML Filter and HTML5+ITS Filter):
    • Continued the implementation of ITS 2.0
    • Fixed issue where the HTML-type special characters were not escaped when converted to inline codes by the code finder.
    • Fixed issue #311 where preserve space property was not applied to attributes.
  • TXML Filter:
    • Fixed issue #266: Translations in the <revisions> elements are now ignored.
  • XLIFF Filter:
    • Improved support for <mrk> elements.
    • Added support for several ITS features.
• Connectors:
  • Microsoft Translator Connector:
    • Fixed the internal conversion of the language code.
• Libraries:
  • Fixed issue #282 for the Abstract Markup Filter.
  • Fixed issues in reporting libraries.
  • Added support for including ITS annotations in the XLIFF 1.2 writer.
• Steps:
  • Rainbow Translation Kit Creation:
    • Added the "Include post-processing hook" option for OmegaT packages. this allows OmegaT to merge back the documents automatically.
  • Remove Target:
    • Fixed issue #270 where the step could not be run without some of the optional parameters set.
  • Microsoft Batch Translation:
    • Added an option to send the generated TMX document as a raw document for the input of the next step.
    • Added a option to point to a .properties file containing a mapping of keys to categories for more convenient lookup. See wiki.
  • Quality Check Step:
    • Fixed issue #304 where the default check on parentheses didn't include full-width characters.
  • Inline Code Removal Step:
    • Added option to replace line break related codes with spaces. By default codes are simply removed.
  • Added the Space Check Step. It allows to fix automatically spaces around inline codes of the target based on the source.
  • Added the Cleanup Step. It allows to normalize quotation marks, punctuations, remove suspect entries, etc. this can be used for example when preparing an aligned document for MT training.