# Changes from M37 to M38

- Changes from M37 to M38
  - General
  - Core
  - Connectors
  - Steps
  - Filters
  - Libraries
  - Applications

## General

    * Fixed [issue #829](https://gitlab.com/okapiframework/okapi/-/issues/829):
      Added the ability to store the filterId in `StartDocument`, `StartSubDocument`
      and `StartSubfilter` and updated all filters to do that.
    * JDK-11 support: Okapi now builds and all unit tests and integrations tests pass with JDK 11.
      This is checked durring the CI run. \
      If you want to try it, add the `-Djdk.source.version=11 -Djdk.target.version=11` switches to all maven invocations,
      or set `[MAVEN_OPTS](https://maven.apache.org/configure.html)`.
    * Updated third party dependencies to the latest versions (except for Lucene)

## Core

    * Fixed [issue #813](https://gitlab.com/okapiframework/okapi/-/issues/813):
      Deprecated `LocaleId.equals(String)` and `LocaleId.compareTo(String)`, added `LocaleId.equalToString(String)` for convenience.
    * Fixed [issue #660](https://gitlab.com/okapiframework/okapi/-/issues/660):
      Some valid UTF-8 sequences where detected as EBCDIC (in `BOMNewlineEncodingDetector`).
    * Deprecated global singletons declared by `Event` (`END_BATCH_EVENT`, `NOOP_EVENT`, `START_BATCH_EVENT`, `START_BATCH_ITEM_EVENT`)

## Connectors

- Microsoft Translator

  - The Microsoft Translator Connector has been upgraded to support the v3 API. Support for v2 has been discontinued.

## Steps

- GTT Batch Translation

  - This step is now deprecated as GTT is shutting down in December 4th 2019.

- Scoping Report

  - Added setter for a `IReportGenerator` so that one can use a custom report generator.

## Filters

- JSON Filter

  - Rename `noteProductionKeys` to `noteRules`. Note rules are now a single regular expression that defines all notes to be extracted
  - Add new regular expression based rules `extractionRules`, `idRules` and `genericMetaRules`. See JSON Filter Documentation

- XLIFF Filter

  - Fixed issue #807: There is now an option (`useSegsForSdlProps`) to use the segment-level SDL-specific properties to re-write the SDL attributes. If this option is not set, the behavior is the same as before, using the SDL-specific properties and the the STATE property set in the target container.
  - The SDL property `text-match` is now extracted as a read-only property for the segments.
  - Fixed issue #844: Attributes with the same name from different namespaces get mixed.
  - Updated XLIFF filter to leave segments that have the `multiple_exact="has_multiple_exact"` attribute unblocked when blocking segments with a `tm_score="100%"` (see issue #844)

- XLIFF 2 Filter

  - Add support for inline codes, notes, groups, metadata and extended attributes and namespaces
  - Remove "Experimental" status

- ICML Filter

  - Fixed issue #846: Forced line-breaks are now extracted as placeholders.

- Markdown Filter

  - Fixed issue #854: the “Translate Code Blocks” option has been split in two to allow separate control over extraction of fenced and inline code blocks. The existing option (`translateCodeBlocks`) has been renamed to “Translated Fenced Code Blocks”. Translation of inline code blocks is now controlled by the new “Translate Inline Code Blocks” option (`translateInlineCodeBlocks`).

- OpenXML Filter

  - Fixed issue #858: crash opening documents that were saved using strict mode.
  - Fixed issue #850: parse XLSX documents with no shared strings table as valid documents with no translatable strings, rather than rejecting them.
  - Fixed issue #843: improved handling of revision metadata related to content that was moved during editing.
  - Fixed issue #835: reorder PPTX notes and comments during extraction so they appear near the slide content to which they refer.
  - Fixed issue #834: improved handling of structural document tags.
  - Fixed issue #830: improved handling of complex field codes spread across multiple paragraphs.
  - Fixed issue #825: include the sheet and cell name in the trans-unit metadata we extract. (This data is exposed as `resname` in resulting XLIFFs.)
  - Fixed issue #823: strip the `a:spc` attribute when aggressively cleaning tags.
  - Fixed issue #803: streamline the way PPTX stlyes are exposed as tags.
  - Fixed issue #864: XLSX merge fails when text followed by empty run with run properties.

## Libraries

- Verification

  - Inline-Code Checker: Added support for the deleteable inline codes.
  - Inline-Code Checker: Added an parameters option (`strictCodeOrder`) to verify the order of the inline codes. If this option is set the order of the codes between source and target is verified using code-types, IDs and code data. Deleteable codes are ignored. If this option is not set, the checker behaves like before, verifying open/close tags. Note that both verifications are done only if no missing or extra tags have been detected.
  - Fixed issue #793: Patterns Checker: You can now use `$N` in the regular expression in addition to the `&lt;same>` special symbol. Each `$N` occurrence is replaced at run-time by the text matching the given group `N`. (You can still use `$` for the end-of-string match).
  - Fixed issue #799: Patterns Checker: There is a new option (Single Pattern) for the pattern rule. If set, we search for that pattern (in the source or target depending on the “search source” option) and trigger a warning if the pattern is found. If not set, we search for the pattern (in the source or target depending on the “search source” option) and trigger an issue if it is not found in the opposite text.
  - Patterns Checker: There is a new parameters option `showOnlyPatternDescription`. If set, we show only the description defined in the pattern rule in the issue message, and you can now use the special code `@@` in the description, it will be replaced by the pattern match found. If not set, you have the old behavior: the message is “The source/target part ‘`<match>`’ has no correspondence in the target/source” (where `<match>` is the text matching the source pattern).
  - General Checker: The issues for white-space now have a segment ID, it is set to the first segment or the last one depending on whether the error is found at the front or at the end of the whole text container.
  - General Checker: Now the white-space verification is not done if the target is empty.
  - General Checker: There is now a parameters option `targetSameAsSourceWithNumbers` to include or not numbers in the segments that are checked. By default they are included (backward compatible)
  - Added a new `SkipCheckAnnotation` to use on the source segments when one want to skip checks on that segments. The steps with segment-level checks have also been updated to take the annotation into account.

## Applications

- Rainbow

  - Can now run from command-line without a display