# Package opennlp.tools.util.normalizer

package opennlp.tools.util.normalizer

- Related Packages
  Package
  Description
  opennlp.tools.util
  opennlp.tools.util.eval
  opennlp.tools.util.featuregen
  opennlp.tools.util.jvm
  opennlp.tools.util.model
  opennlp.tools.util.wordvector
- Class
  Description
  AccentFoldCharSequenceNormalizer
  A `CharSequenceNormalizer` that folds diacritics for matching, the multilingual-safe counterpart to a Latin-only ASCII folding filter.
  AggregateCharSequenceNormalizer
  A `CharSequenceNormalizer` implementation that aggregates the functionality of other normalizers.
  BulletCharSequenceNormalizer
  A `CharSequenceNormalizer` that replaces unambiguous list-bullet characters with a space, so a bullet acts as a token separator rather than sticking to the following word.
  CaseFoldCharSequenceNormalizer
  A `CharSequenceNormalizer` that lower cases text for case-insensitive matching.
  Confusables
  Computes the Unicode confusable skeleton of text, following the skeleton algorithm defined in UTS #39 (Unicode Security Mechanisms).
  ConfusableSkeletonCharSequenceNormalizer
  A `CharSequenceNormalizer` that reduces text to its Unicode confusable skeleton (UTS #39).
  DashCharSequenceNormalizer
  A `CharSequenceNormalizer` that maps every Unicode dash to an ASCII hyphen-minus (`U+002D`), reusing the cursor based `CharClass.dashes()` engine.
  DigitCharSequenceNormalizer
  A `CharSequenceNormalizer` that maps Unicode decimal digits to their ASCII equivalents, so for example Arabic-Indic, Devanagari, or fullwidth digits all become `0`-`9`.
  Dimension
  A layer of the `Term` normalization stack, in increasing order of aggressiveness.
  EllipsisCharSequenceNormalizer
  A `CharSequenceNormalizer` that expands the ellipsis and leader characters to ASCII dots: the horizontal ellipsis (`U+2026`) to `"..."` and the two-dot leader (`U+2025`) to `".."`.
  EmojiAnnotation
  The annotation record of one emoji: an extensible per-symbol store of attribute values, each carrying its own provenance.
  EmojiAnnotation.Value
  One attribute value with its provenance, mirroring one row of the bundled data file.
  EmojiAnnotationJoin
  The joined-facts layer of the emoji annotation record store: a hook through which facts that live in user-installed data, typically a gazetteer (coordinates, containment chain, GeoNames or Who's On First identifiers), are resolved at run time and merged into an `EmojiAnnotation`.
  EmojiAnnotations
  The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category), loaded once from the project-authored `emoji-annotations.txt` resource.
  EmojiAnnotator
  Assembles one `EmojiAnnotation` per symbol from bundled facts (`EmojiAnnotations`), derived facts (`EmojiFlags`), and optional joined facts (`EmojiAnnotationJoin`).
  EmojiCategory
  The document-category hint of an annotated emoji, one constant per group header of the upstream Unicode `emoji-test.txt` (Emoji Keyboard/Display Test Data for UTS #51).
  EmojiCharSequenceNormalizer
  Deprecated, for removal: This API element is subject to removal in a future version.
  Replaces every supplementary-plane code point with a space, not only emoji.
  EmojiEntityType
  The coarse entity type of an annotated emoji, a small project-authored taxonomy assigned from the subgroup headers of the upstream Unicode `emoji-test.txt` (Emoji Keyboard/Display Test Data for UTS #51): for example the `face-smiling` and `face-concerned` subgroups map to `EmojiEntityType.FACE` and `transport-ground` maps to `EmojiEntityType.VEHICLE`.
  EmojiFlags
  The derived-facts layer of the emoji annotation record store: decodes flag emoji to ISO 3166 codes purely from the code point sequence, with no data file.
  EmojiToEmoticonCharSequenceNormalizer
  A `CharSequenceNormalizer` that folds emoji to ASCII emoticons, using the bundled `emoji-emoticons.txt` mapping (for example U+1F642 SLIGHTLY SMILING FACE to `:)`).
  EmoticonToEmojiCharSequenceNormalizer
  A `CharSequenceNormalizer` that folds ASCII emoticons to emoji, using the bundled `emoji-emoticons.txt` mapping (for example `:-)` to U+1F642 SLIGHTLY SMILING FACE).
  FullCaseFoldCharSequenceNormalizer
  A `CharSequenceNormalizer` that applies Unicode full case folding for case-insensitive matching, as defined by the Default Case Algorithms in Section 3.13 of the Unicode Standard, using the bundled `CaseFolding.txt` data of the Unicode Character Database.
  GermanUmlautCharSequenceNormalizer
  A `CharSequenceNormalizer` that transliterates German umlauts and the eszett the way German conventionally expands them (DIN 5007-2): a-umlaut to `ae`, o-umlaut to `oe`, u-umlaut to `ue`, and eszett to `ss`, with the capital umlauts and the capital eszett (U+1E9E) expanded likewise.
  InvisibleCharSequenceNormalizer
  A `CharSequenceNormalizer` that removes invisible format and bidirectional control characters that add no textual content and are a common source of noise and spoofing (the byte-order mark, zero width space, word joiner, bidi marks/embeddings/overrides/isolates, the invisible math operators, soft hyphen, and the Arabic letter mark).
  LineBreakPreservingWhitespaceCharSequenceNormalizer
  An `OffsetAwareNormalizer` that collapses runs of Unicode whitespace like `WhitespaceCharSequenceNormalizer`, but keeps line and paragraph structure: any whitespace run that contains a line break collapses to a single newline (`U+000A`) instead of a space, and leading and trailing whitespace is trimmed.
  NfcCharSequenceNormalizer
  A `CharSequenceNormalizer` that applies Unicode Normalization Form C (canonical composition, UAX #15).
  NfkcCharSequenceNormalizer
  A `CharSequenceNormalizer` that applies Unicode Normalization Form KC (compatibility composition, UAX #15).
  NormalizationProfile
  Per-language normalization settings, mirroring how OpenNLP already selects a Snowball stemmer by language.
  NormalizationProfiles
  A registry of `NormalizationProfile`s by language, with detection-based fallback.
  NumberCharSequenceNormalizer
  A `CharSequenceNormalizer` implementation that normalizes text in terms of numbers: every maximal run of ASCII digits (`0` to `9`), that is the longest unbroken stretch of consecutive digits, is replaced by a single space.
  QuoteCharSequenceNormalizer
  A `CharSequenceNormalizer` that folds typographic quotation marks to their ASCII forms: the single quotes and apostrophes to `'` and the double quotes to `"`.
  ShrinkCharSequenceNormalizer
  A `CharSequenceNormalizer` implementation that shrinks repeated whitespace and repeated characters in text, in three steps: Each run of two or more ASCII whitespace characters (tab, line feed, vertical tab, form feed, carriage return, or space) collapses to a single space.
  SocialMediaCharSequenceNormalizer
  A `CharSequenceNormalizer` implementation that normalizes social media text: hashtags, user handles, retweet markers, emoticons, and repeated laughter.
  Term
  One token as a stack of normalization layers.
  TermAnalyzer
  Builds `Term`s by segmenting text and applying a configured stack of normalization `Dimension`s to each token.
  TermAnalyzer.Builder
  A builder for `TermAnalyzer`.
  TextNormalizer
  Entry point for composing normalization steps into a single `CharSequenceNormalizer`.
  TextNormalizer.Builder
  A fluent builder that appends normalization steps in order and composes them into one `CharSequenceNormalizer` via `TextNormalizer.Builder.build()`.
  UrlCharSequenceNormalizer
  A `CharSequenceNormalizer` implementation that normalizes text in terms of URls and email addresses.
  WhitespaceCharSequenceNormalizer
  A `CharSequenceNormalizer` that collapses each run of Unicode whitespace to a single ASCII space and trims the edges, reusing the cursor based `CharClass.whitespace()` engine.