Package opennlp.tools.util.normalizer

# Class Term

java.lang.Object
opennlp.tools.util.normalizer.Term

public final class Term extends Object
One token as a stack of normalization layers. The `original()` form is the source of truth; the other layers are derived, increasingly aggressive `Dimension`s tuned for matching. The dimensions configured on the producing `TermAnalyzer` are computed eagerly and cached; any other dimension is computed on first request, applied on top of the `configured form`, and then cached.

Because the original is always retained, aggressive folding is safe: a match on a derived layer can always be reported in original coordinates through `span()`. Querying a configured layer, or `peeling` the last-applied one, is O(1); adding an unconfigured dimension costs one transform on first touch and is O(1) thereafter.

Instances are created by `TermAnalyzer` and are thread-safe as long as the analyzer's configured transforms are (see `TermAnalyzer` for the stemmer caveat). Concurrent first requests for the same unconfigured dimension may run its transform more than once, but every thread observes the same cached value.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `String`
  `at(Dimension dimension)`
  Returns the token at `dimension`.
  `String`
  `normalized()`
  Returns the token at the analyzer's final configured dimension; equal to `original()` when no dimensions were configured.
  `String`
  `original()`
  Returns the original token text.
  `String`
  `peel()`
  Returns the token at the dimension just below the final configured one.
  `opennlp.tools.util.Span`
  `span()`
  Returns the source span of this token, or `null` if it was supplied as a pre-tokenized string.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Method Details

  - ### span

    public opennlp.tools.util.Span span()
    Returns the source span of this token, or `null` if it was supplied as a pre-tokenized string. The span indexes into the text passed to `TermAnalyzer.analyze(CharSequence)`.
    Returns:
      the source span of this token, or `null` if it was supplied as a pre-tokenized string

  - ### original

    public String original()
    Returns the original token text.
    Returns:
      the original token text

  - ### normalized

    public String normalized()
    Returns the token at the analyzer's final configured dimension; equal to `original()` when no dimensions were configured.
    Returns:
      the token at the analyzer's final configured dimension; equal to `original()` when no dimensions were configured

  - ### at

    public String at(Dimension dimension)
    Returns the token at `dimension`. Configured dimensions are cached; an unconfigured dimension is computed by applying its transform to `normalized()` and then cached.

    Note: an unconfigured dimension is applied on top of `normalized()` (the most aggressive configured layer), not spliced into pipeline order. Because the transforms do not commute (see `Dimension`), requesting a dimension that ranks earlier than the configured ones can differ from having configured it. For example, asking for `Dimension.CASE_FOLD` on an analyzer configured only through `Dimension.ACCENT_FOLD` case-folds the already accent-folded text, which is not the same as case-folding first. Configure the dimension on the analyzer when pipeline order matters.

    Parameters:
      `dimension` - The dimension to project to. Must not be `null`.
    Returns:
      The token at that dimension.
    Throws:
      `IllegalArgumentException` - if `dimension` is `null`.
      `IllegalStateException` - if the dimension needs an engine or tag that was not configured (see `Dimension.STEM` and `Dimension.LEMMA`).

  - ### peel

    public String peel()
    Returns the token at the dimension just below the final configured one. This is the last-applied layer removed (for example the form before stemming when `Dimension.STEM` is the final dimension); equal to `original()` when at most one dimension is configured.
    Returns:
      the token at the dimension just below the final configured one