Package opennlp.tools.util.normalizer

# Class TermAnalyzer

java.lang.Object
opennlp.tools.util.normalizer.TermAnalyzer

public final class TermAnalyzer extends Object
Builds `Term`s by segmenting text and applying a configured stack of normalization `Dimension`s to each token. The analyzer is the configuration; each `Term` is the layered result for one token, with the configured dimensions computed eagerly and any other dimension computed lazily on first request.

Segmentation uses the Unicode UAX #29 word tokenizer, so the input does not need to be pre-tokenized. The character-level dimensions (`Dimension.NFC` through `Dimension.CONFUSABLE_FOLD`) have built-in defaults; `Dimension.STEM` and `Dimension.LEMMA` are enabled by supplying a `Stemmer` or `Lemmatizer`.

An instance is immutable and is thread-safe when its configured transforms are. The built-in character normalizers are stateless and the Snowball stemmers are thread-safe. When `Dimension.STEM` is enabled through `TermAnalyzer.Builder.stem(StemmerFactory)`, stemming is routed through a `CachingStemmer`: the analyzer is safe to share across threads even when the factory mints stateful stemmers, and repeated words resolve from a bounded per-thread cache instead of being re-stemmed. A raw `Stemmer` passed to `TermAnalyzer.Builder.stem(Stemmer)` is only safe when that stemmer is itself thread-safe or the analyzer is confined to one thread.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static final class`
  `TermAnalyzer.Builder`
  A builder for `TermAnalyzer`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `List<Term>`
  `analyze(CharSequence text)`
  Segments `text` with the UAX #29 word tokenizer and returns one `Term` per word token, in order.
  `List<Term>`
  `analyze(String[] tokens, String[] tags)`
  Returns one `Term` per supplied token, attaching the matching part-of-speech tag so that `Dimension.LEMMA` can be computed.
  `static TermAnalyzer.Builder`
  `builder()`
  Returns a new `TermAnalyzer.Builder`.
  `List<Dimension>`
  `dimensions()`
  Returns the configured dimensions that are computed eagerly, in pipeline order.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Method Details

  - ### builder

    public static TermAnalyzer.Builder builder()
    Returns a new `TermAnalyzer.Builder`. The builder starts with no dimensions enabled and the default UAX #29 word tokenizer; enable dimensions and set a stemmer, lemmatizer, or tokenizer on it, then call `TermAnalyzer.Builder.build()`.
    Returns:
      a new `TermAnalyzer.Builder`

  - ### analyze

    public List<Term> analyze(CharSequence text)
    Segments `text` with the UAX #29 word tokenizer and returns one `Term` per word token, in order. The terms carry no part-of-speech tag, so `Dimension.LEMMA` cannot be computed from this entry point: if a lemmatizer is configured, this method throws -- use `analyze(String[], String[])` when lemmas are needed.
    Parameters:
      `text` - The text to analyze. Must not be `null`.
    Returns:
      The terms.
    Throws:
      `IllegalArgumentException` - if `text` is `null`.
      `IllegalStateException` - if `Dimension.LEMMA` is configured, because no part-of-speech tags are available from raw text.

  - ### analyze

    public List<Term> analyze(String[] tokens, String[] tags)
    Returns one `Term` per supplied token, attaching the matching part-of-speech tag so that `Dimension.LEMMA` can be computed. The terms have no source span.
    Parameters:
      `tokens` - The tokens. Must not be `null` or contain `null` elements.
      `tags` - The part-of-speech tag for each token; must be the same length as `tokens` and must not be `null`. A `null` tag is only acceptable when `Dimension.LEMMA` is not computed for that token.
    Returns:
      The terms.
    Throws:
      `IllegalArgumentException` - if `tokens` or `tags` is `null`, if they differ in length, or if `tokens` contains a `null` element.

  - ### dimensions

    public List<Dimension> dimensions()
    Returns the configured dimensions that are computed eagerly, in pipeline order. The list never includes `Dimension.ORIGINAL`, which is always present.
    Returns:
      the configured dimensions that are computed eagerly, in pipeline order