Package opennlp.tools.tokenize.uax29

# Class WordTokenizer

java.lang.Object
opennlp.tools.tokenize.uax29.WordTokenizer

All Implemented Interfaces:
  `opennlp.tools.tokenize.Tokenizer`

public final class WordTokenizer extends Object implements opennlp.tools.tokenize.Tokenizer
A word tokenizer built on the Unicode Text Segmentation algorithm (UAX #29). It finds segments with `WordSegmenter`, keeps the ones that are words (letters, digits, ideographs, kana, Hangul, Southeast-Asian script, or emoji), drops whitespace and punctuation, and classifies each kept token with a `WordType`. Emoji here means any `Extended_Pictographic` code point, so symbol-like characters such as the copyright, trademark, and double-exclamation signs are kept (typed `WordType.EMOJI`) rather than dropped as punctuation.

A token longer than `maxTokenLength` is emitted as consecutive pieces, never splitting a surrogate pair. The tokenizer reports offset `Span`s, so the original text and its character offsets are preserved for downstream normalization.

It implements `Tokenizer`: `tokenize(String)` returns the token strings and `tokenizePos(String)` their offsets. `tokenizeTyped(CharSequence)` additionally carries each token's `WordType`, and `tokenize(CharSequence, TokenHandler)` streams tokens with no per-token allocation. Instances are immutable and thread-safe.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static interface`
  `WordTokenizer.TokenHandler`
  Receives each word token as a character range and its type, with no allocation.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `DEFAULT_MAX_TOKEN_LENGTH`
  The default maximum token length: 255.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `WordTokenizer()`
  Instantiates a tokenizer with the default maximum token length.
  `WordTokenizer(int maxTokenLength)`
  Instantiates a tokenizer with the given maximum token length.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `tokenize(CharSequence text, WordTokenizer.TokenHandler handler)`
  Streams the word tokens of `text` to `handler` in order, allocating nothing.
  `String[]`
  `tokenize(String text)`
  Returns the word tokens of `text` as strings, in order.
  `opennlp.tools.util.Span[]`
  `tokenizePos(String text)`
  Returns the offset spans of the word tokens of `text`, in order.
  `List<opennlp.tools.util.Span>`
  `tokenizeSpans(CharSequence text)`
  Computes the offset spans of the word tokens in `text`, in order.
  `List<WordToken>`
  `tokenizeTyped(CharSequence text)`
  Returns the word tokens in `text`, each carrying its `WordType`, in order.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### DEFAULT_MAX_TOKEN_LENGTH

    public static final int DEFAULT_MAX_TOKEN_LENGTH
    The default maximum token length: 255.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### WordTokenizer

    public WordTokenizer()
    Instantiates a tokenizer with the default maximum token length.

  - ### WordTokenizer

    public WordTokenizer(int maxTokenLength)
    Instantiates a tokenizer with the given maximum token length.
    Parameters:
      `maxTokenLength` - The maximum number of characters in a token; longer tokens are chopped into consecutive pieces. Must be at least `1`.
    Throws:
      `IllegalArgumentException` - Thrown if `maxTokenLength` is equal to or less than `0`.

- ## Method Details

  - ### tokenize

    public void tokenize(CharSequence text, WordTokenizer.TokenHandler handler)
    Streams the word tokens of `text` to `handler` in order, allocating nothing.
    Parameters:
      `text` - The text to tokenize.
      `handler` - The receiver of the tokens.
    Throws:
      `IllegalArgumentException` - Thrown if `text` or `handler` is `null`.

  - ### tokenize

    public String[] tokenize(String text)
    Returns the word tokens of `text` as strings, in order.
    Specified by:
      `tokenize` in interface `opennlp.tools.tokenize.Tokenizer`
    Parameters:
      `text` - The text to tokenize; an empty text yields no tokens.
    Returns:
      The token strings.
    Throws:
      `IllegalArgumentException` - Thrown if `text` is `null`.

  - ### tokenizePos

    public opennlp.tools.util.Span[] tokenizePos(String text)
    Returns the offset spans of the word tokens of `text`, in order.
    Specified by:
      `tokenizePos` in interface `opennlp.tools.tokenize.Tokenizer`
    Parameters:
      `text` - The text to tokenize; an empty text yields no spans.
    Returns:
      The token spans.
    Throws:
      `IllegalArgumentException` - Thrown if `text` is `null`.

  - ### tokenizeSpans

    public List<opennlp.tools.util.Span> tokenizeSpans(CharSequence text)
    Computes the offset spans of the word tokens in `text`, in order.
    Parameters:
      `text` - The text to tokenize; an empty text yields no spans.
    Returns:
      The word-token spans.
    Throws:
      `IllegalArgumentException` - Thrown if `text` is `null`.

  - ### tokenizeTyped

    public List<WordToken> tokenizeTyped(CharSequence text)
    Returns the word tokens in `text`, each carrying its `WordType`, in order.
    Parameters:
      `text` - The text to tokenize; an empty text yields no tokens.
    Returns:
      The typed word tokens.
    Throws:
      `IllegalArgumentException` - Thrown if `text` is `null`.