Package opennlp.tools.tokenize.uax29

# Class WordSegmenter

java.lang.Object
opennlp.tools.tokenize.uax29.WordSegmenter

public final class WordSegmenter extends Object
Finds word boundaries in text using the Unicode Text Segmentation algorithm (UAX #29), rules WB1 through WB999.

The implementation is a single forward cursor pass with O(1) `WordBreakProperty` lookups and no regular expression. It decodes each code point once, keeps only a constant amount of state, and allocates nothing per character. It implements the "ignore" semantics of WB4 (a base character absorbs following `Extend`, `Format`, and `ZWJ`), the look-ahead rules WB6/WB7/WB7b/WB12, the Hebrew quote rules WB7a-WB7c, the emoji zero-width-joiner rule WB3c, and regional-indicator pairing WB15/WB16. The look-ahead for the WB6/WB7b/WB12 rules is resolved lazily and only at mid-word punctuation, so the common case never scans ahead.

`forEachSegment(CharSequence, SegmentConsumer)` streams the segments with no allocation; `boundaries(CharSequence)` returns every boundary offset (always including `0` and the text length); `segments(CharSequence)` returns the spans between them.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static interface`
  `WordSegmenter.SegmentConsumer`
  Receives each word segment as the half-open character range `[start, end)`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static int[]`
  `boundaries(CharSequence text)`
  Returns the word boundary character offsets in `text`, in ascending order, including the boundaries at `0` and `text.length()`.
  `static void`
  `forEachSegment(CharSequence text, WordSegmenter.SegmentConsumer consumer)`
  Streams the word segments of `text` to `consumer` in order, allocating nothing.
  `static List<opennlp.tools.util.Span>`
  `segments(CharSequence text)`
  Returns the word segments of `text` as spans between consecutive boundaries.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Method Details

  - ### forEachSegment

    public static void forEachSegment(CharSequence text, WordSegmenter.SegmentConsumer consumer)
    Streams the word segments of `text` to `consumer` in order, allocating nothing. Each segment is delivered as the half-open character range `[start, end)`; the segments are contiguous and together cover the whole text.
    Parameters:
      `text` - The text to segment; may be empty, in which case no segment is delivered.
      `consumer` - The receiver of the segment ranges.

  - ### boundaries

    public static int[] boundaries(CharSequence text)
    Returns the word boundary character offsets in `text`, in ascending order, including the boundaries at `0` and `text.length()`.
    Parameters:
      `text` - The text to segment.
    Returns:
      The boundary offsets; for empty text, `[0]`.

  - ### segments

    public static List<opennlp.tools.util.Span> segments(CharSequence text)
    Returns the word segments of `text` as spans between consecutive boundaries.
    Parameters:
      `text` - The text to segment.
    Returns:
      The segment spans, in order.