# Package opennlp.tools.tokenize.uax29

package opennlp.tools.tokenize.uax29

- Related Packages
  Package
  Description
  opennlp.tools.tokenize
  opennlp.tools.tokenize.lang
- Class
  Description
  ExtendedPictographic
  Checks the Unicode `Extended_Pictographic` property of a code point.
  WordBreak
  The Unicode `Word_Break` property values, used by the UAX #29 word boundary algorithm.
  WordBreakProperty
  Looks up the Unicode `Word_Break` property of a code point.
  WordSegmenter
  Finds word boundaries in text using the Unicode Text Segmentation algorithm (UAX #29), rules WB1 through WB999.
  WordSegmenter.SegmentConsumer
  Receives each word segment as the half-open character range `[start, end)`.
  WordToken
  A word token produced by `WordTokenizer`: the character `Span` into the source text together with its `WordType`.
  WordTokenizer
  A word tokenizer built on the Unicode Text Segmentation algorithm (UAX #29).
  WordTokenizer.TokenHandler
  Receives each word token as a character range and its type, with no allocation.
  WordType
  The category of a word token.