# Package opennlp.tools.tokenize

package opennlp.tools.tokenize

- Related Packages
  Package
  Description
  opennlp.tools.tokenize.lang
  opennlp.tools.tokenize.uax29
- Class
  Description
  BPEModel
  The `BPEModel` stores learned BPE merge operations and can be serialized and deserialized for reuse.
  BPETokenizer
  A `Tokenizer` implementation that performs subword tokenization using Byte Pair Encoding (BPE).
  BPETokenizer.SymbolPair
  Represents a pair of adjacent symbols in BPE.
  BPETokenizerFactory
  A `BaseToolFactory` for BPE tokenization that manages the BPE merge rules artifact and its serialization within a `BPEModel`.
  BPETokenizerTrainer
  Learns BPE merge operations from a training corpus and produces a `BPEModel`.
  DefaultTokenContextGenerator
  A default `TokenContextGenerator` which produces events for maxent decisions for tokenization.
  DetokenizationDictionary
  DetokenizationDictionary.Operation
  DictionaryDetokenizer
  A rule based detokenizer.
  SimpleTokenizer
  A basic `Tokenizer` implementation which performs tokenization using character classes.
  ThreadSafeTokenizerME
  Deprecated.
  As of OPENNLP-1816, `TokenizerME` is itself thread-safe.
  TokenizerCrossValidator
  A cross validator for `tokenizers`.
  TokenizerEvaluator
  The `TokenizerEvaluator` measures the performance of the given `Tokenizer` with the provided reference `samples`.
  TokenizerFactory
  The factory that provides `Tokenizer` default implementation and resources.
  TokenizerME
  A `Tokenizer` for converting raw text into separated tokens.
  TokenizerModel
  The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenizerStream
  The `TokenizerStream` uses a `Tokenizer` to tokenize the input string and output `samples`.
  TokenSampleStream
  This class is a `stream filter` which reads in string encoded samples and creates `samples` out of them.
  TokSpanEventStream
  This class reads the `samples` via an `Iterator` and converts the samples into `events` which can be used by the maxent library for training.
  WhitespaceTokenStream
  This stream formats `ObjectStream` of `samples` into whitespace separated token strings.