Package org.apache.sysds.runtime.transform.tokenize

## Interface TokenizerPre

All Superinterfaces:
  `Serializable`

All Known Implementing Classes:
  `TokenizerPreNgram`, `TokenizerPreWhitespaceSplit`

    public interface TokenizerPre
    extends Serializable

### Method Summary

All Methods Instance Methods Abstract Methods
`List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens>` `tokenizePre​(FrameBlock in)`

### Method Detail

#### tokenizePre

    List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tokenizePre​(FrameBlock in)