Package org.apache.sysds.runtime.transform.tokenize

## Class TokenizerPreNgram

- java.lang.Object
- - org.apache.sysds.runtime.transform.tokenize.TokenizerPreNgram

All Implemented Interfaces:
  `Serializable`, `TokenizerPre`

    public class TokenizerPreNgram
    extends Object
    implements TokenizerPre

See Also:
  Serialized Form

### Constructor Summary

Constructors
`TokenizerPreNgram​(List<Integer> idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)`

### Method Summary

All Methods Instance Methods Concrete Methods
`List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens>` `tokenizePre​(FrameBlock in)`
`List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token>` `wordTokenListToNgrams​(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokens)`
`List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token>` `wordTokenToNgrams​(org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token wordTokens)`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### TokenizerPreNgram

    public TokenizerPreNgram​(List<Integer> idCols,
                             int tokenizeCol,
                             org.apache.wink.json4j.JSONObject params)
                      throws org.apache.wink.json4j.JSONException

Throws:
  `org.apache.wink.json4j.JSONException`

### Method Detail

#### wordTokenToNgrams

    public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokenToNgrams​(org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token wordTokens)

#### wordTokenListToNgrams

    public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokenListToNgrams​(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokens)

#### tokenizePre

    public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tokenizePre​(FrameBlock in)

Specified by:
  `tokenizePre` in interface `TokenizerPre`