Package org.apache.sysds.runtime.transform.tokenize.builder

## Class TokenizerBuilderNgram

- java.lang.Object
- - org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder
  - - org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit
    - - org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram

All Implemented Interfaces:
  `Serializable`

    public class TokenizerBuilderNgram
    extends TokenizerBuilderWhitespaceSplit

See Also:
  Serialized Form

### Field Summary

Fields
`int` `maxGram`
`int` `minGram`
`org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram.NgramType` `ngramType`

### Fields inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

`regex`

### Constructor Summary

Constructors
`TokenizerBuilderNgram​(int[] idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)`

### Method Summary

All Methods Instance Methods Concrete Methods
`void` `createInternalRepresentation​(FrameBlock in, DocumentRepresentation[] internalRepresentation, int rowStart, int blk)`
`List<Token>` `splitIntoNgrams​(Token token, int minGram, int maxGram)`

### Methods inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

`splitToTokens`

### Methods inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder

`createInternalRepresentation, getTasks`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### minGram

    public int minGram

#### maxGram

    public int maxGram

#### ngramType

    public org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram.NgramType ngramType

### Constructor Detail

#### TokenizerBuilderNgram

    public TokenizerBuilderNgram​(int[] idCols,
                                 int tokenizeCol,
                                 org.apache.wink.json4j.JSONObject params)
                          throws org.apache.wink.json4j.JSONException

Throws:
  `org.apache.wink.json4j.JSONException`

### Method Detail

#### splitIntoNgrams

    public List<Token> splitIntoNgrams​(Token token,
                                       int minGram,
                                       int maxGram)

#### createInternalRepresentation

    public void createInternalRepresentation​(FrameBlock in,
                                             DocumentRepresentation[] internalRepresentation,
                                             int rowStart,
                                             int blk)

Overrides:
  `createInternalRepresentation` in class `TokenizerBuilderWhitespaceSplit`