Package org.apache.sysds.runtime.transform.tokenize

## Class Tokenizer

- java.lang.Object
- - org.apache.sysds.runtime.transform.tokenize.Tokenizer

All Implemented Interfaces:
  `Serializable`

    public class Tokenizer
    extends Object
    implements Serializable

See Also:
  Serialized Form

### Field Summary

Fields
`static int` `TOKENIZE_NUM_BLOCKS`

### Method Summary

All Methods Instance Methods Concrete Methods
`void` `allocateInternalRepresentation​(int numDocuments)`
`FrameBlock` `apply​(FrameBlock out, int k)`
`void` `build​(FrameBlock in, int k)`
`List<DependencyTask<?>>` `getBuildTasks​(FrameBlock in)`
`int` `getMaxNumRows​(int inRows)`
`long` `getNumCols()`
`int` `getNumRowsEstimate()`
`Types.ValueType[]` `getSchema()`
`FrameBlock` `tokenize​(FrameBlock in)`
`FrameBlock` `tokenize​(FrameBlock in, int k)`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### TOKENIZE_NUM_BLOCKS

    public static final int TOKENIZE_NUM_BLOCKS

### Method Detail

#### getSchema

    public Types.ValueType[] getSchema()

#### getMaxNumRows

    public int getMaxNumRows​(int inRows)

#### getNumRowsEstimate

    public int getNumRowsEstimate()

#### getNumCols

    public long getNumCols()

#### allocateInternalRepresentation

    public void allocateInternalRepresentation​(int numDocuments)

#### tokenize

    public FrameBlock tokenize​(FrameBlock in)

#### tokenize

    public FrameBlock tokenize​(FrameBlock in,
                               int k)

#### apply

    public FrameBlock apply​(FrameBlock out,
                            int k)

#### getBuildTasks

    public List<DependencyTask<?>> getBuildTasks​(FrameBlock in)

#### build

    public void build​(FrameBlock in,
                      int k)