org.apache.sysds.runtime.compress.colgroup.dictionary

## Class MatrixBlockDictionary

- java.lang.Object
- - org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary
  - - org.apache.sysds.runtime.compress.colgroup.dictionary.MatrixBlockDictionary

All Implemented Interfaces:
  Serializable

    public class MatrixBlockDictionary
    extends ADictionary

See Also:
  Serialized Form

### Constructor Summary

Constructors
Constructor and Description
`MatrixBlockDictionary(MatrixBlock data)`

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`void` `addMaxAndMin(double[] ret, int[] colIndexes)` This method adds the max and min values contained in the dictionary to corresponding cells in the ret variable.
`void` `addToEntry(Dictionary d, int fr, int to, int nCol)` Copies and adds the dictionary entry from this dictionary to the d dictionary
`double` `aggregate(double init, Builtin fn)` Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.
`void` `aggregateCols(double[] c, Builtin fn, int[] colIndexes)` Aggregates the columns into the target double array provided.
`double[]` `aggregateTuples(Builtin fn, int nCol)` Aggregate all entries in the rows.
`ADictionary` `apply(ScalarOperator op)` Applies the scalar operation on the dictionary.
`ADictionary` `applyBinaryRowOpLeft(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes)` Apply binary row operation on this dictionary on the left side.
`ADictionary` `applyBinaryRowOpRight(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes)` Apply binary row operation on this dictionary on the right side.
`ADictionary` `applyScalarOp(ScalarOperator op, double newVal, int numCols)` Applies the scalar operation on the dictionary.
`ADictionary` `clone()` Returns a deep clone of the dictionary.
`ADictionary` `cloneAndExtend(int len)` Clone the dictionary, and extend size of the dictionary by a given length
`void` `colProduct(double[] res, int[] counts, int[] colIndexes)`
`void` `colSum(double[] c, int[] counts, int[] colIndexes, boolean square)` Get the column sum of the values contained in the dictionary
`double[]` `colSum(int[] counts, int nCol)` get the column sum of this dictionary only.
`boolean` `containsValue(double pattern)` Detect if the dictionary contains a specific value.
`MatrixBlockDictionary` `getAsMatrixBlockDictionary(int nCol)` Get this dictionary as a matrixBlock dictionary.
`long` `getExactSizeOnDisk()` Calculate the space consumption if the dictionary is stored on disk.
`long` `getInMemorySize()` Returns the memory usage of the dictionary.
`static long` `getInMemorySize(int numberValues, int numberColumns, double sparsity)`
`MatrixBlock` `getMatrixBlock()`
`long` `getNumberNonZeros(int[] counts, int nCol)` Calculate the number of non zeros in the dictionary.
`int` `getNumberOfValues(int ncol)` Get the number of distinct tuples given that the column group has n columns
`String` `getString(int colIndexes)` Get a string representation of the dictionary, that considers the layout of the data.
`double[]` `getTuple(int index, int nCol)` Get the values contained in a specific tuple of the dictionary.
`double` `getValue(int i)` Get Specific value contained in the dictionary at index.
`double[]` `getValues()` Get all the values contained in the dictionary as a linearized double array.
`boolean` `isLossy()` Specify if the Dictionary is lossy.
`MatrixBlockDictionary` `preaggValuesFromDense(int numVals, int[] colIndexes, int[] aggregateColumns, double[] b, int cut)` Pre Aggregate values for right Matrix Multiplication.
`double` `product(int[] counts, int nCol)`
`static MatrixBlockDictionary` `read(DataInput in)`
`ADictionary` `reExpandColumns(int max)` return a new Dictionary that have re expanded all values, based on the entries already contained.
`ADictionary` `replace(double pattern, double replace, int nCol)` Make a copy of the values, and replace all values that match pattern with replacement value.
`ADictionary` `replaceZeroAndExtend(double replace, int nCol)`
`ADictionary` `scaleTuples(int[] scaling, int nCol)` Scale all tuples contained in the dictionary by the scaling factor given in the int list.
`ADictionary` `sliceOutColumnRange(int idxStart, int idxEnd, int previousNumberOfColumns)` Modify the dictionary by removing columns not within the index range.
`ADictionary` `subtractTuple(double[] tuple)` Allocate a new dictionary where the tuple given is subtracted from all tuples in the previous dictionary.
`double` `sum(int[] counts, int ncol)` Get the sum of the values contained in the dictionary
`double[]` `sumAllRowsToDouble(boolean square, int nrColumns)` Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
`double` `sumRow(int k, boolean square, int nrColumns)` Sum the values at a specific row.
`double` `sumsq(int[] counts, int ncol)` Get the square sum of the values contained in the dictionary
`String` `toString()`
`void` `write(DataOutput out)` Write the dictionary to a DataOutput.

### Methods inherited from class org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

`applyBinaryRowOp, getMostCommonTuple`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Constructor Detail

#### MatrixBlockDictionary

    public MatrixBlockDictionary(MatrixBlock data)

### Method Detail

#### getMatrixBlock

    public MatrixBlock getMatrixBlock()

#### getValues

    public double[] getValues()

Description copied from class: `ADictionary`
Get all the values contained in the dictionary as a linearized double array.

Specified by:
  `getValues` in class `ADictionary`
Returns:
  linearized double array

#### getValue

    public double getValue(int i)

Description copied from class: `ADictionary`
Get Specific value contained in the dictionary at index.

Specified by:
  `getValue` in class `ADictionary`
Parameters:
  `i` - The index to extract the value from
Returns:
  The value contained at the index

#### getInMemorySize

    public long getInMemorySize()

Description copied from class: `ADictionary`
Returns the memory usage of the dictionary.

Specified by:
  `getInMemorySize` in class `ADictionary`
Returns:
  a long value in number of bytes for the dictionary.

#### getInMemorySize

    public static long getInMemorySize(int numberValues,
                                       int numberColumns,
                                       double sparsity)

#### aggregate

    public double aggregate(double init,
                            Builtin fn)

Description copied from class: `ADictionary`
Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.

Specified by:
  `aggregate` in class `ADictionary`
Parameters:
  `init` - The initial Value, in cases such as Max value, this could be -infinity
  `fn` - The Function to apply to values
Returns:
  The aggregated value as a double.

#### aggregateTuples

    public double[] aggregateTuples(Builtin fn,
                                    int nCol)

Description copied from class: `ADictionary`
Aggregate all entries in the rows.

Specified by:
  `aggregateTuples` in class `ADictionary`
Parameters:
  `fn` - The aggregate function
  `nCol` - The number of columns contained in the dictionary.
Returns:
  Aggregates for this dictionary tuples.

#### aggregateCols

    public void aggregateCols(double[] c,
                              Builtin fn,
                              int[] colIndexes)

Description copied from class: `ADictionary`
Aggregates the columns into the target double array provided.

Specified by:
  `aggregateCols` in class `ADictionary`
Parameters:
  `c` - The target double array, this contains the full number of columns, therefore the colIndexes for this specific dictionary is needed.
  `fn` - The function to apply to individual columns
  `colIndexes` - The mapping to the target columns from the individual columns

#### apply

    public ADictionary apply(ScalarOperator op)

Description copied from class: `ADictionary`
Applies the scalar operation on the dictionary. Note that this operation modifies the underlying data, and normally require a copy of the original Dictionary to preserve old objects.

Specified by:
  `apply` in class `ADictionary`
Parameters:
  `op` - The operator to apply to the dictionary values.
Returns:
  this dictionary with modified values.

#### applyScalarOp

    public ADictionary applyScalarOp(ScalarOperator op,
                                     double newVal,
                                     int numCols)

Description copied from class: `ADictionary`
Applies the scalar operation on the dictionary. The returned dictionary should contain a new instance of the underlying data. Therefore it will not modify the previous object.

Specified by:
  `applyScalarOp` in class `ADictionary`
Parameters:
  `op` - The operator to apply to the dictionary values.
  `newVal` - The value to append to the dictionary.
  `numCols` - The number of columns stored in the dictionary.
Returns:
  Another dictionary with modified values.

#### applyBinaryRowOpLeft

    public ADictionary applyBinaryRowOpLeft(BinaryOperator op,
                                            double[] v,
                                            boolean sparseSafe,
                                            int[] colIndexes)

Description copied from class: `ADictionary`
Apply binary row operation on this dictionary on the left side.

Specified by:
  `applyBinaryRowOpLeft` in class `ADictionary`
Parameters:
  `op` - The operation to this dictionary
  `v` - The values to use on the left hand side.
  `sparseSafe` - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary
  `colIndexes` - The column indexes to consider inside v.
Returns:
  A new dictionary containing the updated values.

#### applyBinaryRowOpRight

    public ADictionary applyBinaryRowOpRight(BinaryOperator op,
                                             double[] v,
                                             boolean sparseSafe,
                                             int[] colIndexes)

Description copied from class: `ADictionary`
Apply binary row operation on this dictionary on the right side.

Specified by:
  `applyBinaryRowOpRight` in class `ADictionary`
Parameters:
  `op` - The operation to this dictionary
  `v` - The values to use on the right hand side.
  `sparseSafe` - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary
  `colIndexes` - The column indexes to consider inside v.
Returns:
  A new dictionary containing the updated values.

#### clone

    public ADictionary clone()

Description copied from class: `ADictionary`
Returns a deep clone of the dictionary.

Specified by:
  `clone` in class `ADictionary`

#### cloneAndExtend

    public ADictionary cloneAndExtend(int len)

Description copied from class: `ADictionary`
Clone the dictionary, and extend size of the dictionary by a given length

Specified by:
  `cloneAndExtend` in class `ADictionary`
Parameters:
  `len` - The length to extend the dictionary, it is assumed this value is positive.
Returns:
  a clone of the dictionary, extended by len.

#### isLossy

    public boolean isLossy()

Description copied from class: `ADictionary`
Specify if the Dictionary is lossy.

Specified by:
  `isLossy` in class `ADictionary`
Returns:
  A boolean

#### getNumberOfValues

    public int getNumberOfValues(int ncol)

Description copied from class: `ADictionary`
Get the number of distinct tuples given that the column group has n columns

Specified by:
  `getNumberOfValues` in class `ADictionary`
Parameters:
  `ncol` - The number of Columns in the ColumnGroup.
Returns:
  the number of value tuples contained in the dictionary.

#### sumAllRowsToDouble

    public double[] sumAllRowsToDouble(boolean square,
                                       int nrColumns)

Description copied from class: `ADictionary`
Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.

Specified by:
  `sumAllRowsToDouble` in class `ADictionary`
Parameters:
  `square` - If each entry should be squared.
  `nrColumns` - The number of columns in the ColGroup to know how to get the values from the dictionary.
Returns:
  a double array containing the row sums from this dictionary.

#### sumRow

    public double sumRow(int k,
                         boolean square,
                         int nrColumns)

Description copied from class: `ADictionary`
Sum the values at a specific row.

Specified by:
  `sumRow` in class `ADictionary`
Parameters:
  `k` - The row index to sum
  `square` - If each entry should be squared.
  `nrColumns` - The number of columns
Returns:
  The sum of the row.

#### colSum

    public double[] colSum(int[] counts,
                           int nCol)

Description copied from class: `ADictionary`
get the column sum of this dictionary only.

Specified by:
  `colSum` in class `ADictionary`
Parameters:
  `counts` - the counts of the values contained
  `nCol` - The number of columns contained in each tuple.
Returns:
  the colSums of this column group.

#### colSum

    public void colSum(double[] c,
                       int[] counts,
                       int[] colIndexes,
                       boolean square)

Description copied from class: `ADictionary`
Get the column sum of the values contained in the dictionary

Specified by:
  `colSum` in class `ADictionary`
Parameters:
  `c` - The output array allocated to contain all column groups output.
  `counts` - The counts of the individual tuples.
  `colIndexes` - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.
  `square` - Specify if the values should be squared

#### sum

    public double sum(int[] counts,
                      int ncol)

Description copied from class: `ADictionary`
Get the sum of the values contained in the dictionary

Specified by:
  `sum` in class `ADictionary`
Parameters:
  `counts` - The counts of the individual tuples
  `ncol` - The number of columns contained
Returns:
  The sum scaled by the counts provided.

#### sumsq

    public double sumsq(int[] counts,
                        int ncol)

Description copied from class: `ADictionary`
Get the square sum of the values contained in the dictionary

Specified by:
  `sumsq` in class `ADictionary`
Parameters:
  `counts` - The counts of the individual tuples
  `ncol` - The number of columns contained
Returns:
  The square sum scaled by the counts provided.

#### getString

    public String getString(int colIndexes)

Description copied from class: `ADictionary`
Get a string representation of the dictionary, that considers the layout of the data.

Specified by:
  `getString` in class `ADictionary`
Parameters:
  `colIndexes` - The number of columns in the dictionary.
Returns:
  A string that is nicer to print.

#### addMaxAndMin

    public void addMaxAndMin(double[] ret,
                             int[] colIndexes)

Description copied from class: `ADictionary`
This method adds the max and min values contained in the dictionary to corresponding cells in the ret variable. One use case for this method is the squash operation, to go from an overlapping state to normal compression.

Specified by:
  `addMaxAndMin` in class `ADictionary`
Parameters:
  `ret` - The double array that contains all columns min and max.
  `colIndexes` - The column indexes contained in this dictionary.

#### sliceOutColumnRange

    public ADictionary sliceOutColumnRange(int idxStart,
                                           int idxEnd,
                                           int previousNumberOfColumns)

Description copied from class: `ADictionary`
Modify the dictionary by removing columns not within the index range.

Specified by:
  `sliceOutColumnRange` in class `ADictionary`
Parameters:
  `idxStart` - The column index to start at.
  `idxEnd` - The column index to end at (not inclusive)
  `previousNumberOfColumns` - The number of columns contained in the dictionary.
Returns:
  A dictionary containing the sliced out columns values only.

#### reExpandColumns

    public ADictionary reExpandColumns(int max)

Description copied from class: `ADictionary`
return a new Dictionary that have re expanded all values, based on the entries already contained.

Specified by:
  `reExpandColumns` in class `ADictionary`
Parameters:
  `max` - The number of output columns possible.
Returns:
  The re expanded Dictionary.

#### containsValue

    public boolean containsValue(double pattern)

Description copied from class: `ADictionary`
Detect if the dictionary contains a specific value.

Specified by:
  `containsValue` in class `ADictionary`
Parameters:
  `pattern` - The value to search for
Returns:
  true if the value is contained else false.

#### getNumberNonZeros

    public long getNumberNonZeros(int[] counts,
                                  int nCol)

Description copied from class: `ADictionary`
Calculate the number of non zeros in the dictionary. The number of non zeros should be scaled with the counts given. This gives the exact number of non zero values in the parent column group.

Specified by:
  `getNumberNonZeros` in class `ADictionary`
Parameters:
  `counts` - The counts of each dictionary entry
  `nCol` - The number of columns in this dictionary
Returns:
  The nonZero count

#### addToEntry

    public void addToEntry(Dictionary d,
                           int fr,
                           int to,
                           int nCol)

Description copied from class: `ADictionary`
Copies and adds the dictionary entry from this dictionary to the d dictionary

Specified by:
  `addToEntry` in class `ADictionary`
Parameters:
  `d` - the target dictionary
  `fr` - the from index
  `to` - the to index
  `nCol` - the number of columns

#### getTuple

    public double[] getTuple(int index,
                             int nCol)

Description copied from class: `ADictionary`
Get the values contained in a specific tuple of the dictionary. If the entire row is zero return null.

Specified by:
  `getTuple` in class `ADictionary`
Parameters:
  `index` - The index where the values are located
  `nCol` - The number of columns contained in this dictionary
Returns:
  a materialized double array containing the tuple.

#### subtractTuple

    public ADictionary subtractTuple(double[] tuple)

Description copied from class: `ADictionary`
Allocate a new dictionary where the tuple given is subtracted from all tuples in the previous dictionary.

Specified by:
  `subtractTuple` in class `ADictionary`
Parameters:
  `tuple` - a double list representing a tuple, it is given that the tuple with is the same as this dictionaries.
Returns:
  a new instance of dictionary with the tuple subtracted.

#### getAsMatrixBlockDictionary

    public MatrixBlockDictionary getAsMatrixBlockDictionary(int nCol)

Description copied from class: `ADictionary`
Get this dictionary as a matrixBlock dictionary. This allows us to use optimized kernels coded elsewhere in the system, such as matrix multiplication.

Specified by:
  `getAsMatrixBlockDictionary` in class `ADictionary`
Parameters:
  `nCol` - The number of columns contained in this column group.
Returns:
  A Dictionary containing a MatrixBlock.

#### toString

    public String toString()

Overrides:
  `toString` in class `Object`

#### scaleTuples

    public ADictionary scaleTuples(int[] scaling,
                                   int nCol)

Description copied from class: `ADictionary`
Scale all tuples contained in the dictionary by the scaling factor given in the int list.

Specified by:
  `scaleTuples` in class `ADictionary`
Parameters:
  `scaling` - The ammout to multiply the given tuples with
  `nCol` - The number of columns contained in this column group.
Returns:
  A New dictionary (since we don't want to modify the underlying dictionary)

#### write

    public void write(DataOutput out)
               throws IOException

Description copied from class: `ADictionary`
Write the dictionary to a DataOutput.

Specified by:
  `write` in class `ADictionary`
Parameters:
  `out` - the output sink to write the dictionary to.
Throws:
  `IOException` - if the sink fails.

#### read

    public static MatrixBlockDictionary read(DataInput in)
                                      throws IOException

Throws:
  `IOException`

#### getExactSizeOnDisk

    public long getExactSizeOnDisk()

Description copied from class: `ADictionary`
Calculate the space consumption if the dictionary is stored on disk.

Specified by:
  `getExactSizeOnDisk` in class `ADictionary`
Returns:
  the long count of bytes to store the dictionary.

#### preaggValuesFromDense

    public MatrixBlockDictionary preaggValuesFromDense(int numVals,
                                                       int[] colIndexes,
                                                       int[] aggregateColumns,
                                                       double[] b,
                                                       int cut)

Description copied from class: `ADictionary`
Pre Aggregate values for right Matrix Multiplication.

Specified by:
  `preaggValuesFromDense` in class `ADictionary`
Parameters:
  `numVals` - The number of values contained in this dictionary
  `colIndexes` - The column indexes that is associated with the parent column group
  `aggregateColumns` - The column to aggregate, this is preprocessed, to find remove consideration for empty columns
  `b` - The values in the right hand side matrix
  `cut` - The number of columns in b.
Returns:
  A new dictionary with the pre aggregated values.

#### replace

    public ADictionary replace(double pattern,
                               double replace,
                               int nCol)

Description copied from class: `ADictionary`
Make a copy of the values, and replace all values that match pattern with replacement value. If needed add a new column index.

Specified by:
  `replace` in class `ADictionary`
Parameters:
  `pattern` - The value to look for
  `replace` - The value to replace the other value with
  `nCol` - The number of columns contained in the dictionary.
Returns:
  A new Column Group, reusing the index structure but with new values.

#### replaceZeroAndExtend

    public ADictionary replaceZeroAndExtend(double replace,
                                            int nCol)

Specified by:
  `replaceZeroAndExtend` in class `ADictionary`

#### product

    public double product(int[] counts,
                          int nCol)

Specified by:
  `product` in class `ADictionary`

#### colProduct

    public void colProduct(double[] res,
                           int[] counts,
                           int[] colIndexes)

Specified by:
  `colProduct` in class `ADictionary`