org.apache.sysds.runtime.compress.colgroup.dictionary

## Class ADictionary

- java.lang.Object
- - org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

Direct Known Subclasses:
  Dictionary, MatrixBlockDictionary, QDictionary

    public abstract class ADictionary
    extends Object

This dictionary class aims to encapsulate the storage and operations over unique tuple values of a column group.

### Constructor Summary

Constructors
Constructor and Description
`ADictionary()`

### Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
`abstract void` `addMaxAndMin(double[] ret, int[] colIndexes)` This method adds the max and min values contained in the dictionary to corresponding cells in the ret variable.
`abstract void` `addToEntry(Dictionary d, int fr, int to, int nCol)` Copies and adds the dictionary entry from this dictionary to the d dictionary
`abstract double` `aggregate(double init, Builtin fn)` Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.
`abstract void` `aggregateCols(double[] c, Builtin fn, int[] colIndexes)` Aggregates the columns into the target double array provided.
`abstract double[]` `aggregateTuples(Builtin fn, int nCol)` Aggregate all entries in the rows.
`abstract ADictionary` `apply(ScalarOperator op)` Applies the scalar operation on the dictionary.
`ADictionary` `applyBinaryRowOp(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes, boolean left)` Apply binary row operation on this dictionary.
`abstract ADictionary` `applyBinaryRowOpLeft(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes)` Apply binary row operation on this dictionary on the left side.
`abstract ADictionary` `applyBinaryRowOpRight(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes)` Apply binary row operation on this dictionary on the right side.
`abstract ADictionary` `applyScalarOp(ScalarOperator op, double newVal, int numCols)` Applies the scalar operation on the dictionary.
`abstract ADictionary` `clone()` Returns a deep clone of the dictionary.
`abstract ADictionary` `cloneAndExtend(int len)` Clone the dictionary, and extend size of the dictionary by a given length
`abstract void` `colSum(double[] c, int[] counts, int[] colIndexes, boolean square)` Get the column sum of the values contained in the dictionary
`abstract double[]` `colSum(int[] counts, int nCol)` get the column sum of this dictionary only.
`abstract boolean` `containsValue(double pattern)` Detect if the dictionary contains a specific value.
`abstract MatrixBlockDictionary` `getAsMatrixBlockDictionary(int nCol)` Get this dictionary as a matrixBlock dictionary.
`abstract long` `getExactSizeOnDisk()` Calculate the space consumption if the dictionary is stored on disk.
`abstract long` `getInMemorySize()` Returns the memory usage of the dictionary.
`double[]` `getMostCommonTuple(int[] counts, int nCol)` Get the most common tuple element contained in the dictionary returns null if that tuple is all zero values.
`abstract long` `getNumberNonZeros(int[] counts, int nCol)` Calculate the number of non zeros in the dictionary.
`abstract int` `getNumberOfValues(int ncol)` Get the number of distinct tuples given that the column group has n columns
`abstract String` `getString(int colIndexes)` Get a string representation of the dictionary, that considers the layout of the data.
`abstract double[]` `getTuple(int index, int nCol)` Get the values contained in a specific tuple of the dictionary.
`abstract double` `getValue(int i)` Get Specific value contained in the dictionary at index.
`abstract double[]` `getValues()` Get all the values contained in the dictionary as a linearized double array.
`abstract boolean` `isLossy()` Specify if the Dictionary is lossy.
`abstract void` `preaggValuesFromDense(int numVals, int[] colIndexes, int[] aggregateColumns, double[] b, double[] ret, int cut)` Pre Aggregate values for right Matrix Multiplication.
`abstract ADictionary` `reExpandColumns(int max)` return a new Dictionary that have re expanded all values, based on the entries already contained.
`abstract ADictionary` `replace(double pattern, double replace, int nCol, boolean safe)` Make a copy of the values, and replace all values that match pattern with replacement value.
`abstract ADictionary` `scaleTuples(int[] scaling, int nCol)` Scale all tuples contained in the dictionary by the scaling factor given in the int list.
`abstract ADictionary` `sliceOutColumnRange(int idxStart, int idxEnd, int previousNumberOfColumns)` Modify the dictionary by removing columns not within the index range.
`abstract ADictionary` `subtractTuple(double[] tuple)` Allocate a new dictionary where the tuple given is subtracted from all tuples in the previous dictionary.
`abstract double` `sum(int[] counts, int nCol)` Get the sum of the values contained in the dictionary
`abstract double[]` `sumAllRowsToDouble(boolean square, int nrColumns)` Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
`abstract double` `sumRow(int k, boolean square, int nrColumns)` Sum the values at a specific row.
`abstract double` `sumsq(int[] counts, int nCol)` Get the square sum of the values contained in the dictionary
`abstract void` `write(DataOutput out)` Write the dictionary to a DataOutput.

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### ADictionary

    public ADictionary()

### Method Detail

#### getValues

    public abstract double[] getValues()

Get all the values contained in the dictionary as a linearized double array.

Returns:
  linearized double array

#### getValue

    public abstract double getValue(int i)

Get Specific value contained in the dictionary at index.

Parameters:
  `i` - The index to extract the value from
Returns:
  The value contained at the index

#### getInMemorySize

    public abstract long getInMemorySize()

Returns the memory usage of the dictionary.

Returns:
  a long value in number of bytes for the dictionary.

#### aggregate

    public abstract double aggregate(double init,
                                     Builtin fn)

Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.

Parameters:
  `init` - The initial Value, in cases such as Max value, this could be -infinity
  `fn` - The Function to apply to values
Returns:
  The aggregated value as a double.

#### aggregateTuples

    public abstract double[] aggregateTuples(Builtin fn,
                                             int nCol)

Aggregate all entries in the rows.

Parameters:
  `fn` - The aggregate function
  `nCol` - The number of columns contained in the dictionary.
Returns:
  Aggregates for this dictionary tuples.

#### apply

    public abstract ADictionary apply(ScalarOperator op)

Applies the scalar operation on the dictionary. Note that this operation modifies the underlying data, and normally require a copy of the original Dictionary to preserve old objects.

Parameters:
  `op` - The operator to apply to the dictionary values.
Returns:
  this dictionary with modified values.

#### applyScalarOp

    public abstract ADictionary applyScalarOp(ScalarOperator op,
                                              double newVal,
                                              int numCols)

Applies the scalar operation on the dictionary. The returned dictionary should contain a new instance of the underlying data. Therefore it will not modify the previous object.

Parameters:
  `op` - The operator to apply to the dictionary values.
  `newVal` - The value to append to the dictionary.
  `numCols` - The number of columns stored in the dictionary.
Returns:
  Another dictionary with modified values.

#### applyBinaryRowOp

    public ADictionary applyBinaryRowOp(BinaryOperator op,
                                        double[] v,
                                        boolean sparseSafe,
                                        int[] colIndexes,
                                        boolean left)

Apply binary row operation on this dictionary.

Parameters:
  `op` - The operation to this dictionary
  `v` - The values to use on the left hand side.
  `sparseSafe` - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary
  `colIndexes` - The column indexes to consider inside v.
  `left` - A Boolean specifying if the operation is done on the left or right side of the dictionary.
Returns:
  A new dictionary containing the updated values.

#### applyBinaryRowOpLeft

    public abstract ADictionary applyBinaryRowOpLeft(BinaryOperator op,
                                                     double[] v,
                                                     boolean sparseSafe,
                                                     int[] colIndexes)

Apply binary row operation on this dictionary on the left side.

Parameters:
  `op` - The operation to this dictionary
  `v` - The values to use on the left hand side.
  `sparseSafe` - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary
  `colIndexes` - The column indexes to consider inside v.
Returns:
  A new dictionary containing the updated values.

#### applyBinaryRowOpRight

    public abstract ADictionary applyBinaryRowOpRight(BinaryOperator op,
                                                      double[] v,
                                                      boolean sparseSafe,
                                                      int[] colIndexes)

Apply binary row operation on this dictionary on the right side.

Parameters:
  `op` - The operation to this dictionary
  `v` - The values to use on the right hand side.
  `sparseSafe` - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary
  `colIndexes` - The column indexes to consider inside v.
Returns:
  A new dictionary containing the updated values.

#### clone

    public abstract ADictionary clone()

Returns a deep clone of the dictionary.

Overrides:
  `clone` in class `Object`

#### cloneAndExtend

    public abstract ADictionary cloneAndExtend(int len)

Clone the dictionary, and extend size of the dictionary by a given length

Parameters:
  `len` - The length to extend the dictionary, it is assumed this value is positive.
Returns:
  a clone of the dictionary, extended by len.

#### aggregateCols

    public abstract void aggregateCols(double[] c,
                                       Builtin fn,
                                       int[] colIndexes)

Aggregates the columns into the target double array provided.

Parameters:
  `c` - The target double array, this contains the full number of columns, therefore the colIndexes for this specific dictionary is needed.
  `fn` - The function to apply to individual columns
  `colIndexes` - The mapping to the target columns from the individual columns

#### write

    public abstract void write(DataOutput out)
                        throws IOException

Write the dictionary to a DataOutput.

Parameters:
  `out` - the output sink to write the dictionary to.
Throws:
  `IOException` - if the sink fails.

#### getExactSizeOnDisk

    public abstract long getExactSizeOnDisk()

Calculate the space consumption if the dictionary is stored on disk.

Returns:
  the long count of bytes to store the dictionary.

#### isLossy

    public abstract boolean isLossy()

Specify if the Dictionary is lossy.

Returns:
  A boolean

#### getNumberOfValues

    public abstract int getNumberOfValues(int ncol)

Get the number of distinct tuples given that the column group has n columns

Parameters:
  `ncol` - The number of Columns in the ColumnGroup.
Returns:
  the number of value tuples contained in the dictionary.

#### sumAllRowsToDouble

    public abstract double[] sumAllRowsToDouble(boolean square,
                                                int nrColumns)

Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.

Parameters:
  `square` - If each entry should be squared.
  `nrColumns` - The number of columns in the ColGroup to know how to get the values from the dictionary.
Returns:
  a double array containing the row sums from this dictionary.

#### sumRow

    public abstract double sumRow(int k,
                                  boolean square,
                                  int nrColumns)

Sum the values at a specific row.

Parameters:
  `k` - The row index to sum
  `square` - If each entry should be squared.
  `nrColumns` - The number of columns
Returns:
  The sum of the row.

#### colSum

    public abstract double[] colSum(int[] counts,
                                    int nCol)

get the column sum of this dictionary only.

Parameters:
  `counts` - the counts of the values contained
  `nCol` - The number of columns contained in each tuple.
Returns:
  the colSums of this column group.

#### colSum

    public abstract void colSum(double[] c,
                                int[] counts,
                                int[] colIndexes,
                                boolean square)

Get the column sum of the values contained in the dictionary

Parameters:
  `c` - The output array allocated to contain all column groups output.
  `counts` - The counts of the individual tuples.
  `colIndexes` - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.
  `square` - Specify if the values should be squared

#### sum

    public abstract double sum(int[] counts,
                               int nCol)

Get the sum of the values contained in the dictionary

Parameters:
  `counts` - The counts of the individual tuples
  `nCol` - The number of columns contained
Returns:
  The sum scaled by the counts provided.

#### sumsq

    public abstract double sumsq(int[] counts,
                                 int nCol)

Get the square sum of the values contained in the dictionary

Parameters:
  `counts` - The counts of the individual tuples
  `nCol` - The number of columns contained
Returns:
  The square sum scaled by the counts provided.

#### getString

    public abstract String getString(int colIndexes)

Get a string representation of the dictionary, that considers the layout of the data.

Parameters:
  `colIndexes` - The number of columns in the dictionary.
Returns:
  A string that is nicer to print.

#### addMaxAndMin

    public abstract void addMaxAndMin(double[] ret,
                                      int[] colIndexes)

This method adds the max and min values contained in the dictionary to corresponding cells in the ret variable. One use case for this method is the squash operation, to go from an overlapping state to normal compression.

Parameters:
  `ret` - The double array that contains all columns min and max.
  `colIndexes` - The column indexes contained in this dictionary.

#### sliceOutColumnRange

    public abstract ADictionary sliceOutColumnRange(int idxStart,
                                                    int idxEnd,
                                                    int previousNumberOfColumns)

Modify the dictionary by removing columns not within the index range.

Parameters:
  `idxStart` - The column index to start at.
  `idxEnd` - The column index to end at (not inclusive)
  `previousNumberOfColumns` - The number of columns contained in the dictionary.
Returns:
  A dictionary containing the sliced out columns values only.

#### reExpandColumns

    public abstract ADictionary reExpandColumns(int max)

return a new Dictionary that have re expanded all values, based on the entries already contained.

Parameters:
  `max` - The number of output columns possible.
Returns:
  The re expanded Dictionary.

#### containsValue

    public abstract boolean containsValue(double pattern)

Detect if the dictionary contains a specific value.

Parameters:
  `pattern` - The value to search for
Returns:
  true if the value is contained else false.

#### getNumberNonZeros

    public abstract long getNumberNonZeros(int[] counts,
                                           int nCol)

Calculate the number of non zeros in the dictionary. The number of non zeros should be scaled with the counts given. This gives the exact number of non zero values in the parent column group.

Parameters:
  `counts` - The counts of each dictionary entry
  `nCol` - The number of columns in this dictionary
Returns:
  The nonZero count

#### addToEntry

    public abstract void addToEntry(Dictionary d,
                                    int fr,
                                    int to,
                                    int nCol)

Copies and adds the dictionary entry from this dictionary to the d dictionary

Parameters:
  `d` - the target dictionary
  `fr` - the from index
  `to` - the to index
  `nCol` - the number of columns

#### getMostCommonTuple

    public double[] getMostCommonTuple(int[] counts,
                                       int nCol)

Get the most common tuple element contained in the dictionary returns null if that tuple is all zero values.

Parameters:
  `counts` - The counts of the individual tuples contained, managed by the column group.
  `nCol` - The number of columns contained in this dictionary
Returns:
  a new double array containing the most common value

#### getTuple

    public abstract double[] getTuple(int index,
                                      int nCol)

Get the values contained in a specific tuple of the dictionary.

Parameters:
  `index` - The index where the values are located
  `nCol` - The number of columns contained in this dictionary
Returns:
  a materialized double array containing the tuple.

#### subtractTuple

    public abstract ADictionary subtractTuple(double[] tuple)

Allocate a new dictionary where the tuple given is subtracted from all tuples in the previous dictionary.

Parameters:
  `tuple` - a double list representing a tuple, it is given that the tuple with is the same as this dictionaries.
Returns:
  a new instance of dictionary with the tuple subtracted.

#### getAsMatrixBlockDictionary

    public abstract MatrixBlockDictionary getAsMatrixBlockDictionary(int nCol)

Get this dictionary as a matrixBlock dictionary. This allows us to use optimized kernels coded elsewhere in the system, such as matrix multiplication.

Parameters:
  `nCol` - The number of columns contained in this column group.
Returns:
  A Dictionary containing a MatrixBlock.

#### scaleTuples

    public abstract ADictionary scaleTuples(int[] scaling,
                                            int nCol)

Scale all tuples contained in the dictionary by the scaling factor given in the int list.

Parameters:
  `scaling` - The ammout to multiply the given tuples with
  `nCol` - The number of columns contained in this column group.
Returns:
  A New dictionary (since we don't want to modify the underlying dictionary)

#### preaggValuesFromDense

    public abstract void preaggValuesFromDense(int numVals,
                                               int[] colIndexes,
                                               int[] aggregateColumns,
                                               double[] b,
                                               double[] ret,
                                               int cut)

Pre Aggregate values for right Matrix Multiplication.

Parameters:
  `numVals` - The number of values contained in this dictionary
  `colIndexes` - The column indexes that is associated with the parent column group
  `aggregateColumns` - The column to aggregate, this is preprocessed, to find remove consideration for empty columns
  `b` - The values in the right hand side matrix
  `ret` - The double array to put in the aggregate.
  `cut` - The number of columns in b.

#### replace

    public abstract ADictionary replace(double pattern,
                                        double replace,
                                        int nCol,
                                        boolean safe)

Make a copy of the values, and replace all values that match pattern with replacement value. If needed add a new column index.

Parameters:
  `pattern` - The value to look for
  `replace` - The value to replace the other value with
  `nCol` - The number of columns contained in the dictionary.
  `safe` - Specify if the operation require consideration of adding a new tuple. This happens if the dictionary have allocated the last zero tuple or not.
Returns:
  A new Column Group, reusing the index structure but with new values.