Package org.apache.sysds.runtime.instructions.spark.utils

## Class RDDAggregateUtils

- java.lang.Object
- - org.apache.sysds.runtime.instructions.spark.utils.RDDAggregateUtils

    public class RDDAggregateUtils
    extends Object

Collection of utility methods for aggregating binary block rdds. As a general policy always call stable algorithms which maintain corrections over blocks per key. The performance overhead over a simple reducebykey is roughly 7-10% and with that acceptable.

### Constructor Summary

Constructors
`RDDAggregateUtils()`

### Method Summary

All Methods Static Methods Concrete Methods
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `aggByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, AggregateOperator aop)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `aggByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, AggregateOperator aop, boolean deepCopyCombiner)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `aggByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, AggregateOperator aop, int numPartitions, boolean deepCopyCombiner)`
`static MatrixBlock` `aggStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, AggregateOperator aop)` Single block aggregation over pair rdds with corrections for numerical stability.
`static MatrixBlock` `aggStable​(org.apache.spark.api.java.JavaRDD<MatrixBlock> in, AggregateOperator aop)` Single block aggregation over rdds with corrections for numerical stability.
`static TensorBlock` `aggStableTensor​(org.apache.spark.api.java.JavaPairRDD<TensorIndexes,​TensorBlock> in, AggregateOperator aop)` Single block aggregation over pair rdds with corrections for numerical stability.
`static TensorBlock` `aggStableTensor​(org.apache.spark.api.java.JavaRDD<TensorBlock> in, AggregateOperator aop)` Single block aggregation over rdds with corrections for numerical stability.
`static double` `max​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `mergeByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)` Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `mergeByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, boolean deepCopyCombiner)` Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `mergeByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)` Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `mergeRowsByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​RowMatrixBlock> in)` Merges disjoint data of all blocks per key.
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `sumByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `sumByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, boolean deepCopyCombiner)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `sumByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in, int numPartitions, boolean deepCopyCombiner)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double>` `sumCellsByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double> in)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double>` `sumCellsByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double> in, int numParts)`
`static MatrixBlock` `sumStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)`
`static MatrixBlock` `sumStable​(org.apache.spark.api.java.JavaRDD<MatrixBlock> in)`

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### RDDAggregateUtils

    public RDDAggregateUtils()

### Method Detail

#### sumStable

    public static MatrixBlock sumStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)

#### sumStable

    public static MatrixBlock sumStable​(org.apache.spark.api.java.JavaRDD<MatrixBlock> in)

#### sumByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> sumByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)

#### sumByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> sumByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                                                                                        boolean deepCopyCombiner)

#### sumByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> sumByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                                                                                        int numPartitions,
                                                                                                        boolean deepCopyCombiner)

#### sumCellsByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double> sumCellsByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double> in)

#### sumCellsByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double> sumCellsByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​Double> in,
                                                                                                        int numParts)

#### aggStable

    public static MatrixBlock aggStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                        AggregateOperator aop)

Single block aggregation over pair rdds with corrections for numerical stability.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
  `aop` - aggregate operator
Returns:
  matrix block

#### aggStable

    public static MatrixBlock aggStable​(org.apache.spark.api.java.JavaRDD<MatrixBlock> in,
                                        AggregateOperator aop)

Single block aggregation over rdds with corrections for numerical stability.

Parameters:
  `in` - matrix as `JavaRDD<MatrixBlock>`
  `aop` - aggregate operator
Returns:
  matrix block

#### aggStableTensor

    public static TensorBlock aggStableTensor​(org.apache.spark.api.java.JavaPairRDD<TensorIndexes,​TensorBlock> in,
                                              AggregateOperator aop)

Single block aggregation over pair rdds with corrections for numerical stability.

Parameters:
  `in` - tensor as `JavaPairRDD<TensorIndexes, TensorBlock>`
  `aop` - aggregate operator
Returns:
  tensor block

#### aggStableTensor

    public static TensorBlock aggStableTensor​(org.apache.spark.api.java.JavaRDD<TensorBlock> in,
                                              AggregateOperator aop)

Single block aggregation over rdds with corrections for numerical stability.

Parameters:
  `in` - tensor as `JavaRDD<TensorBlock>`
  `aop` - aggregate operator
Returns:
  tensor block

#### aggByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> aggByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                                                                                        AggregateOperator aop)

#### aggByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> aggByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                                                                                        AggregateOperator aop,
                                                                                                        boolean deepCopyCombiner)

#### aggByKeyStable

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> aggByKeyStable​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                                                                                        AggregateOperator aop,
                                                                                                        int numPartitions,
                                                                                                        boolean deepCopyCombiner)

#### max

    public static double max​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)

#### mergeByKey

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> mergeByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in)

Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`

#### mergeByKey

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> mergeByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                                                                                    boolean deepCopyCombiner)

Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
  `deepCopyCombiner` - indicator if the createCombiner functions needs to deep copy the input block
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`

#### mergeByKey

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> mergeByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> in,
                                                                                                    int numPartitions,
                                                                                                    boolean deepCopyCombiner)

Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`
  `numPartitions` - number of output partitions
  `deepCopyCombiner` - indicator if the createCombiner functions needs to deep copy the input block
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`

#### mergeRowsByKey

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> mergeRowsByKey​(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​RowMatrixBlock> in)

Merges disjoint data of all blocks per key. Note: The behavior of this method is undefined for both sparse and dense data if the assumption of disjoint data is violated.

Parameters:
  `in` - matrix as `JavaPairRDD<MatrixIndexes, RowMatrixBlock>`
Returns:
  matrix as `JavaPairRDD<MatrixIndexes, MatrixBlock>`