Package org.apache.sysds.runtime.instructions.spark.utils

## Class RDDConverterUtilsExt

- java.lang.Object
- - org.apache.sysds.runtime.instructions.spark.utils.RDDConverterUtilsExt

    public class RDDConverterUtilsExt
    extends Object

NOTE: These are experimental converter utils. Once thoroughly tested, they can be moved to RDDConverterUtils.

### Nested Class Summary

Nested Classes
`static class` `RDDConverterUtilsExt.AddRowID`
`static class` `RDDConverterUtilsExt.RDDConverterTypes`

### Constructor Summary

Constructors
`RDDConverterUtilsExt()`

### Method Summary

All Methods Static Methods Concrete Methods
`static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>` `addIDToDataFrame​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, org.apache.spark.sql.SparkSession sparkSession, String nameOfCol)` Add element indices as new column to DataFrame
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `coordinateMatrixToBinaryBlock​(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input, DataCharacteristics mcIn, boolean outputEmptyBlocks)`
`static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock>` `coordinateMatrixToBinaryBlock​(org.apache.spark.SparkContext sc, org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input, DataCharacteristics mcIn, boolean outputEmptyBlocks)`
`static void` `copyRowBlocks​(MatrixBlock mb, int rowIndex, MatrixBlock ret, int numRowsPerBlock, int rlen, int clen)`
`static void` `copyRowBlocks​(MatrixBlock mb, int rowIndex, MatrixBlock ret, long numRowsPerBlock, long rlen, long clen)`
`static void` `copyRowBlocks​(MatrixBlock mb, long rowIndex, MatrixBlock ret, int numRowsPerBlock, int rlen, int clen)`
`static void` `copyRowBlocks​(MatrixBlock mb, long rowIndex, MatrixBlock ret, long numRowsPerBlock, long rlen, long clen)`
`static void` `postProcessAfterCopying​(MatrixBlock ret)`
`static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>` `projectColumns​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df, ArrayList<String> columns)`
`static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>` `stringDataFrameToVectorDataFrame​(org.apache.spark.sql.SparkSession sparkSession, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> inputDF)` Convert a dataframe of comma-separated string rows to a dataframe of ml.linalg.Vector rows.

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### RDDConverterUtilsExt

    public RDDConverterUtilsExt()

### Method Detail

#### coordinateMatrixToBinaryBlock

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> coordinateMatrixToBinaryBlock​(org.apache.spark.api.java.JavaSparkContext sc,
                                                                                                                       org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
                                                                                                                       DataCharacteristics mcIn,
                                                                                                                       boolean outputEmptyBlocks)

#### coordinateMatrixToBinaryBlock

    public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,​MatrixBlock> coordinateMatrixToBinaryBlock​(org.apache.spark.SparkContext sc,
                                                                                                                       org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
                                                                                                                       DataCharacteristics mcIn,
                                                                                                                       boolean outputEmptyBlocks)

#### projectColumns

    public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> projectColumns​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
                                                                                        ArrayList<String> columns)

#### copyRowBlocks

    public static void copyRowBlocks​(MatrixBlock mb,
                                     int rowIndex,
                                     MatrixBlock ret,
                                     int numRowsPerBlock,
                                     int rlen,
                                     int clen)

#### copyRowBlocks

    public static void copyRowBlocks​(MatrixBlock mb,
                                     long rowIndex,
                                     MatrixBlock ret,
                                     int numRowsPerBlock,
                                     int rlen,
                                     int clen)

#### copyRowBlocks

    public static void copyRowBlocks​(MatrixBlock mb,
                                     int rowIndex,
                                     MatrixBlock ret,
                                     long numRowsPerBlock,
                                     long rlen,
                                     long clen)

#### copyRowBlocks

    public static void copyRowBlocks​(MatrixBlock mb,
                                     long rowIndex,
                                     MatrixBlock ret,
                                     long numRowsPerBlock,
                                     long rlen,
                                     long clen)

#### postProcessAfterCopying

    public static void postProcessAfterCopying​(MatrixBlock ret)

#### addIDToDataFrame

    public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> addIDToDataFrame​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
                                                                                          org.apache.spark.sql.SparkSession sparkSession,
                                                                                          String nameOfCol)

Add element indices as new column to DataFrame

Parameters:
  `df` - input data frame
  `sparkSession` - the Spark Session
  `nameOfCol` - name of index column
Returns:
  new data frame

#### stringDataFrameToVectorDataFrame

    public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> stringDataFrameToVectorDataFrame​(org.apache.spark.sql.SparkSession sparkSession,
                                                                                                          org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> inputDF)

Convert a dataframe of comma-separated string rows to a dataframe of ml.linalg.Vector rows.

Example input rows:
`((1.2, 4.3, 3.4))
(1.2, 3.4, 2.2)
[[1.2, 34.3, 1.2, 1.25]]
[1.2, 3.4]
`

Parameters:
  `sparkSession` - Spark Session
  `inputDF` - dataframe of comma-separated row strings to convert to dataframe of ml.linalg.Vector rows
Returns:
  dataframe of ml.linalg.Vector rows