org.apache.sysds.runtime.controlprogram.parfor

## Class RemoteParForUtils

- java.lang.Object
- - org.apache.sysds.runtime.controlprogram.parfor.RemoteParForUtils

    public class RemoteParForUtils
    extends Object

Common functionalities for parfor workers in MR jobs. Used by worker wrappers in mappers (base RemoteParFor) and reducers (fused data partitioning and parfor)

### Constructor Summary

Constructors
Constructor and Description
`RemoteParForUtils()`

### Method Summary

All Methods Static Methods Concrete Methods
`static void` `cleanupWorkingDirectories()` Cleanup all temporary files created by this SystemDS process.
`static void` `cleanupWorkingDirectoriesOnShutdown()` Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.
`static void` `exportLineageItems(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage)` Export lineage for remote Spark parfor workers.
`static ArrayList<String>` `exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars)` For remote Spark parfor workers.
`static void` `exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, HashMap<String,String> rvarFnames, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)` For remote MR parfor workers.
`static void` `exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)`
`static Lineage[]` `getLineages(LocalVariableMap[] results)`
`static LocalVariableMap[]` `getResults(List<scala.Tuple2<Long,String>> out, org.apache.commons.logging.Log LOG)`
`static void` `incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)`
`static void` `setupBufferPool(long workerID)` Init and register-cleanup of buffer pool

### Methods inherited from class java.lang.Object

`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Constructor Detail

#### RemoteParForUtils

    public RemoteParForUtils()

### Method Detail

#### incrementParForMRCounters

    public static void incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter,
                                                 long deltaTasks,
                                                 long deltaIterations)

#### exportResultVariables

    public static void exportResultVariables(long workerID,
                                             LocalVariableMap vars,
                                             ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                             org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)
                                      throws IOException

Throws:
  `IOException`

#### exportResultVariables

    public static void exportResultVariables(long workerID,
                                             LocalVariableMap vars,
                                             ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                             HashMap<String,String> rvarFnames,
                                             org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> out)
                                      throws IOException

For remote MR parfor workers.

Parameters:
  `workerID` - worker id
  `vars` - local variable map
  `resultVars` - list of result variables
  `rvarFnames` - ?
  `out` - output collectors
Throws:
  `IOException` - if IOException occurs

#### exportResultVariables

    public static ArrayList<String> exportResultVariables(long workerID,
                                                          LocalVariableMap vars,
                                                          ArrayList<ParForStatementBlock.ResultVar> resultVars)
                                                   throws IOException

For remote Spark parfor workers. This is a simplified version compared to MR.

Parameters:
  `workerID` - worker id
  `vars` - local variable map
  `resultVars` - list of result variables
Returns:
  list of result variables
Throws:
  `IOException` - if IOException occurs

#### exportLineageItems

    public static void exportLineageItems(long workerID,
                                          LocalVariableMap vars,
                                          ArrayList<ParForStatementBlock.ResultVar> resultVars,
                                          Lineage lineage)
                                   throws IOException

Export lineage for remote Spark parfor workers.

Parameters:
  `workerID` - worker id
  `vars` - local variable map
  `resultVars` - list of result variables
  `lineage` - lineage object
Throws:
  `IOException` - if IOException occurs

#### cleanupWorkingDirectories

    public static void cleanupWorkingDirectories()

Cleanup all temporary files created by this SystemDS process.

#### cleanupWorkingDirectoriesOnShutdown

    public static void cleanupWorkingDirectoriesOnShutdown()

Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.

#### getLineages

    public static Lineage[] getLineages(LocalVariableMap[] results)

#### getResults

    public static LocalVariableMap[] getResults(List<scala.Tuple2<Long,String>> out,
                                                org.apache.commons.logging.Log LOG)

#### setupBufferPool

    public static void setupBufferPool(long workerID)
                                throws IOException

Init and register-cleanup of buffer pool

Parameters:
  `workerID` - worker id
Throws:
  `IOException` - exception