# Package org.apache.sysds.runtime.controlprogram.parfor

Class Summary
CachedReuseVariables
DataPartitioner: This is the base class for all data partitioner.
DataPartitionerLocal: Partitions a given matrix into row or column partitions with a two pass-approach.
DataPartitionerRemoteSpark: MR job class for submitting parfor remote partitioning MR jobs.
DataPartitionerRemoteSparkMapper: NOTE: for the moment we only support binary block here TODO extend impl for binarycell and textcell Interface of Writable output in order to support both PairWritableBlock and PairWritableCell.
DataPartitionerRemoteSparkReducer
LocalParWorker: Instances of this class can be used to execute tasks in parallel.
LocalTaskQueue<T>: This class provides a way of dynamic task distribution to multiple workers in local multi-threaded environments.
ParForBody: Wrapper for exchanging parfor body data structures.
ParWorker: Super class for master/worker pattern implementations.
RemoteDPParForSpark: TODO heavy hitter maintenance TODO data partitioning with binarycell
RemoteDPParForSparkWorker
RemoteParForJobReturn: Wrapper for job return of ParFor REMOTE for transferring statistics and result symbol table.
RemoteParForSpark: This class serves two purposes: (1) isolating Spark imports to enable running in environments where no Spark libraries are available, and (2) to follow the same structure as the parfor remote_mr job submission.
RemoteParForSparkWorker
RemoteParForUtils: Common functionalities for parfor workers in MR jobs.
ResultMerge<T extends CacheableData<?>>
ResultMergeFrameLocalMemory
ResultMergeLocalAutomatic
ResultMergeLocalFile: TODO potential extension: parallel merge (create individual staging files concurrently) NOTE: file merge typically used due to memory constraints - parallel merge would increase the memory consumption again.
ResultMergeLocalMemory: Local in-memory realization of result merge.
ResultMergeMatrix: Due to independence of all iterations, any result has the following properties:
ResultMergeRemoteGrouping
ResultMergeRemoteSorting
ResultMergeRemoteSpark
ResultMergeRemoteSparkWCompare
ResultMergeTaggedMatrixIndexes: This class serves as composite key for the remote result merge job (for any data format) in order to sort on both matrix indexes and tag but group all blocks according to matrix indexes only.
Task: A task is a logical group of one or multiple iterations (each iteration is assigned to exactly one task).
TaskPartitioner: This is the base class for all task partitioner.
TaskPartitionerFactoring: This factoring task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks.
TaskPartitionerFactoringCmax: Factoring with maximum constraint (e.g., if LIX matrix out-of-core and we need to bound the maximum number of iterations per map task -> memory bounds)
TaskPartitionerFactoringCmin: Factoring with minimum constraint (e.g., if communication is expensive)
TaskPartitionerFixedsize: This naive task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to the given task size.
TaskPartitionerNaive: This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.
TaskPartitionerStatic: This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.

Enum Summary
Task.TaskType