# All Classes

All Classes Interface Summary Class Summary Enum Summary Exception Summary
ABitmap
ABooleanArray
AColGroup: Abstract Class that is the lowest class type for the Compression framework.
AColGroup.CompressionType: Public super types of compression ColGroups supported
AColGroupCompressed: Base class for column groups encoded Encoded in a compressed manner.
AColGroupOffset: Base class for column groups encoded with various types of bitmap encoding.
AColGroupValue
AColIndex
AColumnCoCoder
AComEst: Main abstract class for estimating size of compressions on columns.
ACostEstimate: A cost estimator interface.
ADictBasedColGroup
ADictionary: This dictionary class aims to encapsulate the storage and operations over unique tuple values of a column group.
ADictionary.DictType
AFederatedPlanner
AggBinaryOp
AggBinaryOp.MMultMethod
AggBinaryOp.SparkAggType
AggregateBinaryCPInstruction
AggregateBinaryFEDInstruction
AggregateBinaryGPUInstruction
AggregateBinaryOperator
AggregateBinarySPInstruction: Class to group the different MM `SPInstruction`s together.
AggregateDropCorrectionFunction
AggregateOperator
AggregateTernaryCPInstruction
AggregateTernaryFEDInstruction
AggregateTernaryOperator
AggregateTernarySPInstruction
AggregateUnaryCPInstruction
AggregateUnaryCPInstruction.AUType
AggregateUnaryFEDInstruction
AggregateUnaryGPUInstruction: Implements aggregate unary instructions for CUDA
AggregateUnaryOperator
AggregateUnarySketchSPInstruction
AggregateUnarySPInstruction
AggregateUnarySPInstruction.RDDUAggFunction2: Similar to RDDUAggFunction but single output block.
AggregateUnarySPInstruction.RDDUTensorAggFunction2: Similar to RDDUAggFunction but single output block.
AggUnaryOp
AInsertionSorter: This abstract class is for sorting the IntArrayList entries efficiently for SDC Column Groups construction.
AIterator: Iterator interface, that returns a iterator of the indexes while also maintaining a data index.
AMapToData: This Class's job is to link into the dictionary entries for column groups.
AMapToDataGroup
AMorphingMMColGroup: Abstract class for column group types that do not perform matrix Multiplication, and decompression for performance reasons but instead transforms into another type of column group type to perform that operation.
And
AOffset: Offset list encoder interface.
AOffset.OffsetSliceInfo
AOffsetIterator: Iterator interface that only iterate through offsets without considering data index.
AOffsetsGroup
Append
AppendCPInstruction
AppendCPInstruction.AppendType
AppendFEDInstruction
AppendG: TODO Additional compiler enhancements: 1) Partial Shuffle Elimination - Any full or aligned blocks could be directly output from the mappers to the result index.
AppendGAlignedSP
AppendGAlignedSPInstruction
AppendGAlignedSPInstruction.ShiftColumnIndex
AppendGSPInstruction
AppendGSPInstruction.MergeWithShiftedBlocks
AppendGSPInstruction.ShiftMatrix
AppendM
AppendM.CacheType
AppendMSPInstruction
AppendPropagator
AppendR
AppendRSPInstruction
AppendSPInstruction: Class to group the different append `SPInstruction`s together.
APreAgg: Abstract class for all the column groups that use preAggregation for Left matrix multiplications.
ArithmeticBinaryGPUInstruction
Array<T>: Generic, resizable native arrays for the internal representation of the columns in the FrameBlock.
ArrayFactory
ArrayFactory.FrameArrayType
ArrayIndex
ASDC: Column group that sparsely encodes the dictionary values.
ASDCZero
AssignmentStatement
AutoDiff
AWTreeNode: A workload tree is a compact representation of the operations on a matrix and derived intermediates, including the basic control structure and inlined functions as well as links to categories.
AWTreeNode.WTNodeType
BalanceToAvgFederatedScheme: Balance to Avg Federated scheme When the parameter server runs in federated mode it cannot pull in the data which is already on the workers.
BaseModel
BasicProgramBlock
BasicTensorBlock
Binary
Binary
Binary: Lop to perform binary operation.
BinaryBlockInputFormat: Custom binary block input format to return the custom record reader.
BinaryBlockRecordReader: Custom record reader for binary block.
BinaryBlockSerialization: This custom serialization class can be used via job.set(MRConfigurationNames.IO_SERIALIZATIONS, "org.apache.sysds.runtime.io.BinaryBlockSerialization");
BinaryBlockSerialization.MatrixBlockDeserializer
BinaryBlockSerialization.MatrixBlockSerializer
BinaryBlockToTextCellConverter
BinaryCPInstruction
BinaryExpression
BinaryFEDInstruction
BinaryFrameFrameCPInstruction
BinaryFrameFrameSPInstruction
BinaryFrameMatrixCPInstruction
BinaryFrameMatrixSPInstruction
BinaryM: Lop to perform binary operation.
BinaryM.VectorType
BinaryMatrixBVectorSPInstruction
BinaryMatrixMatrixCPInstruction
BinaryMatrixMatrixFEDInstruction
BinaryMatrixMatrixSPInstruction
BinaryMatrixScalarCPInstruction
BinaryMatrixScalarFEDInstruction
BinaryMatrixScalarSPInstruction
BinaryOp
BinaryOp.AppendMethod
BinaryOp.MMBinaryMethod
BinaryOperator
BinaryScalar: Lop to perform binary scalar operations.
BinaryScalarScalarCPInstruction
BinarySPInstruction
BinaryTensorTensorBroadcastSPInstruction
BinaryTensorTensorCPInstruction
BinaryTensorTensorSPInstruction
BinaryUAggChain
BinUaggChainSPInstruction: Instruction that performs res = X / rowsum(x)
BinUaggChainSPInstruction.RDDBinUaggChainFunction
Bitmap: Uncompressed representation of one or more columns in bitmap format.
BitmapEncoder: Static functions for extracting bitmaps from a MatrixBlock.
BitMapValueCombiner
BitSetArray
BitwAnd
BitwOr
BitwShiftL
BitwShiftR
BitwXor
Block
BlockPartitioner: Default partitioner used for all binary block rdd operations in order to enable sufficient local aggregation independent of the aggregation direction (row/col-wise).
BooleanArray
BooleanArray32
BooleanExpression
BooleanIdentifier
BooleanObject
BroadcastCPInstruction
BroadcastObject<T extends CacheBlock<?>>
Builtin: Class with pre-defined set of objects.
Builtin.BuiltinCode
BuiltinBinaryGPUInstruction
BuiltinConstant: These are the builtin constants
BuiltinFunctionExpression
BuiltinNaryCPInstruction: Instruction to handle a variable number of input operands.
BuiltinNarySPInstruction
BuiltinNarySPInstruction.PadBlocksFunction
Builtins: Enum to represent all builtin functions in the default name space.
BuiltinUnaryGPUInstruction
ByteBuffer: Wrapper for WriteBuffer byte array per matrix/frame in order to support matrix/frame serialization outside global lock.
ByteBufferDataInput
ByteClassLoader
CacheableData<T extends CacheBlock<?>>: Each object of this class is a cache envelope for some large piece of data called "cache block".
CacheableData.CacheStatus: Defines all possible cache status types for a data blob.
CacheBlock<T>: Interface for all blocks handled by lazy write buffer.
CacheBlockFactory: Factory to create instances of matrix/frame blocks given internal codes.
CacheDataInput
CacheDataOutput: Custom DataOutput to serialize directly into the given byte array.
CachedReuseVariables
CacheEvictionQueue
CacheMaintenanceService
CacheStatistics: This singleton provides basic caching statistics in CP.
CacheStatistics.Stat
CastFEDInstruction
CastSPInstruction
CBindPropagator
Cell: Helper class for representing text cell and binary cell records in order to allow for buffering and buffered read/write.
CentralMoment: Lop to perform cross product operation
CentralMomentCPInstruction
CentralMomentFEDInstruction
CentralMomentSPInstruction
CharArray
CheckedConstraintsLog: Class counting the checked privacy constraints and the loaded privacy constraints.
Checkpoint: Lop for checkpoint operations.
CheckpointSPInstruction
CiphertextMatrix: This class abstracts over an encrypted matrix of ciphertexts.
CLALibAppend
CLALibBinaryCellOp
CLALibCMOps
CLALibCombine
CLALibCompAgg
CLALibDecompress: Library to decompress a list of column groups into a matrix.
CLALibLeftMultBy
CLALibMatrixMult
CLALibMMChain
CLALibRexpand
CLALibRightMultBy
CLALibScalar
CLALibSlice
CLALibSquash
CLALibTSMM
CLALibUnary
CLALibUtils
CM: GENERAL NOTE: * 05/28/2014: We decided to do handle weights consistently to SPSS in an operation-specific manner, i.e., we (1) round instead of casting where required (e.g.
CM_COV_Object
CM_N_COVCell
CMOperator
CMOperator.AggregateOperationTypes
CNode
CNodeBinary
CNodeBinary.BinType
CNodeCell
CNodeData
CNodeMultiAgg
CNodeNary
CNodeNary.NaryType
CNodeOuterProduct
CNodeRow
CNodeTernary
CNodeTernary.TernaryType
CNodeTpl
CNodeUnary
CNodeUnary.UnaryType
CoCodeBinPacking: Column group partitioning with bin packing heuristic.
CoCodeGreedy
CoCodeHybrid: This cocode strategy starts combines the use of CoCodePriorityQue and CoCodeGreedy.
CoCodePriorityQue: Column group partitioning by number of distinct items estimated.
CoCoderFactory
CoCoderFactory.PartitionerType: The Valid coCoding techniques
CoCodeStatic: Column group co coding with static distribution heuristic.
CodegenStatistics
CodegenUtils
CodeTemplate
ColGroupConst
ColGroupDDC: Class to encapsulate information about a column group that is encoded with dense dictionary encoding (DDC).
ColGroupDDCFOR: Class to encapsulate information about a column group that is encoded with dense dictionary encoding (DDC).
ColGroupDeltaDDC: Class to encapsulate information about a column group that is first delta encoded then encoded with dense dictionary encoding (DeltaDDC).
ColGroupEmpty
ColGroupFactory: Factory class for constructing ColGroups.
ColGroupIO: IO for ColGroups, it enables read and write ColGroups
ColGroupLinearFunctional
ColGroupOLE: Class to encapsulate information about a column group that is encoded with simple lists of offsets for each set of distinct values.
ColGroupRLE: A group of columns compressed with a single run-length encoded bitmap.
ColGroupSDC: Column group that sparsely encodes the dictionary values.
ColGroupSDCFOR: ColGroup for Patched Frame Of Reference.
ColGroupSDCSingle: Column group that sparsely encodes the dictionary values.
ColGroupSDCSingleZeros: Column group that sparsely encodes the dictionary values.
ColGroupSDCZeros: Column group that sparsely encodes the dictionary values.
ColGroupSizes
ColGroupUncompressed: Column group type for columns that are stored as dense arrays of doubles.
ColGroupUtils
ColIndexes
ColIndexFactory
CollectionUtils
ColumnEncoder: Base class for all transform encoders providing both a row and block interface for decoding frames to matrices.
ColumnEncoder.EncoderType
ColumnEncoderBin
ColumnEncoderBin.BinMethod
ColumnEncoderComposite: Simple composite encoder that applies a list of encoders in specified order.
ColumnEncoderDummycode
ColumnEncoderFeatureHash: Class used for feature hashing transformation of frames.
ColumnEncoderFeatureHash.FeatureHashSparseApplyTask
ColumnEncoderPassThrough
ColumnEncoderPassThrough.PassThroughSparseApplyTask
ColumnEncoderRecode
ColumnEncoderUDF
ColumnMetadata
ComEstExact: Exact compressed size estimator (examines entire dataset).
ComEstFactory
ComEstSample: Estimate compression size based on subsample of data.
CommonThreadPool: This common thread pool provides an abstraction to obtain a shared thread pool, specifically the ForkJoinPool.commonPool, for all requests of the maximum degree of parallelism.
CompilerConfig: Basic wrapper for all compiler configurations that are configured dynamically on a per script invocation basis.
CompilerConfig.ConfigType
CompressedEncode
CompressedMatrixBlock
CompressedMatrixBlockFactory: Factory pattern to compress a Matrix Block into a CompressedMatrixBlock.
CompressedSizeInfo: A helper reusable object for maintaining information about estimated compression
CompressedSizeInfoColGroup: Information collected about a specific ColGroup's compression size.
CompressedWriteBlock: Write block for serializing either a instance of MatrixBlock or CompressedMatrixBlock, To allow spark to read in either or.
Compression
Compression.CompressConfig
CompressionCPInstruction
CompressionSettings: Compression Settings class, used as a bundle of parameters inside the Compression framework.
CompressionSettingsBuilder: Builder pattern for Compression Settings.
CompressionSPInstruction
CompressionSPInstruction.CompressionFunction
CompressionSPInstruction.CompressionWorkloadFunction
CompressionSPInstruction.SizeFunction
CompressionStatistics: Compression Statistics contain the main information gathered from the compression, such as sizes of the original matrix, vs the compressed representation at different stages of the compression.
CompressUnwrap
CompressWrap
ComputationCostEstimator
ComputationCPInstruction
ComputationFEDInstruction
ComputationSPInstruction
ComputeBinaryBlockNnzFunction
ComputeCost: Class with methods estimating compute costs of operations.
ConditionalPredicate
ConfigurableAPI: This interface defines the programmatic access to dml configuration properties (as defined in SystemDS-config.xml) to ensure API consistency across all APIs.
ConfigurationManager: Singleton for accessing the parsed and merged system configuration.
Connection: Interaction with SystemDS using the JMLC (Java Machine Learning Connector) API is initiated with a `Connection` object.
Constants
Constants
ConstEncoding: Const encoding for cases where the entire group of columns is the same value
ConstIdentifier
ConstScheme
Converter<K1 extends org.apache.hadoop.io.Writable,​V1 extends org.apache.hadoop.io.Writable,​K2 extends org.apache.hadoop.io.Writable,​V2 extends org.apache.hadoop.io.Writable>
ConvertFrameBlockToIJVLines
ConvertMatrixBlockToIJVLines
ConvertStringToLongTextPair
CoordinatorConnectionModel
CoordinatorController
CoordinatorModel
CoordinatorService
CopyBinaryCellFunction
CopyFrameBlockFunction: General purpose copy function for binary block rdds.
CopyFrameBlockPairFunction: General purpose copy function for binary block rdds.
CopyMatrixBlockFunction: General purpose copy function for binary block rdds.
CopyMatrixBlockPairFunction: General purpose copy function for binary block rdds.
CopyTensorBlockFunction: General purpose copy function for binary block rdds.
CopyTensorBlockPairFunction: General purpose copy function for binary block rdds.
CopyTextInputFunction
CorrMatrixBlock
CostEstimationWrapper
CostEstimationWrapper.CostType
CostEstimator
CostEstimator: Base class for all potential cost estimators TODO account for shared read-only matrices when computing aggregated stats
CostEstimator.DataFormat
CostEstimator.ExcludeType
CostEstimator.TestMeasure
CostEstimatorBuilder
CostEstimatorFactory: Factory class for the construction of cost estimators for compression
CostEstimatorFactory.CostType
CostEstimatorHops
CostEstimatorRuntime: Cost estimator for runtime programs.
CostEstimatorStaticRuntime
CountDistinctFunctionSketch
CountDistinctOperator
CountDistinctOperatorTypes
CountDistinctSketch
COV: GENERAL NOTE: * 05/28/2014: We decided to do handle weights consistently to SPSS in an operation-specific manner, i.e., we (1) round instead of casting where required (e.g.
CoVariance: Lop to compute covariance between two 1D matrices
CovarianceCPInstruction
CovarianceFEDInstruction
CovarianceSPInstruction
COVOperator
CPInstruction
CPInstruction.CPType
CPInstructionParser
CPlanCSERewriter
CPlanMemoTable
CPlanMemoTable.MemoTableEntry
CPlanMemoTable.MemoTableEntrySet
CPlanOpRewriter: This cplan rewriter is meant to be the central place for any cplan enhancements before code generation.
CpmmSPInstruction: Cpmm: cross-product matrix multiplication operation (distributed matrix multiply by join over common dimension and subsequent aggregation of partial results).
CPOperand
CreateSparseBlockFunction: General purpose copy function for binary block values.
CSRPointer: Compressed Sparse Row (CSR) format for CUDA Generalized matrix multiply is implemented for CSR format in the cuSparse library among other operations Since we assume that the matrix is stored with zero-based indexing (i.e.
CSVReBlock: Lop to convert CSV data into SystemDS data format
CSVReblockSPInstruction
Ctable: Lop to perform ternary operation.
CTable
Ctable.OperationTypes
CtableCPInstruction
CtableFEDInstruction
CTableMap: Ctable map is an abstraction for the hashmap used for ctable's hash group-by because this structure is passed through various interfaces.
CtableSPInstruction
CtableSPInstruction.MapJoinSignature1
CtableSPInstruction.MapJoinSignature2
CtableSPInstruction.MapJoinSignature3
CudaMemoryAllocator
CudaSupportFunctions: DESIGN DOCUMENTATION FOR SUPPORTING LOWER PRECISION: 1.
CumulativeAggregateSPInstruction
CumulativeOffsetBinary
CumulativeOffsetFEDInstruction
CumulativeOffsetSPInstruction
CumulativePartialAggregate
CustomErrorListener
CustomErrorListener.ParseIssueType: Parse issues can be syntax errors, validation errors, and validation warnings.
CustomProperties
Dag<N extends Lop>: Class to maintain a DAG of lops and compile it into runtime instructions, incl piggybacking into jobs.
DArrCounts
Data: Lop to represent data objects.
Data
DataAugmentation
DataCharacteristics
DataConverter: This class provides methods to read and write matrix blocks from to HDFS using different data formats.
DataExpression
DataGen: Defines a LOP that generates data.
DataGenCPInstruction
DataGenOp: A DataGenOp can be rand (or matrix constructor), sequence, and sample - these operators have different parameters and use a map of parameter type to hop position.
DataIdentifier
DataObjectModel
DataOp: A DataOp can be either a persistent read/write or transient read/write - writes will always have at least one input, but all types can have parameters (e.g., for csv literals of delimiter, header, etc).
DataPartitioner: This is the base class for all data partitioner.
DataPartitionerLocal: Partitions a given matrix into row or column partitions with a two pass-approach.
DataPartitionerRemoteSpark: MR job class for submitting parfor remote partitioning MR jobs.
DataPartitionerRemoteSparkMapper: NOTE: for the moment we only support binary block here TODO extend impl for binarycell and textcell Interface of Writable output in order to support both PairWritableBlock and PairWritableCell.
DataPartitionerRemoteSparkReducer
DataPartitionerSparkAggregator
DataPartitionerSparkMapper
DataPartitionFederatedScheme
DataPartitionFederatedScheme.BalanceMetrics
DataPartitionFederatedScheme.Result
DataPartitionLocalScheme
DataPartitionSparkScheme
DataRange: A DataRange instance marks a part of a CacheableData data object.
DatasetObject
DataTensorBlock
DblArray: Helper class used for bitmap extraction.
DblArrayCountHashMap
DblArrayIntListHashMap
DblArrayIntListHashMap.DArrayIListEntry
DCLocalScheme: Disjoint_Contiguous data partitioner: for each worker, use a right indexing operation X[beg:end,] to obtain contiguous, non-overlapping partitions of rows.
DCounts
DCSparkScheme: Spark Disjoint_Contiguous data partitioner:
DDCScheme
Decoder: Base class for all transform decoders providing both a row and block interface for decoding matrices to frames.
DecoderComposite: Simple composite decoder that applies a list of decoders in specified order.
DecoderDummycode: Simple atomic decoder for dummycoded columns.
DecoderFactory
DecoderFactory.DecoderType
DecoderPassThrough: Simple atomic decoder for passing through numeric columns to the output.
DecoderRecode: Simple atomic decoder for recoded columns.
DeCompression
DeCompression.CompressConfig
DeCompressionCPInstruction
DeCompressionSPInstruction
DeCompressionSPInstruction.DeCompressionFunction
DeltaDictionary: This dictionary class is a specialization for the DeltaDDCColgroup.
DenseBlock: This DenseBlock is an abstraction for different dense, row-major matrix formats.
DenseBlock.Type
DenseBlockBool
DenseBlockDRB
DenseBlockFactory
DenseBlockFP32
DenseBlockFP64
DenseBlockInt32
DenseBlockInt64
DenseBlockLBool
DenseBlockLDRB: Dense Large Row Blocks have multiple 1D arrays (blocks), which contain complete rows.
DenseBlockLFP32
DenseBlockLFP64
DenseBlockLInt32
DenseBlockLInt64
DenseBlockLString
DenseBlockString
DenseEncoding
DependencyTask<E>
DependencyThreadPool
DependencyWrapperTask<E>
DerbyRepository
DiagIndex
Dictionary: This dictionary class aims to encapsulate the storage and operations over unique floating point values of a column group.
DictionaryFactory
DictionaryFactory.Type
DictLibMatrixMult: Utility interface for dictionary matrix multiplication
DistinctCostEstimator: A Cost based estimator that based the compression and co-coding cost on the number of distinct elements in the column groups.
Divide
DmlBaseListener: This class provides an empty implementation of `DmlListener`, which can be extended to create a listener which only needs to handle a subset of the available methods.
DMLCompressionException
DMLCompressionStatistics
DMLConfig
DMLException: Exception occurring in the DML framework.
DmlLexer
DmlListener: This interface defines a complete listener for a parse tree produced by `DmlParser`.
DMLOptions: Set of DMLOptions that can be set through the command line and `MLContext` The values have been initialized with the default values Despite there being a DML and PyDML, this class is named DMLOptions to keep it consistent with `DMLOptions` and `DMLOptions`
DmlParser
DmlParser.AccumulatorAssignmentStatementContext
DmlParser.AddSubExpressionContext
DmlParser.AssignmentStatementContext
DmlParser.AtomicExpressionContext
DmlParser.BooleanAndExpressionContext
DmlParser.BooleanNotExpressionContext
DmlParser.BooleanOrExpressionContext
DmlParser.BuiltinFunctionExpressionContext
DmlParser.CommandlineParamExpressionContext
DmlParser.CommandlinePositionExpressionContext
DmlParser.ConstDoubleIdExpressionContext
DmlParser.ConstFalseExpressionContext
DmlParser.ConstIntIdExpressionContext
DmlParser.ConstStringIdExpressionContext
DmlParser.ConstTrueExpressionContext
DmlParser.DataIdentifierContext
DmlParser.DataIdExpressionContext
DmlParser.DataTypeContext
DmlParser.ExpressionContext
DmlParser.ExternalFunctionDefExpressionContext
DmlParser.ForStatementContext
DmlParser.FunctionCallAssignmentStatementContext
DmlParser.FunctionCallMultiAssignmentStatementContext
DmlParser.FunctionStatementContext
DmlParser.IfdefAssignmentStatementContext
DmlParser.IfStatementContext
DmlParser.ImportStatementContext
DmlParser.IndexedExpressionContext
DmlParser.InternalFunctionDefExpressionContext
DmlParser.IterablePredicateColonExpressionContext
DmlParser.IterablePredicateContext
DmlParser.IterablePredicateSeqExpressionContext
DmlParser.MatrixDataTypeCheckContext
DmlParser.MatrixMulExpressionContext
DmlParser.Ml_typeContext
DmlParser.ModIntDivExpressionContext
DmlParser.MultDivExpressionContext
DmlParser.MultiIdExpressionContext
DmlParser.ParameterizedExpressionContext
DmlParser.ParForStatementContext
DmlParser.PathStatementContext
DmlParser.PowerExpressionContext
DmlParser.ProgramrootContext
DmlParser.RelationalExpressionContext
DmlParser.SimpleDataIdentifierExpressionContext
DmlParser.StatementContext
DmlParser.StrictParameterizedExpressionContext
DmlParser.StrictParameterizedKeyValueStringContext
DmlParser.TypedArgAssignContext
DmlParser.TypedArgNoAssignContext
DmlParser.UnaryExpressionContext
DmlParser.ValueTypeContext
DmlParser.WhileStatementContext
DMLParserWrapper: This is the main entry point for the Antlr4 parser.
DmlPreprocessor: Minimal pre-processing of user function definitions which take precedence over built-in functions in cases where names conflict.
DMLPrivacyException: This exception should be thrown to flag DML runtime errors related to the violation of privacy constraints.
DMLProgram
DMLRuntimeException: This exception should be thrown to flag runtime errors -- DML equivalent to java.lang.RuntimeException.
DMLScript
DMLScriptException: This exception should be thrown to flag DML Script errors.
DmlSyntacticValidator
DmlSyntacticValidator.ConvertedDMLSyntax: Information about built in functions converted to a common format between PyDML and DML for the runtime.
DMLTranslator
DMVUtils
DMVUtils.LEVEL_ENUM
DnnCPInstruction
DnnGPUInstruction
DnnOp
DnnParameters: This class is container that stores parameters required for executing following operations: conv2d, conv2d_backward_data, conv2d_backward_filter, maxpooling, maxpooling_backward
DnnSPInstruction
DnnTransform
DnnUtils
DocumentRepresentation
DoubleArray
DoubleCountHashMap
DoubleIdentifier
DoubleIntListHashMap: This class provides a memory-efficient replacement for `HashMap<Double,IntArrayList>` for restricted use cases.
DoubleIntListHashMap.DIListEntry
DoubleObject
DoublePrecisionCudaSupportFunctions
DRLocalScheme: Data partitioner Disjoint_Random: for each worker, use a permutation multiply P[beg:end,] %*% X, where P is constructed for example with P=table(seq(1,nrow(X)),sample(nrow(X), nrow(X))), i.e., sampling without replacement to ensure disjointness.
DRRLocalScheme: Disjoint_Round_Robin data partitioner: for each worker, use a permutation multiply or simpler a removeEmpty such as removeEmpty (target=X, margin=rows, select=(seq(1,nrow(X))%%k)==id)
DRRSparkScheme: Spark Disjoint_Round_Robin data partitioner:
DRSparkScheme: Spark data partitioner Disjoint_Random: For the current row block, find all the shifted place for each row (WorkerID => (row block ID, matrix)
EMAUtils
EmptyEncoding: Empty encoding for cases where the entire group of columns is zero
EmptyScheme
Encoder
EncoderFactory
EncoderMVImpute
EncoderMVImpute.MVMethod
EncoderOmit
EncodingFactory
Encrypted: This class abstracts over an encrypted data.
EnvironmentHelper: This class is useful in setting environment variable for loading MKL library (done by Native Helper)
Equals
EstimationFactors: Compressed Size Estimation factors.
EstimationUtils
EstimatorBasicAvg: Basic average case estimator for matrix sparsity: sp = 1 - Math.pow(1-sp1*sp2, k)
EstimatorBasicWorst: Basic average case estimator for matrix sparsity: sp = Math.min(1, sp1 * k) * Math.min(1, sp2 * k).
EstimatorBitsetMM: This estimator implements a naive but rather common approach of boolean matrix multiplies which allows to infer the exact non-zero structure and thus is also useful for sparse result preallocation.
EstimatorBitsetMM.BitsetMatrix
EstimatorBitsetMM.BitsetMatrix1: This class represents a boolean matrix and provides key operations.
EstimatorBitsetMM.BitsetMatrix2
EstimatorDensityMap: This estimator implements an approach called density maps, as introduced in David Kernert, Frank Köhler, Wolfgang Lehner: SpMacho - Optimizing Sparse Linear Algebra Expressions with Probabilistic Density Estimation.
EstimatorDensityMap.DensityMap
EstimatorLayeredGraph: This estimator implements an approach based on a so-called layered graph, introduced in Edith Cohen.
EstimatorLayeredGraph.LayeredGraph
EstimatorMatrixHistogram: This estimator implements a remarkably simple yet effective approach for incorporating structural properties into sparsity estimation.
EstimatorMatrixHistogram.MatrixHistogram
EstimatorSample: This estimator implements an approach based on row/column sampling Yongyang Yu, MingJie Tang, Walid G.
EstimatorSampleRa: This estimator implements an approach based on row/column sampling Rasmus Resen Amossen, Andrea Campagna, Rasmus Pagh: Better Size Estimation for Sparse Matrix Products.
EvalNaryCPInstruction: Eval built-in function instruction Note: it supports only single matrix[double] output
EventModel
EventStageModel
ExecutionConfig: Java Wrapper to specify CUDA execution configuration for launching custom kernels
ExecutionContext
ExecutionContextFactory
ExecutionContextMap
Explain
Explain.ExplainCounts
Explain.ExplainType
Expression
Expression.BinaryOp: Binary operators.
Expression.BooleanOp: Boolean operators.
Expression.DataOp: Data operators.
Expression.FunctCallOp: Function call operators.
Expression.RelationalOp: Relational operators.
ExpressionInfo: This class exists solely to prevent compiler warnings.
ExpressionList
ExtractBlockForBinaryReblock
ExtractGroup
ExtractGroup.ExtractGroupBroadcast
ExtractGroup.ExtractGroupJoin
ExtractGroupNWeights
FastBufferedDataInputStream
FastBufferedDataOutputStream: This buffered output stream is essentially a merged version of BufferedOutputStream and DataOutputStream, wrt SystemDS requirements.
FastStringTokenizer
FastStringTokenizer: This string tokenizer is essentially a simplified StringTokenizer.
Federated
FederatedCompilationTimer
FederatedCompilationTimer.TimeEntry
FederatedCost: Class storing execution cost estimates for federated executions with cost estimates split into different categories such as compute, read, and transfer cost.
FederatedCostEstimator: Cost estimator for federated executions with methods and constants for going through DML programs to estimate costs.
FederatedData
FederatedData.FederatedRequestEncoder
FederatedDataPartitioner
FederatedLocalData
FederatedLookupTable: Lookup table mapping from a FedUniqueCoordID (funCID) to an ExecutionContextMap (ECM) so that every coordinator can address federated variables with its own local sequential variable IDs.
FederatedMonitoringServer
FederatedMonitoringServerHandler
FederatedPlannerCostbased
FederatedPlannerFedAll: Baseline federated planner that compiles all hops that support federated execution on federated inputs to forced federated operations.
FederatedPlannerFedHeuristic
FederatedPlannerUtils: Utility class for federated planners.
FederatedPSControlThread
FederatedRange
FederatedReadCache
FederatedReadCache.ReadCacheEntry: Class representing an entry of the federated read cache.
FederatedRequest
FederatedRequest.RequestType
FederatedResponse
FederatedResponse.ResponseType
FederatedStatistics
FederatedStatistics.FedStatsCollectFunction
FederatedStatistics.FedStatsCollection
FederatedUDF
FederatedWorker
FederatedWorker.FederatedResponseEncoder
FederatedWorkerHandler: Note: federated worker handler created for every command; and concurrent parfor threads at coordinator need separate execution contexts at the federated sites too
FederatedWorkerHandlerException: Exception to throw when an exception occurs in FederatedWorkerHandler during handling of FederatedRequest.
FederatedWorkloadAnalyzer
FederationMap
FederationUtils
FEDInstruction
FEDInstruction.FederatedOutput
FEDInstruction.FEDType
FEDInstructionParser
FEDInstructionUtils
FileFormatProperties
FileFormatPropertiesCSV
FileFormatPropertiesHDF5
FileFormatPropertiesLIBSVM
FileFormatPropertiesMM
FileFormatPropertiesMM.MMField
FileFormatPropertiesMM.MMFormat
FileFormatPropertiesMM.MMSymmetry
FilterDiagMatrixBlocksFunction
FilterNonEmptyBlocksFunction
FilterNonEmptyBlocksFunction2
FineGrainedPrivacy
FineGrainedPrivacyList: Simple implementation of retrieving fine-grained privacy constraints based on pairs in an ArrayList.
FineGrainedPrivacyMap: Simple implementation of retrieving fine-grained privacy constraints based on iterating a LinkedHashMap.
FloatArray
ForProgramBlock
ForStatement
ForStatementBlock
Frame: Frame encapsulates a SystemDS frame.
FrameAppendCPInstruction
FrameAppendMSPInstruction
FrameAppendRSPInstruction
FrameBlock
FrameBlock.FrameMapFunction
FrameCompressionStatistics
FrameFormat: FrameFormat represents the different frame formats supported by the MLContext API.
FrameFromMatrixBlock
FrameGenerateReader
FrameGenerateReader.FrameReaderRowIrregular
FrameGenerateReader.FrameReaderRowRegularColIrregular
FrameGenerateReader.FrameReaderRowRegularColRegular
FrameIndexingCPInstruction
FrameIndexingSPInstruction: This class implements the frame indexing functionality inside Spark.
FrameLibAppend
FrameLibApplySchema
FrameLibCompress
FrameLibDetectSchema
FrameLibRemoveEmpty
FrameMetadata: Frame metadata, such as the number of rows, the number of columns, the number of non-zero values, the number of rows per block, and the number of columns per block in the frame.
FrameObject
FrameRDDAggregateUtils
FrameRDDConverterUtils
FrameRDDConverterUtils.LongFrameToLongWritableFrameFunction
FrameRDDConverterUtils.LongWritableFrameToLongFrameFunction
FrameRDDConverterUtils.LongWritableTextToLongTextFunction
FrameRDDConverterUtils.LongWritableToSerFunction
FrameReader: Base class for all format-specific frame readers.
FrameReaderBinaryBlock: Single-threaded frame binary block reader.
FrameReaderBinaryBlockParallel: Multi-threaded frame binary block reader.
FrameReaderFactory
FrameReaderJSONL
FrameReaderJSONLParallel
FrameReaderProto
FrameReaderTextCell: Single-threaded frame textcell reader.
FrameReaderTextCellParallel: Multi-threaded frame textcell reader.
FrameReaderTextCSV: Single-threaded frame text csv reader.
FrameReaderTextCSVParallel: Multi-threaded frame text csv reader.
FrameReblockBuffer
FrameSchema: The frame schema, stored as a list of `ValueType` values.
FrameUtil
FrameWriter: Base class for all format-specific frame writers.
FrameWriterBinaryBlock: Single-threaded frame binary block writer.
FrameWriterBinaryBlockParallel: Multi-threaded frame binary block writer.
FrameWriterCompressed
FrameWriterFactory
FrameWriterJSONL
FrameWriterJSONLParallel
FrameWriterProto
FrameWriterTextCell: Single-threaded frame text cell writer.
FrameWriterTextCellParallel: Multi-threaded frame text cell writer.
FrameWriterTextCSV: Single-threaded frame text csv writer.
FrameWriterTextCSVParallel: Single-threaded frame text csv writer.
FTypes
FTypes.AlignType
FTypes.FederatedPlanner
FTypes.FPartitioning
FTypes.FReplication
FTypes.FType
FunctionCallCP
FunctionCallCPInstruction
FunctionCallGraph
FunctionCallIdentifier
FunctionCallSizeInfo: Auxiliary data structure to hold function call summaries in terms of information about number of function calls, consistent dimensions, consistent sparsity, and dimension-preserving functions.
FunctionDictionary<T extends Types.FunctionBlock>: Dictionary of all functions of a namespace, represented as a simple key-value map of function names and function statement blocks.
FunctionObject
FunctionOp: This FunctionOp represents the call to a DML-bodied or external function.
FunctionOp.FunctionType
FunctionProgramBlock
FunctionStatement
FunctionStatementBlock
GenerateReader
GenerateReader.GenerateReaderFrame
GenerateReader.GenerateReaderMatrix
GetMIMBFromRow
GPUContext: Represents a context per GPU accessible through the same JVM.
GPUContextPool
GPUInstruction
GPUInstruction.GPUINSTRUCTION_TYPE
GPUInstructionParser
GPULazyCudaFreeMemoryManager
GPUMatrixMemoryManager
GPUMemoryAllocator
GPUMemoryEviction
GPUMemoryManager: - All cudaFree and cudaMalloc in SystemDS should go through this class to avoid OOM or incorrect results.
GPUMemoryManager.EvictionPolicyBasedComparator: Class that governs the eviction policy
GPUObject: Handle to a matrix block on the GPU
GPUStatistics: Measures performance numbers when GPU mode is enabled Printed as part of `Statistics`.
GreaterThan
GreaterThanEquals
GroupedAggregate: Lop to perform grouped aggregates
GroupedAggregateM: Lop to perform mr map-side grouped aggregates (restriction: sum, w/o weights, ngroups), groups broadcasted
GroupedAggregateM.CacheType
H5
H5BTree
H5BufferBuilder
H5Constants
H5ContiguousDataset
H5DataLayoutMessage
H5DataSpaceMessage
H5DataTypeMessage
H5DoubleDataType
H5FillValueMessage
H5GroupSymbolTableNode
H5LocalHeap
H5Message
H5NilMessage
H5ObjectHeader
H5ObjectModificationTimeMessage
H5RootObject
H5RuntimeException
H5Superblock
H5SymbolTableEntry
H5SymbolTableMessage
Hash: A class containing different hashing functions.
Hash.HashType: Available Hashing techniques
HassAndStokes
HassAndStokes.MethodOfMomentsFunction
HDFSTool
HeavyHitterModel
HEParamServer: This class implements Homomorphic Encryption (HE) for LocalParamServer.
Hop
HopDagValidator: This class allows to check hop dags for validity, e.g., parent-child linking.
HopRel: HopRel provides a representation of the relation between a hop, the cost of setting a given FederatedOutput value, and the input dependency with the given FederatedOutput value.
HopRewriteRule: Base class for all hop rewrites in order to enable generic application of all rules.
HopRewriteUtils
HopsException: Exception occurring in the HOP level.
HybridCostEstimator
ICLAScheme: Abstract class for a scheme instance.
IColIndex: Class to contain column indexes for the compression column groups.
IColIndex.ColIndexType
IColIndex.SliceResult: A Class for slice results containing indexes for the slicing of dictionaries, and the resulting column index
IController
Identifier
IdentityDictionary
IdentityDictionarySlice
IDHandler: Functionalities for extracting numeric IDs from Hadoop taskIDs and other things related to modification of IDs.
IDSequence: ID sequence for generating unique long identifiers with start 0 and increment 1.
IEncode: This interface covers an intermediate encoding for the samples to improve the efficiency of the joining of sample column groups.
IfElse
IfProgramBlock
IfStatement
IfStatementBlock
IIterate: Class to iterate through the columns of a IColIndex.
IJV: Helper class for external key/value exchange.
ILinearize: A interface for the linearization algorithms that order the DAG nodes into a sequence of instructions to execute.
ILinearize.DagLinearization
ImportStatement
IndexedIdentifier
IndexedMatrixValue
IndexedTensorBlock
IndexFunction
IndexingCPInstruction
IndexingFEDInstruction
IndexingOp
IndexingSPInstruction: This class implements the matrix indexing functionality inside Spark.
IndexRange
InfrastructureAnalyzer: Central place for analyzing and obtaining static infrastructure properties such as memory and number of logical processors.
InitFEDInstruction
InputOutputInfo
InputStreamInputFormat: Custom input format and record reader to redirect common implementation of csv read over record readers (which are required for the parallel readers) to an input stream.
InsertionSorterFactory
InsertionSorterFactory.SORT_TYPE
InstallDependencyForIntegrationTests: This class will be used to setup dependency on Eclipse environment as well as on Jenkins server
Instruction
Instruction.IType
InstructionParser
InstructionTypeCounter
InstructionUtils
IntArrayList
IntegerArray
IntegerDivide
InterestingPoint: Interesting decision point with regard to materialization of intermediates.
InterestingPoint.DecisionType
InterProceduralAnalysis: This Inter Procedural Analysis (IPA) serves two major purposes: 1) Inter-Procedure Analysis: propagate statistics from calling program into functions and back into main program.
IntIdentifier
IntObject
IOUtilFunctions
IOUtilFunctions.CountRowsTask
IPAPass: Base class for all IPA passes.
IPAPassApplyStaticAndDynamicHopRewrites: This rewrite applies static hop dag and statement block rewrites such as constant folding and branch removal in order to simplify statistic propagation.
IPAPassCompressionWorkloadAnalysis: This rewrite obtains workload summaries for all hops candidates amenable for compression as a basis for workload-aware compression planning.
IPAPassEliminateDeadCode: This rewrite eliminates unnecessary sub-DAGs that produce transient write outputs which are never consumed subsequently.
IPAPassFlagFunctionsRecompileOnce: This rewrite marks functions with loops as recompile once in order to reduce recompilation overhead.
IPAPassFlagNonDeterminism
IPAPassForwardFunctionCalls: This rewrite forwards a function call to a function with a simple function call that only consumes function parameters and literals into the original call location.
IPAPassInlineFunctions: This rewrite inlines single statement block functions, which have fewer operations than an internal threshold.
IPAPassPropagateReplaceLiterals: This rewrite propagates and replaces literals into functions in order to enable subsequent rewrites such as branch removal.
IPAPassRemoveConstantBinaryOps: This rewrite identifies binary operations with constant matrices such as X * ones, where ones might be created as a vector of ones before a loop.
IPAPassRemoveUnnecessaryCheckpoints: This rewrite identifies and removes unnecessary checkpoints, i.e., persisting of Spark RDDs into a given storage level.
IPAPassRemoveUnusedFunctions: This rewrite identifies and removes unused functions in order to reduce compilation overhead and other overheads such as parfor worker creation, where we construct function copies.
IPAPassReplaceEvalFunctionCalls: This rewrite applies static hop dag and statement block rewrites such as constant folding and branch removal in order to simplify statistic propagation.
IPAPassRewriteFederatedPlan: This rewrite generates a federated execution plan by estimating and setting costs and the FederatedOutput values of all relevant hops in the DML program.
IRepository
IsBlockInList
IsBlockInRange
IsFrameBlockInRange
IterablePredicate
IteratorFactory: Factory pattern for construction of rowIterators of the FrameBlock.
JCudaKernels
JMLCUtils: Utility class containing static methods for working with JMLC.
JSONHelper
KahanFunction: Runtime function type to perform the summation of values using the Kahan summation algorithm.
KahanObject
KahanPlus
KahanPlusSq: Runtime function to perform the summation of squared values using the Kahan summation algorithm.
KeepDataOnWorkerFederatedScheme: Keep Data on Worker Federated scheme When the parameter server runs in federated mode it cannot pull in the data which is already on the workers.
KMVSketch: KMV synopsis(for k minimum values) Distinct-Value Estimation Kevin S.
LanguageException: Exception occurring at the Language level.
LanguageException.LanguageErrorCodes
LazyIterableIterator<T>: This class is a generic base class for lazy, single pass iterator classes in order to simplify the implementation of lazy iterators for mapPartitions use cases.
LazyWriteBuffer
LazyWriteBuffer.RPolicy
LeftIndex
LeftIndex.LixCacheType
LeftIndexingOp
LeftIndexingOp.LeftIndexingMethod
LeftScalarOperator: Scalar operator for scalar-matrix operations with scalar on the left-hand-side.
LegacyEncoder: Base class for all transform encoders providing both a row and block interface for decoding frames to matrices.
LessThan
LessThanEquals
LibCommonsMath: Library for matrix operations that need invocation of Apache Commons Math library.
LibMatrixAgg: MB: Library for matrix aggregations including ak+, uak+ for all combinations of dense and sparse representations, and corrections.
LibMatrixBincell: Library for binary cellwise operations (incl arithmetic, relational, etc).
LibMatrixBincell.BinaryAccessType
LibMatrixCountDistinct: This class contains various methods for counting the number of distinct values inside a MatrixBlock
LibMatrixCUDA: All CUDA kernels and library calls are redirected through this class
LibMatrixCuDNN: This class contains method that invoke CuDNN operations.
LibMatrixCuDNNConvolutionAlgorithm: This class is a wrapper that contain necessary data structures to invoke a cudnn convolution* functions (such as cudnnConvolutionForward, etc) It implements autocloseable to simplify the LibMatrixCuDNN code and also avoids potential memory leaks.
LibMatrixCuDNNInputRowFetcher: Performs a slice operation: out = in[(n+1):(n+1), 1:numColumns]
LibMatrixCuDNNPoolingDescriptors: This class is a wrapper that contain necessary data structures to invoke a cudnn convolution* functions (such as cudnnConvolutionForward, etc) It implements autocloseable to simplify the LibMatrixCuDNN code and also avoids potential memory leaks.
LibMatrixCuDNNRnnAlgorithm
LibMatrixCuMatMult
LibMatrixDatagen
LibMatrixDNN
LibMatrixDNN.PoolingType
LibMatrixDNNConv2d: This class contains the set of operators used for performing conv2d
LibMatrixDNNHelper
LibMatrixDNNIm2Col: This class contains the different implementation of im2col operation
LibMatrixDNNPooling: This class contains the set of operators used for performing pooling
LibMatrixDNNRelu: This class contains the different implementation of rotate180 operation
LibMatrixDNNRelu.ReluBackward: Performs the operation: (X gt 0) * dout
LibMatrixDNNRotate180: This class contains the different implementation of rotate180 operation
LibMatrixDNNRotate180.Rotate180Worker
LibMatrixEquals: Equals library for MatrixBLocks:
LibMatrixMult: MB: Library for matrix multiplications including MM, MV, VV for all combinations of dense, sparse, ultrasparse representations and special operations such as transpose-self matrix multiplication.
LibMatrixNative
LibMatrixOuterAgg: ACS: Purpose of this library is to make some of the unary outer aggregate operator more efficient.
LibMatrixReorg: MB: Library for selected matrix reorg operations including special cases and all combinations of dense and sparse representations.
LibMatrixSketch
LibMatrixTercell: Library for ternary cellwise operations.
LibSpoofPrimitives: This library contains all vector primitives that are used in generated source code for fused operators.
LIBSVMReblockSPInstruction
LibTensorAgg
LibTensorBincell
LibTensorReorg
Lineage
LineageCache
LineageCacheConfig
LineageCacheConfig.LineageCachePolicy
LineageCacheConfig.ReuseCacheType
LineageCacheEntry
LineageCacheEviction
LineageCacheStatistics: This singleton provides basic lineage caching statistics in CP.
LineageCodegenItem
LineageDebugger
LineageDedupBlock
LineageDedupUtils
LineageEstimator
LineageEstimatorStatistics
LineageGPUCacheEviction
LineageItem
LineageItem.LineageItemType
LineageItemUtils
LineageMap
LineageObject
LineageParser
LineageRecomputeUtils
LineageRewriteReuse
LineageTokenizer
LineageTraceable
LinearRegression
ListAppendPropagator
ListAppendRemoveCPInstruction
ListIndexingCPInstruction
ListObject
ListReader
ListRemovePropagator
ListWriter
LiteralOp
LiteralReplacement
LiveVariableAnalysis
Local
LocalCPInstruction: A simple instruction that take whatever input it is and return it as a local matrix.
LocalDataPartitioner
LocalFileUtils
LocalParamServer
LocalParWorker: Instances of this class can be used to execute tasks in parallel.
LocalPSWorker
LocalTaskQueue<T>: This class provides a way of dynamic task distribution to multiple workers in local multi-threaded environments.
LocalVariableMap: Replaces `HashMap⟨String, Data⟩` as the table of variable names and references.
LongArray
LongLongDoubleHashMap: This native long long - double hashmap is specifically designed for ctable operations which only require addvalue - extract semantics.
LongLongDoubleHashMap.ADoubleEntry
LongLongDoubleHashMap.EntryType
Lop: Base class for all Lops.
Lop.SimpleInstType: Lop types
Lop.Type
Lop.VisitStatus
LopComparator<N extends Lop>: Comparator class used in sorting the LopDAG in topological order.
LopProperties
LopRewriter
LopRewriteRule
LopsException
MapInputSignature
MapJoinSignature
MapmmChainSPInstruction
MapmmSPInstruction
MapMult
MapMult.CacheType
MapMultChain
MapMultChain.ChainType
MapperService
MapToBit
MapToByte
MapToChar
MapToCharPByte
MapToFactory
MapToFactory.MAP_TYPE
MapToInt
MapToUByte
MapToZero
MarkForLineageReuse
MaterializeSort
MatMultCP
Matrix: Matrix encapsulates a SystemDS matrix.
MatrixAppendCPInstruction
MatrixAppendGPUInstruction: Implements the cbind and rbind functions for matrices
MatrixAppendMSPInstruction
MatrixAppendRSPInstruction
MatrixBlock
MatrixBlock.SparsityEstimate
MatrixBlockDataInput: Any data input that is intended to support fast deserialization / read of entire blocks should implement this interface.
MatrixBlockDataOutput: Any data output that is intended to support fast serialization / write of entire blocks should implement this interface.
MatrixBlockDictionary
MatrixBlockFromFrame
MatrixBuiltinGPUInstruction
MatrixBuiltinNaryCPInstruction
MatrixCell
MatrixCharacteristics
MatrixFormat: MatrixFormat represents the different matrix formats supported by the MLContext API.
MatrixGenerateReader
MatrixGenerateReader.MatrixReaderRowIrregular
MatrixGenerateReader.MatrixReaderRowRegularColIrregular
MatrixGenerateReader.MatrixReaderRowRegularColRegular
MatrixIndexes: This represent the indexes to the blocks of the matrix.
MatrixIndexingCPFileInstruction: This instruction is used if a single partition is too large to fit in memory.
MatrixIndexingCPInstruction
MatrixIndexingGPUInstruction
MatrixIndexingSPInstruction: This class implements the matrix indexing functionality inside CP.
MatrixLineagePair: Class to represent the relation between a MatrixObject and the respective LineageItem.
MatrixMatrixArithmeticGPUInstruction
MatrixMatrixAxpyGPUInstruction
MatrixMatrixBinaryOpFunction
MatrixMatrixBuiltinGPUInstruction
MatrixMatrixRelationalBinaryGPUInstruction
MatrixMetadata: Matrix metadata, such as the number of rows, the number of columns, the number of non-zero values, the number of rows per block, and the number of columns per block in the matrix.
MatrixMultiplicationPropagator: Used for propagating constraints in a matrix multiplication.
MatrixMultiplicationPropagatorNaive: MatrixMultiplicationPropagator that overrides generateFineGrainedConstraints with a naive propagation of the fine-grained constraints.
MatrixMultiplicationPropagatorPrivateFirst: MatrixMultiplicationPropagator that overrides generateFineGrainedConstraints by finding the private elements first followed by propagating PrivateAggregation in case of non-aggregating operator types.
MatrixMultiplicationPropagatorPrivateFirstOptimized: MatrixMultiplicationPropagator that overrides generateFineGrainedConstraints by finding the private elements first while propagating PrivateAggregation in case of non-aggregating operator types.
MatrixObject: Represents a matrix in control program.
MatrixObject.UpdateType
MatrixObjectFuture
MatrixReader: Base class for all format-specific matrix readers.
MatrixReaderFactory
MatrixReshapeGPUInstruction
MatrixReshapeSPInstruction
MatrixScalarUnaryFunction
MatrixSketch
MatrixSketchFactory
MatrixValue
MatrixValue.CellIndex
MatrixVectorBinaryOpPartitionFunction
MatrixWriter: Base class for all format-specific matrix writers.
MatrixWriterFactory
Mean
Memorizer
MemoryCostEstimator
MemoryEstimates: Memory Estimates is a helper class containing static classes that estimate the memory requirements of different types of objects in java.
MemoTable: Memoization of federated execution alternatives.
MemoTable: Memoization Table (hop id, worst-case matrix characteristics).
MergeSort
Metadata: Abstract metadata class for MLContext API.
MetaData: Class to store metadata associated with a file (e.g., a matrix) on disk.
MetaDataAll
MetaDataFormat
MetaDataUtils
Minus
Minus1Multiply
MinusMultiply
MinusNz
MLContext: The MLContext API offers programmatic access to SystemDS on Spark from languages such as Scala, Java, and Python.
MLContext.ExecutionType: The different types of execution environments supported by SystemDS.
MLContext.ExplainLevel: The different explain levels supported by SystemDS.
MLContextConversionUtil: Utility class containing methods to perform data conversions.
MLContextException: Uncaught exception representing SystemDS exceptions that occur through the MLContext API.
MLContextProxy: The purpose of this proxy is to shield systemds internals from direct access to MLContext which would try to load spark libraries and hence fail if these are not available.
MLContextUtil: Utility class containing methods for working with the MLContext API.
MLResults: MLResults handles the results returned from executing a Script using the MLContext API.
MMChainCPInstruction
MMChainFEDInstruction
MMCJ: Lop to perform cross product operation
MMCJ.MMCJType
MMFEDInstruction
MMNode: Helper class to represent matrix multiply operators in a DAG along with references to its abstract data handles.
MMRJ: Lop to perform cross product operation
MMTSJ: Lop to perform transpose-identity operation (t(X)%*%X or X%*%t(X)), used to represent CP and MR instruction but in case of MR there is an additional Aggregate at the reducers.
MMTSJ.MMTSJType
MMTSJCPInstruction
MMTSJGPUInstruction
MMZip: Lop to perform zip matrix multiplication
Modulus: Integer modulus, where we adhere to the defined R semantics: ("%% indicates x mod y and %/% indicates integer division.
MultiAssignmentStatement
MultiColBitmap: Uncompressed representation of one or more columns in bitmap format.
MultiColumnEncoder
Multiply
Multiply2
MultiReturnBuiltinCPInstruction
MultiReturnParameterizedBuiltinCPInstruction
MultiReturnParameterizedBuiltinFEDInstruction
MultiReturnParameterizedBuiltinFEDInstruction.CreateFrameEncoder
MultiReturnParameterizedBuiltinFEDInstruction.ExecuteFrameEncoder
MultiReturnParameterizedBuiltinSPInstruction
MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuild2Function
MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction: This function pre-aggregates distinct values of recoded columns per partition (part of distributed recode map construction, used for recoding, binning and dummy coding).
MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeGroup2Function
MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeGroupFunction: This function assigns codes to globally distinct values of recoded columns and writes the resulting column map in textcell (IJV) format to the output.
MultiThreadedHop: Optional hop interface, to be implemented by multi-threaded hops.
MultiThreadedOperator
Nary: Lop to perform an operation on a variable number of operands.
NaryOp: The NaryOp Hop allows for a variable number of operands.
NativeHEHelper
NativeHelper: This class helps in loading native library.
NativeHelper.NativeBlasState
NativeStatistics
NetworkTrafficCounter
NormalPRNGenerator: Class that can generate a stream of random numbers from standard normal distribution N(0,1).
Not
NotEquals
ObjectRowIterator
OffsetByte
OffsetChar
OffsetColumnIndex
OffsetEmpty
OffsetFactory
OffsetFactory.OFF_TYPE: The specific underlying types of offsets.
OffsetFactory.OFF_TYPE_SPECIALIZATIONS: Specialized types of underlying offsets.
OffsetSingle
OffsetTwo
Op
OperationsOnMatrixValues
Operator
OperatorOrderingUtils
OperatorType
OpMetadata
OpNormal
OpSided
OptimizationWrapper: Wrapper to ParFOR cost estimation and optimizer.
Optimizer: Generic optimizer super class that defines the interface of all implemented optimizers.
Optimizer.CostModelType
Optimizer.PlanInputType
OptimizerConstrained: Rule-Based ParFor Optimizer (time: O(n)): Applied rule-based rewrites: - see base class.
OptimizerHeuristic: Heuristic ParFor Optimizer: This optimizer extends the rule-based optimizer by a time-based cost estimate for execution type decisions.
OptimizerRuleBased: Rule-Based ParFor Optimizer (time: O(n)): Applied rule-based rewrites - 1) rewrite set data partitioner (incl.
OptimizerUtils
OptimizerUtils.MemoryManager: Memory managers (static partitioned, unified)
OptimizerUtils.OptimizationLevel: Optimization Types for Compilation O0 STATIC - Decisions for scheduling operations on CP/MR are based on predefined set of rules, which check if the dimensions are below a fixed/static threshold (OLD Method of choosing between CP and MR).
OptionalArray<T>
OptNode: Internal representation of a plan alternative for program blocks and instructions in order to enable efficient and simple recursive enumeration and plan changes.
OptNode.ExecType
OptNode.NodeType
OptNode.ParamType
OptNodeStatistics: TODO extend to right as well (see PerfTestTool, currently only trained with regard to left) TODO integrate mem and exec time for reuse?
OptTree: Represents a complete plan of a top-level parfor.
OptTreeConverter: Converter for creating an internal plan representation for a given runtime program and to modify/create the runtime program according to the optimized plan.
OptTreePlanChecker
OptTreePlanMapping: Helper class for mapping nodes of the internal plan representation to statement blocks and hops / function call statements of a given DML program.
OptTreePlanMappingAbstract
OptTreePlanMappingRuntime
Or
ORLocalScheme: Data partitioner Overlap_Reshuffle: for each worker, use a new permutation multiply P %*% X, where P is constructed for example with P=table(seq(1,nrow(X),sample(nrow(X), nrow(X))))
ORSparkScheme: Spark data partitioner Overlap_Reshuffle:
OuterVectorBinaryOpFunction
OutputParameters: class to maintain output parameters for a lop.
OutputStatement
PageCache
Pair<K,​V>
PairWritableBlock: Custom writable for a pair of matrix indexes and matrix block as required for binaryblock in remote data partitioning.
PairWritableCell: Custom writable for a pair of matrix indexes and matrix cell as required for binarycell in remote data partitioning.
ParameterBuilder: Class to help setting variables in a script.
ParameterExpression
ParameterizedBuiltin: Defines a LOP for functions.
ParameterizedBuiltin: Function object for builtin function that takes a list of name=value parameters.
ParameterizedBuiltin.ParameterizedBuiltinCode
ParameterizedBuiltin.ProbabilityDistributionCode
ParameterizedBuiltinCPInstruction
ParameterizedBuiltinFEDInstruction
ParameterizedBuiltinFEDInstruction.DecodeMatrix
ParameterizedBuiltinFunctionExpression
ParameterizedBuiltinOp: Defines the HOP for calling an internal function (with custom parameters) from a DML script.
ParameterizedBuiltinSPInstruction
ParameterizedBuiltinSPInstruction.CreateMatrixCell
ParameterizedBuiltinSPInstruction.RDDContainsFunction
ParameterizedBuiltinSPInstruction.RDDFrameReplaceFunction
ParameterizedBuiltinSPInstruction.RDDMapGroupedAggFunction
ParameterizedBuiltinSPInstruction.RDDMapGroupedAggFunction2: Similar to RDDMapGroupedAggFunction but single output block.
ParameterizedBuiltinSPInstruction.RDDRemoveEmptyFunction
ParameterizedBuiltinSPInstruction.RDDRemoveEmptyFunctionInMem
ParameterizedBuiltinSPInstruction.RDDReplaceFunction
ParameterizedBuiltinSPInstruction.RDDRExpandFunction
ParameterizedBuiltinSPInstruction.RDDTokenizeFunction
ParameterizedBuiltinSPInstruction.RDDTransformApplyFunction
ParameterizedBuiltinSPInstruction.RDDTransformApplyOffsetFunction
ParameterizedBuiltinSPInstruction.RDDTransformDecodeExpandFunction
ParameterizedBuiltinSPInstruction.RDDTransformDecodeFunction
ParamservBuiltinCPInstruction
ParamServer
ParamServStatistics
ParamservUtils
ParForBody: Wrapper for exchanging parfor body data structures.
ParForProgramBlock: The ParForProgramBlock has the same execution semantics as a ForProgamBlock but executes the independent iterations in parallel.
ParForProgramBlock.PartitionFormat: Convenience class to package PDataPartitionFormat and its parameters.
ParForProgramBlock.PDataPartitioner
ParForProgramBlock.PDataPartitionFormat
ParForProgramBlock.PExecMode
ParForProgramBlock.POptMode
ParForProgramBlock.PResultMerge
ParForProgramBlock.PTaskPartitioner
ParForStatement: This ParForStatement is essentially identical to a ForStatement, except an extended toString method for printing the 'parfor' keyword.
ParForStatementBlock: This ParForStatementBlock is essentially identical to a ForStatementBlock, except an extended validate for checking/setting optional parfor parameters and running the loop dependency analysis.
ParForStatementBlock.ResultVar
ParForStatistics
ParseException: This exception is thrown when parse issues are encountered.
ParseInfo
ParserFactory
ParserWrapper: Base class for all dml parsers in order to make the various compilation chains independent of the used parser.
PartialAggregate: Lop to perform a partial aggregation.
PartitionedBlock<T extends CacheBlock>: This class is for partitioned matrix/frame blocks, to be used as broadcasts.
PartitionedBroadcast<T extends CacheBlock>: This class is a wrapper around an array of broadcasts of partitioned matrix/frame blocks, which is required due to 2GB limitations of Spark's broadcast handling.
ParWorker: Super class for master/worker pattern implementations.
PathStatement
PerformGroupByAggInCombiner
PerformGroupByAggInReducer
PickByCount
PickByCount.OperationTypes
PlaintextMatrix: This class abstracts over an encrypted matrix of ciphertexts.
PlanAnalyzer: Utility functions to extract structural information from the memo table, including connected components (aka partitions) of partial fusion plans, materialization points of partitions, and root nodes of partitions.
PlanPartition
PlanSelection
PlanSelection.VisitMarkCost
PlanSelectionFuseAll: This plan selection heuristic aims for maximal fusion, which potentially leads to overlapping fused operators and thus, redundant computation but with a minimal number of materialized intermediate results.
PlanSelectionFuseCostBased: This cost-based plan selection algorithm chooses fused operators based on the DAG structure and resulting overall costs.
PlanSelectionFuseCostBasedV2: This cost-based plan selection algorithm chooses fused operators based on the DAG structure and resulting overall costs.
PlanSelectionFuseNoRedundancy: This plan selection heuristic aims for fusion without any redundant computation, which, however, potentially leads to more materialized intermediates than the fuse all heuristic.
Plus
PlusMultiply
PMapmmSPInstruction: This pmapmm matrix multiplication instruction is still experimental not integrated in automatic operator selection yet.
PMapMult
PMMJ
PMMJ.CacheType
PMMJCPInstruction
PmmSPInstruction
PoissonPRNGenerator: Class that can generate a stream of random numbers from Poisson distribution with specified mean.
PorterStemmer: Stemmer, implementing the Porter Stemming Algorithm The Stemmer class transforms a word into its root form.
Power
Power2
PrefetchCPInstruction
PreparedScript: Representation of a prepared (precompiled) DML/PyDML script.
PrintStatement
PrintStatement.PRINTTYPE: The PRINTTYPE options are: PRINT, PRINTF, and STOP.
PrivacyConstraint: PrivacyConstraint holds all privacy constraints for data in the system at compile time and runtime.
PrivacyConstraint.PrivacyLevel
PrivacyConstraintLoader
PrivacyConstraintLoader.PrivacyConstraintRetriever: FederatedUDF for retrieving privacy constraint of data stored in file name.
PrivacyConstraintLoaderMock: This class is a mockup of the PrivacyConstraintLoader which replaces the call to loadFederatedPrivacyConstraints.
PrivacyMonitor
PrivacyPropagator: Class with static methods merging privacy constraints of operands in expressions to generate the privacy constraints of the output.
PrivacyUtils
PRNGenerator
Program
ProgramBlock
ProgramConverter: Program converter functionalities for (1) creating deep copies of program blocks, instructions, function program blocks, and (2) serializing and parsing of programs, program blocks, functions program blocks.
ProgramRecompiler
ProgramRewriter: This program rewriter applies a variety of rule-based rewrites on all hop dags of the given program in one pass over the entire program.
ProgramRewriteStatus
ProjectInfo: Obtains information that is stored in the manifest when the SystemDS jar is built.
Propagator: Interface for all propagator instances with a single output.
PropagatorMultiReturn: Interface for all propagator instances with multiple outputs.
PSRpcCall
PSRpcFactory
PSRpcHandler
PSRpcObject
PSRpcResponse
PSRpcResponse.Type
PSWorker
PublicKey
Py4jConverterUtils: Utils for converting python data to java.
PythonDMLScript
QDictionary: This dictionary class aims to encapsulate the storage and operations over unique floating point values of a column group.
QuantilePickCPInstruction
QuantilePickFEDInstruction
QuantilePickFEDInstruction.CreateMatrixFromFrame
QuantilePickFEDInstruction.GetHistogram
QuantilePickFEDInstruction.GetValuesInRange
QuantilePickFEDInstruction.GetValuesInRanges
QuantilePickFEDInstruction.MinMax
QuantilePickSPInstruction
QuantileSortCPInstruction: This class supports two variants of sort operation on a 1-dimensional input matrix.
QuantileSortFEDInstruction
QuantileSortSPInstruction: This class supports two variants of sort operation on a 1-dimensional input matrix.
QuaternaryCPInstruction
QuaternaryFEDInstruction
QuaternaryOp: Note: this hop should be called AggQuaternaryOp in consistency with AggUnaryOp and AggBinaryOp; however, since there does not exist a real QuaternaryOp yet - we can leave it as is for now.
QuaternaryOperator
QuaternarySPInstruction
QuaternaryWCeMMFEDInstruction
QuaternaryWDivMMFEDInstruction
QuaternaryWSigmoidFEDInstruction
QuaternaryWSLossFEDInstruction
QuaternaryWUMMFEDInstruction
RandNPair: Class that generates a pair of random numbers from standard normal distribution N(0,1).
RandomMatrixGenerator
RandomMatrixGenerator.PDF: Types of Probability density functions
RandSPInstruction
RangeIndex
RawRow
RBindPropagator
RDDAggregateUtils: Collection of utility methods for aggregating binary block rdds.
RDDConverterUtils
RDDConverterUtils.BinaryCellToBinaryBlockFunction
RDDConverterUtils.DataFrameExtractIDFunction
RDDConverterUtilsExt: NOTE: These are experimental converter utils.
RDDConverterUtilsExt.AddRowID
RDDConverterUtilsExt.RDDConverterTypes
RDDObject
RDDSortUtils
RDDSortUtils.IndexComparator
RDDSortUtils.IndexComparator2
ReachabilityGraph
ReachabilityGraph.SubProblem
ReaderBinaryBlock
ReaderBinaryBlockParallel
ReaderColumnSelection: Base class for all column selection readers.
ReaderColumnSelectionDenseMultiBlock
ReaderColumnSelectionDenseMultiBlockTransposed
ReaderColumnSelectionDenseSingleBlock
ReaderColumnSelectionDenseSingleBlockTransposed
ReaderColumnSelectionSparse: Used to extract the values at certain indexes from each row in a sparse matrix Keeps returning all-zeros arrays until reaching the last possible index.
ReaderColumnSelectionSparseTransposed: Used to extract the values at certain indexes from each row in a sparse matrix Keeps returning all-zeros arrays until reaching the last possible index.
ReaderCompressed
ReaderHDF5
ReaderHDF5Parallel
ReaderMapping
ReaderMapping.FrameReaderMapping
ReaderMapping.MatrixReaderMapping
ReaderTextCell
ReaderTextCellParallel: Parallel version of ReaderTextCell.java.
ReaderTextCellParallel.CellBuffer: Useful class for buffering unordered cells before locking target onces and appending all buffered cells.
ReaderTextCellParallel.CountNnzTask
ReaderTextCellParallel.ReadTask
ReaderTextCSV
ReaderTextCSVParallel: Parallel version of ReaderTextCSV.java.
ReaderTextLIBSVM
ReaderTextLIBSVMParallel
ReaderWriterFederated: This class serves as the reader for federated objects.
ReadProperties
ReBlock: Lop to perform reblock operation
ReblockBuffer
ReblockFEDInstruction
ReblockSPInstruction
ReblockTensorFunction
Recompiler: Dynamic recompilation of hop dags to runtime instructions, which includes the following substeps: (1) deep copy hop dag, (2) refresh matrix characteristics, (3) apply dynamic rewrites, (4) refresh memory estimates, (5) construct lops (incl operator selection), and (6) generate runtime program (incl piggybacking).
Recompiler.ResetType
RecompileStatistics
RecompileStatus
RecomputeNnzFunction
ReduceAll
ReduceCol
ReduceDiag
ReduceRow
ReIndexOperator
RelationalBinaryGPUInstruction
RelationalExpression
RemoteDPParForSpark: TODO heavy hitter maintenance TODO data partitioning with binarycell
RemoteDPParForSparkWorker
RemoteParForJobReturn: Wrapper for job return of ParFor REMOTE for transferring statistics and result symbol table.
RemoteParForSpark: This class serves two purposes: (1) isolating Spark imports to enable running in environments where no Spark libraries are available, and (2) to follow the same structure as the parfor remote_mr job submission.
RemoteParForSparkWorker
RemoteParForUtils: Common functionalities for parfor workers in MR jobs.
ReorgCPInstruction
ReorgFEDInstruction
ReorgFEDInstruction.DiagMatrix
ReorgFEDInstruction.Rdiag
ReorgGPUInstruction
ReorgMapFunction
ReorgOp: Reorg (cell) operation: aij Properties: Symbol: ', rdiag, rshape, rsort 1 Operand (except sort and reshape take additional arguments) Semantic: change indices (in mapper or reducer) NOTE MB: reshape integrated here because (1) ParameterizedBuiltinOp requires name-value pairs for params and (2) most importantly semantic of reshape is exactly a reorg op.
ReorgOperator
ReorgSPInstruction
ReplicateBlockFunction
ReplicateTensorFunction
ReplicateToMaxFederatedScheme: Replicate to Max Federated scheme When the parameter server runs in federated mode it cannot pull in the data which is already on the workers.
ReplicateVectorFunction
Request
RequestModel
ReshapeCPInstruction
ReshapeFEDInstruction
Response
ResultMerge<T extends CacheableData<?>>
ResultMergeFrameLocalMemory
ResultMergeLocalAutomatic
ResultMergeLocalFile: TODO potential extension: parallel merge (create individual staging files concurrently) NOTE: file merge typically used due to memory constraints - parallel merge would increase the memory consumption again.
ResultMergeLocalMemory: Local in-memory realization of result merge.
ResultMergeMatrix: Due to independence of all iterations, any result has the following properties: (1) non local var, (2) matrix object, and (3) completely independent.
ResultMergeRemoteGrouping
ResultMergeRemoteSorting
ResultMergeRemoteSpark
ResultMergeRemoteSparkWCompare
ResultMergeTaggedMatrixIndexes: This class serves as composite key for the remote result merge job (for any data format) in order to sort on both matrix indexes and tag but group all blocks according to matrix indexes only.
ResultVariables: A ResultVariables object holds the data returned by a call to `PreparedScript`'s `executeScript` method, which executes a DML/PyDML script.
RevIndex
RewriteAddBroadcastLop
RewriteAddChkpointLop
RewriteAddPrefetchLop
RewriteAlgebraicSimplificationDynamic: Rule: Algebraic Simplifications.
RewriteAlgebraicSimplificationStatic: Rule: Algebraic Simplifications.
RewriteBlockSizeAndReblock: Rule: BlockSizeAndReblock.
RewriteCommonSubexpressionElimination: Rule: CommonSubexpressionElimination.
RewriteCompressedReblock: Rule: Compressed Re block if config compressed.linalg is enabled, we inject compression directions after read of matrices if number of rows is above 1000 and cols at least 1.
RewriteConstantFolding: Rule: Constant Folding.
RewriteElementwiseMultChainOptimization: Prerequisite: RewriteCommonSubexpressionElimination must run before this rule.
RewriteFixIDs
RewriteForLoopVectorization: Rule: Simplify program structure by pulling if or else statement body out (removing the if statement block ifself) in order to allow intra-procedure analysis to propagate exact statistics.
RewriteGPUSpecificOps
RewriteHoistLoopInvariantOperations: Rule: Simplify program structure by hoisting loop-invariant operations out of while, for, or parfor loops.
RewriteIndexingVectorization: Rule: Indexing vectorization.
RewriteInjectSparkLoopCheckpointing: Rule: Insert checkpointing operations for caching purposes.
RewriteInjectSparkPReadCheckpointing: Rule: BlockSizeAndReblock.
RewriteMarkLoopVariablesUpdateInPlace: Rule: Mark loop variables that are only read/updated through cp left indexing for update in-place.
RewriteMatrixMultChainOptimization: Rule: Determine the optimal order of execution for a chain of matrix multiplications Solution: Classic Dynamic Programming Approach: Currently, the approach based only on matrix dimensions Goal: To reduce the number of computations in the run-time (map-reduce) layer
RewriteMatrixMultChainOptimizationSparse: Rule: Determine the optimal order of execution for a chain of matrix multiplications Solution: Classic Dynamic Programming Approach: Currently, the approach based only on matrix dimensions and sparsity estimates using the MNC sketch Goal: To reduce the number of computations in the run-time (map-reduce) layer
RewriteMergeBlockSequence: Rule: Simplify program structure by merging sequences of last-level statement blocks in order to create optimization opportunities.
RewriteRemoveDanglingParentReferences: This rewrite is a general-purpose cleanup pass that removes any dangling parent references in one pass through the hop DAG.
RewriteRemoveEmptyBasicBlocks: Rule: Simplify program structure by removing empty last-level blocks, which may originate from the original program or due to a sequence of rewrites (e.g., checkpoint injection and subsequent IPA).
RewriteRemoveForLoopEmptySequence: Rule: Simplify program structure by removing (par)for statements iterating over an empty sequence, i.e., (par)for-loops without a single iteration.
RewriteRemovePersistentReadWrite: This rewrite is a custom rewrite for JMLC in order to replace all persistent reads and writes with transient reads and writes from the symbol table.
RewriteRemoveReadAfterWrite: Rule: RemoveReadAfterWrite.
RewriteRemoveUnnecessaryBranches: Rule: Simplify program structure by pulling if or else statement body out (removing the if statement block ifself) in order to allow intra-procedure analysis to propagate exact statistics.
RewriteRemoveUnnecessaryCasts: Rule: RemoveUnnecessaryCasts.
RewriteSplitDagDataDependentOperators: Rule: Split Hop DAG after specific data-dependent operators.
RewriteSplitDagUnknownCSVRead: Rule: Split Hop DAG after CSV reads with unknown size.
RewriteTransientWriteParentHandling: Rule: Eliminate for Transient Write DataHops to have no parents Solution: Move parent edges of Transient Write Hop to parent of its child Reason: Transient Write not being a root messes up analysis for Lop's to Instruction translation (according to Amol)
RightIndex
RightScalarOperator: Scalar operator for scalar-matrix operations with scalar on the right-hand-side.
RmmSPInstruction
RowIterator<T>
RowMatrixBlock
SampleEstimatorFactory
SampleEstimatorFactory.EstimationType
SampleProperties
ScalarAppendCPInstruction
ScalarBuiltinNaryCPInstruction: The ScalarBuiltinMultipleCPInstruction class is responsible for printf-style Java-based string formatting.
ScalarMatrixArithmeticGPUInstruction
ScalarMatrixBuiltinGPUInstruction
ScalarMatrixRelationalBinaryGPUInstruction
ScalarObject
ScalarObjectFactory
ScalarOperator: Base class for all scalar operators.
Script: A Script object encapsulates a DML or PYDML script.
ScriptExecutor: ScriptExecutor executes a DML or PYDML Script object using SystemDS.
ScriptExecutorUtils
ScriptFactory: Factory for creating DML and PYDML Script objects from strings, files, URLs, and input streams.
SEALClient
SEALServer
SerLongWritable: Wrapper for LongWritable in order to make it serializable as required for shuffle in spark instructions.
SerText: Wrapper for Text in order to make it serializable as required for shuffle in spark instructions.
ShadowBuffer
ShlosserEstimator
ShlosserJackknifeEstimator
ShlosserJackknifeEstimator.CriticalValue
ShuffleFederatedScheme: Shuffle Federated scheme When the parameter server runs in federated mode it cannot pull in the data which is already on the workers.
SimpleOperator
SingleIndex
SinglePrecisionCudaSupportFunctions
SingletonLookupHashMap: This class allows sharing of objects across the entire program.
SmallestPriorityQueue: Deceiving name, but is used to contain the k smallest values inserted.
SmoothedJackknifeEstimator
SortIndex: This index function is NOT used for actual sorting but just as a reference in ReorgOperator in order to identify sort operations.
SortKeys
SortKeys.OperationTypes
SortUtils: Utilities for sorting, primarily used for SparseRows.
SparkDataPartitioner
SparkExecutionContext
SparkExecutionContext.SparkClusterConfig: Captures relevant spark cluster configuration properties, e.g., memory budgets and degree of parallelism.
SparkParamservUtils
SparkPSBody: Wrapper class containing all needed for launching spark remote worker
SparkPSProxy
SparkPSWorker
SparkStatistics
SparkUtils
SparseBlock: This SparseBlock is an abstraction for different sparse matrix formats.
SparseBlock.Type
SparseBlockCOO: SparseBlock implementation that realizes a traditional 'coordinate matrix' representation, where the entire sparse block is stored as triples in three arrays: row indexes, column indexes, and values, where row indexes and colunm indexes are sorted in order to allow binary search.
SparseBlockCSR: SparseBlock implementation that realizes a traditional 'compressed sparse row' representation, where the entire sparse block is stored as three arrays: ptr of length rlen+1 to store offsets per row, and indexes/values of length nnz to store column indexes and values of non-zero entries.
SparseBlockFactory

SparseBlockMCSR

SparseBlock implementation that realizes a 'modified compressed sparse row' representation, where each compressed row is stored as a separate SparseRow object which provides flexibility for unsorted row appends without the need for global reshifting of values/indexes but it incurs additional memory overhead per row for object/array headers per row which also slows down memory-bound operations due to higher memory bandwidth requirements.

SparseEncoding: Most common is zero encoding
SparseRow: Base class for sparse row implementations such as sparse row vectors and sparse scalars (single value per row).
SparseRowScalar
SparseRowVector
SparsityEstimator
SparsityEstimator.OpCode
SPInstruction
SPInstruction.SPType
SPInstructionParser
SpoofCellwise
SpoofCellwise.AggOp
SpoofCellwise.CellType
SpoofCompiler
SpoofCompiler.CompilerType
SpoofCompiler.GeneratorAPI
SpoofCompiler.IntegrationType
SpoofCompiler.PlanCachePolicy
SpoofCompiler.PlanSelector
SpoofCompiler.RegisterAlloc
SpoofCPInstruction
SpoofCUDACellwise
SpoofCUDAInstruction
SpoofCUDAInstruction.DoublePrecision
SpoofCUDAInstruction.SinglePrecision
SpoofCUDAOperator
SpoofCUDAOperator.PrecisionProxy
SpoofCUDARowwise
SpoofFEDInstruction
SpoofFused
SpoofFusedOp
SpoofFusedOp.SpoofOutputDimsType
SpoofMultiAggregate
SpoofOperator
SpoofOperator.SideInput
SpoofOperator.SideInputSparseCell
SpoofOperator.SideInputSparseRow
SpoofOuterProduct
SpoofOuterProduct.OutProdType
SpoofRowwise
SpoofRowwise.RowType
SpoofSPInstruction
SpoofSPInstruction.ReplicateRightFactorFunction
Sql
SqlCPInstruction
StagingFileUtils
Stat: Collection of all ParFor statistic types.
Statement
Statement.FederatedPSScheme
Statement.PSCheckpointing
Statement.PSFrequency
Statement.PSModeType
Statement.PSRuntimeBalancing
Statement.PSScheme
Statement.PSUpdateType
StatementBlock
StatementBlockRewriteRule: Base class for all hop rewrites in order to enable generic application of all rules.
StatementInfo: This class exists solely to prevent compiler warnings.
StatisticMonitor: This singleton statistic monitor is used to consolidate all parfor runtime statistics.
Statistics: This class captures all statistics.
StatisticsController
StatisticsModel
StatisticsOptions
StatisticsService
StringArray
StringIdentifier
StringInitCPInstruction
StringObject
StringRowIterator
SubsampleToMinFederatedScheme: Subsample to Min Federated scheme When the parameter server runs in federated mode it cannot pull in the data which is already on the workers.
SwapIndex
SystemDSLoaderUtils
Task: A task is a logical group of one or multiple iterations (each iteration is assigned to exactly one task).
Task.TaskType
TaskPartitioner: This is the base class for all task partitioner.
TaskPartitionerFactoring: This factoring task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks.
TaskPartitionerFactoringCmax: Factoring with maximum constraint (e.g., if LIX matrix out-of-core and we need to bound the maximum number of iterations per map task -> memory bounds)
TaskPartitionerFactoringCmin: Factoring with minimum constraint (e.g., if communication is expensive)
TaskPartitionerFixedsize: This naive task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to the given task size.
TaskPartitionerNaive: This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.
TaskPartitionerStatic: This static task partitioner virtually iterates over the given FOR loop (from, to, incr), creates iterations and group them to tasks according to a task size of numIterations/numWorkers.
TemplateBase
TemplateBase.CloseType
TemplateBase.TemplateType
TemplateCell
TemplateCell.HopInputComparator: Comparator to order input hops of the cell template.
TemplateMultiAgg
TemplateOuterProduct
TemplateRow
TemplateRow.HopInputComparator: Comparator to order input hops of the row aggregate template.
TemplateUtils
TensorBlock: A `TensorBlock` is the most top level representation of a tensor.
TensorCharacteristics
TensorIndexes: This represent the indexes to the blocks of the tensor.
TensorObject
TensorReader
TensorReaderBinaryBlock
TensorReaderBinaryBlockParallel
TensorReaderFactory
TensorReaderTextCell
TensorReaderTextCellParallel
TensorTensorBinaryOpFunction
TensorTensorBinaryOpPartitionFunction
TensorWriter
TensorWriterBinaryBlock
TensorWriterBinaryBlockParallel
TensorWriterBinaryBlockParallel.WriteFileTask
TensorWriterFactory
TensorWriterTextCell
TensorWriterTextCellParallel
Ternary
Ternary
Ternary: Lop to perform Sum of a matrix with another matrix multiplied by Scalar.
TernaryAggregate
TernaryCPInstruction
TernaryFEDInstruction
TernaryFrameScalarCPInstruction
TernaryFrameScalarFEDInstruction
TernaryFrameScalarSPInstruction
TernaryOp: Primary use cases for now, are `quantile (<n-1-matrix>, <n-1-matrix>, <literal>): quantile (A, w, 0.5)` `quantile (<n-1-matrix>, <n-1-matrix>, <scalar>): quantile (A, w, s)` `interquantile (<n-1-matrix>, <n-1-matrix>, <scalar>): interquantile (A, w, s)` Keep in mind, that we also have binaries for it w/o weights.
TernaryOperator
TernarySPInstruction
TernaryValueFunction
TernaryValueFunction.ValueFunctionWithConstant
TextToBinaryCellConverter
TfMetaUtils
TfOffsetMap
TfUtils
TfUtils.TfMethod
Timing: Helper class for encapsulated time measurements.
Token
Token.SubToken
Tokenizer
TokenizerApplier
TokenizerApplierCount
TokenizerApplierHash
TokenizerApplierPosition
TokenizerBuilder
TokenizerBuilderNgram
TokenizerBuilderWhitespaceSplit
TokenizerFactory
TrafficModel
Transform
TransformStatistics
TriggerBroadcastTask
TriggerCheckpointTask
TriggerPrefetchTask
TripleIndexes
TripleIndexes.Comparator
Tsmm2SPInstruction
TsmmFEDInstruction
TsmmSPInstruction
TwoIndex
Types
Types.AggOp
Types.BlockType: Serialization block types (empty, dense, sparse, ultra-sparse)
Types.CorrectionLocationType
Types.DataType: Data types (tensor, matrix, scalar, frame, object, unknown).
Types.Direction: Type of aggregation direction
Types.ExecMode: Execution mode for entire script.
Types.ExecType: Execution type of individual operations.
Types.FileFormat
Types.FunctionBlock: Common type for both function statement blocks and function program blocks
Types.OpOp1
Types.OpOp2
Types.OpOp3
Types.OpOp4
Types.OpOpData
Types.OpOpDG
Types.OpOpDnn
Types.OpOpN
Types.ParamBuiltinOp
Types.ReOrgOp
Types.ReturnType: Type of builtin or user-defined function with regard to its number of return variables.
Types.ValueType: Value types (int, double, string, boolean, unknown).
UAggOuterChain: TODO Currently this lop only support the right hand side in distributed cache.
UaggOuterChainCPInstruction
UaggOuterChainSPInstruction: Two types of broadcast variables used -- 1.
Unary
Unary
Unary: Lop to perform following operations: with one operand -- NOT(A), ABS(A), SQRT(A), LOG(A) with two operands where one of them is a scalar -- H=H*i, H=H*5, EXP(A,2), LOG(A,2)
UnaryCP
UnaryCPInstruction
UnaryFEDInstruction
UnaryFrameCPInstruction
UnaryFrameSPInstruction
UnaryMatrixCPInstruction
UnaryMatrixFEDInstruction
UnaryMatrixSPInstruction
UnaryOp
UnaryOperator
UnaryScalarCPInstruction
UnarySketchOperator
UnarySPInstruction
UnifiedMemoryAllocator
UnifiedMemoryManager: Unified Memory Manager - Initial Design Motivation: The Unified Memory Manager, henceforth UMM, will act as a central manager of in-memory matrix (uncompressed and compressed), frame, and tensor blocks within SystemDS control program.
UniformPRNGenerator
Util
UtilFunctions
UtilizationModel
Utils
ValueComparisonFunction: Abstraction for comparison (relational) operators in order to force a proper implementation by all relevant subclasses.
ValueFunction
ValueTrimFormat
VariableCPInstruction
VariableCPInstruction.VariableOperationCode
VariableFEDInstruction
VariableSet
VarStats
Warnings
WeightedCell
WeightedCrossEntropy
WeightedCrossEntropy.WCeMMType
WeightedCrossEntropyR
WeightedDivMM
WeightedDivMM.WDivMMType
WeightedDivMMR
WeightedPair
WeightedSigmoid
WeightedSigmoid.WSigmoidType
WeightedSigmoidR
WeightedSquaredLoss
WeightedSquaredLoss.WeightsType
WeightedSquaredLossR
WeightedUnaryMM
WeightedUnaryMM.WUMMType
WeightedUnaryMMR
WhileProgramBlock
WhileStatement
WhileStatementBlock
WorkerController
WorkerModel
WorkerService
WorkloadAnalyzer
WriterBinaryBlock
WriterBinaryBlockParallel
WriterCompressed
WriterHDF5
WriterHDF5Parallel
WriterMatrixMarket
WriterMatrixMarketParallel
WriterTextCell
WriterTextCellParallel
WriterTextCSV
WriterTextCSVParallel
WriterTextLIBSVM
WriterTextLIBSVMParallel
WriteSPInstruction
WTreeNode: A Node in the WTree, this is used for any nodes that are not the root.
WTreeRoot: The root node of the tree, located at the top of the tree.
Xor
ZipmmSPInstruction