Ab Initio implements parallelism in mainly 3 ways:
Data parallelism : data is divided among many partitions known as multi-files. During processing, each partition is processed in parallel.
Component parallelism : multiple components are run in parallel. Components execute simultaneously on different branches of a graph.
Pipeline parallelism : when a record is processed in one component and a previous record is being processed in another components. Operations like sorting and aggregation break pipeline parallelism.
Data parallelism : data is divided among many partitions known as multi-files. During processing, each partition is processed in parallel.
Component parallelism : multiple components are run in parallel. Components execute simultaneously on different branches of a graph.
Pipeline parallelism : when a record is processed in one component and a previous record is being processed in another components. Operations like sorting and aggregation break pipeline parallelism.
No comments:
Post a Comment