Do anyone know if there exist any widely used parallel computing programming paradigms/models not based on a master/slave concept?
Parallel computing programming paradigms/models not based on a master/slave concept?
parallel computing;hpc
For parallel models I see them as communication-based or work decomposition-based:Communication-based modelsShared memory: implicit communication via sharing memory.Message passing: explicit communication by passing messages between parallel programs.Work partitioning modelsDomain decomposition. Problem domain is divided into sub-domains, problem is solved by solving all the sub-problems.Task-based decomposition: compute work is organised in the form of tasks (it could also be objects). Tasks could be homogeneous or each task could perform different work. Examples for this can be seen in OpenMP, Intel thread building blocks, starPU/Quark.Implicit dependencies: constructs are used to implicitly represent relations between tasks, think of OpenMP and its #pragma omp parallel for.Explicit dependencies: relations between task are the core of the representation. A good discussion is given hereParadigms examplesSingle program multiple data (SPMD): Domain decomposition.Parallel functional programming: implicit task model.Master-slave paradigm: explicit task-based model.Divide and conquer: domain decomposition.GPU computing: most commonly domain decomposition.Work pool and/or parallel queues: task-based model or domain decomposition.With the heterogeneity and hierarchical structure of parallel computing architectures currently available, programs tend to mix models and paradigms to better match the architecture. Consider for instance message passing between nodes while using shared memory intra-node, that would match with a divide and conquer approach to distribute work across the nodes while locally each sub-domain could be solved using SPMD over multiple cores, a GPU, or both.