MPI and Distributed Computing
MPI — The way we use many processors, processes, or computers to solve the world’s problems, maybe…
So, provided you’ve completed the tasks on openMP parallelisation seems pretty chill, it also might seem bottlenecked by the number of cores your computer has, which for largely upscaled problems, one extremely large CPU still cannot outperform many connected individual processors or standalone computers even. The Message Passing Interface MPI is the way separate processes can pass messages to each other in order to complete the execution of a program in parallel. This is required because not every node (core) has access to the same memory as all the other nodes, ie, the memory is distributed, much like the computing load. In this blog you will be introduced to MPI and implement it!
A computational cluster can often have a hybrid structure, where certain nodes have shared memory that is isolated from other clusters of nodes. Importantly though, without MPI a supercomputer cluster would not be feasible, the ability to link up multiple processors gives massive scalability to parallelise a problem.
With all that said, let’s hop into some basics of MPI!
Distributed computing uses an implementation of MPI such as openMPI, MPICH or MVAPICH
These are all libraries that are built on the MPI standard. There is a large range of topics that can be covered within distributed computing under the MPI umbrella; point-to-point communication, collective communication, non-blocking communication and others. In this post the basics of point to point communication will be covered. However, much like in the openMP blog post released previously, distributed computing is capable of performing reductions and other bulk procedures for a parallel algorithm.
Now! For the code!
First Up let’s get onto our AWS instance, use the previous blog posts if you’ve forgotten how to get in, or are just joining us here. The small cloud instance we created earlier is capable of performing parallelised code using MPI. Later on we can also use MPI to use the instance coupled with our local machine to make a mini-cluster to distribute the programs across multiple machines.
First step is to install MPICH or openMPI, in this blog I will be using MPICH.