Raspberry Pi Hacks – Part 1: building MPI for python on a Raspberry Pi cluster

This article assumes that a raspberry Pi cluster is running the latest Raspbian OS and the MPICH2 interface is built and is operational.
(if you haven’t built a cluster and want to , do comment here with your email id/some contact on social media and I can provide the resource and our procedure sheet)
Now the conventional way to install the MPI for python (which is called mpi4py) will not work. That is using the command:

 sudo apt-get install python-mpi4py

will install the mpi4py, but when its run to execute, it fails or crashes. This will be observed only by the developers who have installed MPICH2 interface in their cluster. The reason why it crashes is, unknowingly,  the command above will install instances of openMPI. OpenMPI is a different interface that clashes with the one that is already installed, MPICH2. A system is usually designed to run only one interface and when there are multiple instances running, it leads to a system failure.

To avoid this failure and the tedious task to restore the operating system back to its previous state, a work around exists. This work around is to build the mpi4py manually on each of the node in the cluster.

The following are the steps to build it:

1) download the mpi4py package.

      curl –k –O https://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz

      We can use wget instead of curl but I couldn’t find an option that bypasses the certificate    issue that hasn’t been resolved by the website maintenance team.

2) Unpack it. And change to that folder.

       tar –zxf mpi4py-1.3.1.tar.gz
cd mpi4py-1.3.1.tar.gz

3) Before the build is started, it is important to make sure that all the python development tools     are available.

This ensures that many important header files like Python.h is present and can be used by the build function.

(This step can be skipped if the python development tools are already installed)

         sudo apt-get update –fix-missing

         sudo apt-get install python-dev

4) Now, we can build the package.

           cd mpi4py-1.3.1.tar.gz

           sudo python setup.py build  –mpicc=/usr/local/mpich2/bin/mpicc

    few things that have to be noted here:

  •        The option –mpicc is used to provide the build file the location of the MPI compiler.
  •        The option –mpicc has to be used only if the location of that compiler doesn’t already exist in the system path.
  •        The path /usr/local/mpich2/bin/mpicc is the location on my node, where the mpich2 is built. It might not be the same for everyone and so that has to be replaced with the path, where mpicc is located in that system.

The only thing now left do is to install the build.to install change working directory to mpi4py:

cd mpi4py

After shifting to this directory, run the command :

sudo python setup.py install

Once this is done, repeat the process in every other node in the cluster. Then the demo program helloworld.py can be run to test if mpi4py is installed on all the node successfully and is running correctly.

If the nodes of the cluster aren’t already built, then the easier way to do it would be to perform the above procedure on one node and read the entire image of the OS and write it into the SD cards of each of the other node. This would eliminate building of mpi4py package on each node individually.


Hack 1 : Running a C program without main()

This  post is a simple example of deception. It shows how a programmer can defy the very important rule of having a main() in c program and still make the program run. This illustrates the concept on a simple program though it can be scaled to  much bigger and more complex programs.


#define decode(s,t,u,m,p,e,d) m##s##u##t
#define begin decode(a,n,i,m,a,t,e)

int begin()

printf(” hello “);

This program runs without main().


Here we are using preprocessor(a program which processes the source code before compilation.) directive #define with arguments to give an impression that the program runs without main. But in reality it runs with a hidden main function.

The ‘##‘ operator is called the token pasting or token merging operator. That is we can merge two or more characters with it.

In the 2nd line of the program-

#define decode(s,t,u,m,p,e,d) m##s##u##t

What is the preprocessor doing here. The macro decode(s,t,u,m,p,e,d) is being expanded as “msut” (The ## operator merges m,s,u & t into msut). The logic is when you pass (s,t,u,m,p,e,d) as argument it merges the 4th,1st,3rd & the 2nd characters(tokens)

Now look at the third line of the program –

#define begin decode(a,n,i,m,a,t,e)

Here the preprocessor replaces the macro “begin” with the expansion decode(a,n,i,m,a,t,e). According to the macro definition in the previous line the argument must be expanded so that the 4th,1st,3rd & the 2nd characters must be merged. In the argument (a,n,i,m,a,t,e) 4th,1st,3rd & the 2nd characters are ‘m’,’a’,’i’ & ‘n’.

So the third line “int begin” is replaced by “int main” by the preprocessor before the program is passed on for the compiler. That’s it…

So actually C program can never run without a main() . We are just disguising the main() with the preprocessor, but actually there exists a hidden main function in the program.