High Performance Computing Service

Compiling and development


There are both GNU and Intel compilers on Darwin (in several versions depending on the currently loaded modules. The following table summarises the available compiler commands on the cluster:

Recommended optimisation flag -O3-O3 -ipo -xHost

For best performance, use of the Intel compilers is recommended. Please see the online documentation for the Intel compilers and the Math Kernel library. For more information about optimisation for GNU, please refer to the respective man and info pages (or here).

The commands referred to above should be used directly for single processor or OpenMP parallel applications. For MPI parallel applications it is preferable to use the MPI-specific wrappers which automatically locate the correct MPI header and library files. These wrappers are available in the default user environment and are:

Command mpicc mpif77mpif90mpicxx

These commands are wrappers around the Intel compilers (unless overridden, see below) and ensure that the correct MPI include and library files are used. Since they are wrappers, the same optimisation flags can be used as with the standard Intel compilers.

Typically, applications use a Makefile that has to be adapted for compilation. (Please refer to the application's documentation in order to adapt the Makefile for a Linux/x86_64 cluster.) Frequently, it is sufficient to choose a Makefile specifically for a Linux MPI environment and to adapt the CC & FC parameters in the Makefile, which should point to mpicc and mpif77 (or mpif90 in the case of Fortran90 code) respectively.

By default mpicc/mpif77 etc use the Intel compilers. To use another compiler, e.g. the GNU compiler, the following flags will override the default behaviour:

mpicc  -cc=gcc
mpif77 -fc=gfortran
mpif90 -f90=gfortran
mpicxx  -cxx=g++

Parallel programming: an MPI example

The sample code below contains the complete communications skeleton for a dynamically load balanced master/slave application. Following the code is a description of some of the functions necessary for writing typical parallel applications.

#include <mpi.h>
#define WORKTAG     1
#define DIETAG     2
main(argc, argv)
int argc;
char *argv[];
	int         myrank;
	MPI_Init(&argc, &argv);   /* initialize MPI */
	MPI_COMM_WORLD,   /* always use this */
	&myrank);      /* process rank, 0 thru N-1 */
	if (myrank == 0) {
	} else {
	MPI_Finalize();       /* cleanup MPI */

	int	ntasks, rank, work;
	double       result;
	MPI_Status     status;
	MPI_COMM_WORLD,   /* always use this */
	&ntasks);          /* #processes in application */
* Seed the slaves.
	for (rank = 1; rank < ntasks; ++rank) {
		work = /* get_next_work_request */;
		MPI_Send(&work,         /* message buffer */
		1,              /* one data item */
		MPI_INT,        /* data item is an integer */
		rank,           /* destination process rank */
		WORKTAG,        /* user chosen message tag */
		MPI_COMM_WORLD);/* always use this */

* Receive a result from any slave and dispatch a new work
* request work requests have been exhausted.
 	work = /* get_next_work_request */;
	while (/* valid new work request */) {
		MPI_Recv(&result,       /* message buffer */
		1,              /* one data item */
		MPI_DOUBLE,     /* of type double real */
		MPI_ANY_SOURCE, /* receive from any sender */
		MPI_ANY_TAG,    /* any type of message */
		MPI_COMM_WORLD, /* always use this */
		&status);       /* received message info */
		MPI_Send(&work, 1, MPI_INT, status.MPI_SOURCE,
		work = /* get_next_work_request */;
* Receive results for outstanding work requests.
	for (rank = 1; rank < ntasks; ++rank) {
		MPI_Recv(&result, 1, MPI_DOUBLE, MPI_ANY_SOURCE,
* Tell all the slaves to exit.
	for (rank = 1; rank < ntasks; ++rank) {
		MPI_Send(0, 0, MPI_INT, rank, DIETAG, MPI_COMM_WORLD);

	double              result;
	int                 work;
	MPI_Status          status;
	for (;;) {
		MPI_Recv(&work, 1, MPI_INT, 0, MPI_ANY_TAG,
		MPI_COMM_WORLD, &status);
* Check the tag of the received message.
		if (status.MPI_TAG == DIETAG) {
		result = /* do the work */;
		MPI_Send(&result, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD);

Processes are represented by a unique rank (integer) and ranks are numbered 0, 1, 2, ..., N-1. MPI_COMM_WORLD means all the processes in the MPI application. It is called a communicator and it provides all information necessary to do message passing. Portable libraries do more with communicators to provide synchronisation protection that most other systems cannot handle.

Enter and Exit MPI

As with other systems, two functions are provided to initialise and clean up an MPI process:

	MPI_Init(&argc, &argv);
	MPI_Finalize( );

Who Am I? Who Are They?

Typically, a process in a parallel application needs to know who it is (its rank) and how many other processes exist. A process finds out its own rank by calling:

MPI_Comm_rank( ): 
   Int myrank;
   MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

The total number of processes is returned by MPI_Comm_size( ):

   int nprocs;
   MPI_Comm_size(MPI_COMM_WORLD, &nprocs);

Sending messages

A message is an array of elements of a given data type. MPI supports all the basic data types and allows a more elaborate application to construct new data types at runtime. A message is sent to a specific process and is marked by a tag (integer value) specified by the user. Tags are used to distinguish between different message types a process might send/receive. In the sample code above, the tag is used to distinguish between work and termination messages.

MPI_Send(buffer, count, datatype, destination, tag, MPI_COMM_WORLD);

Receiving messages

A receiving process specifies the tag and the rank of the sending process. MPI_ANY_TAG and MPI_ANY_SOURCE may be used optionally to receive a message of any tag and from any sending process.

MPI_Recv(buffer, maxcount, datatype, source, tag, MPI_COMM_WORLD, &status);

Information about the received message is returned in a status variable. The received message tag is status. MPI_TAG and the rank of the sending process is status.MPI_SOURCE. Another function, not used in the sample code, returns the number of data type elements received. It is used when the number of elements received might be smaller than maxcount.

MPI_Get_count(&status, datatype, &nelements);

There are many other, more exotic functions in MPI, but all can be built upon those presented here so far.