Using CMAKE with PETSc or MPI

A large part of developing software is getting the build process to work. As a new student in RPI's Scientific Computation Research Center (SCOREC) I started trying to figure out how to build some of our tools so that we can interface them with new projects.

More specifically I have written a Euler-Bernoulli and Timoshenko Beam finite element solver using PETSc. The goal is to interface this codebase with a multiscale fiber network model that SCOREC is working on for the NIH.

Although my original goal was to get CMAKE to work with PETSc the eventual solution also pertains to compiling any software that uses MPI with CMAKE.

Traditional PETSc Makefile

The easiest way to get started compiling your program is by using the compiling commands included with PETSc. My makefile looks like this:

CPPFLAGS         = -std=c++0x

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules

all: beam

beam: beam_fem.o element.o node.o material.o reader.o chkopts
    -${CLINKER} beam_fem.o element.o node.o material.o reader.o ${PETSC_KSP_LIB} -o beam_fem.out

beam_fem.o: ./src/beam_fem.cpp ./include/element.h ./include/node.h
    -$(PETSC_COMPILE) -c ./src/beam_fem.cpp -I ./include

element.o: ./src/element.cpp ./include/element.h ./include/node.h
    -$(PETSC_CXXCOMPILE) -c ./src/element.cpp -I ./include

node.o: ./src/node.cpp ./include/node.h
    -$(PETSC_CXXCOMPILE) -c ./src/node.cpp -I ./include

material.o: ./src/material.cpp ./include/material.h
    -$(PETSC_CXXCOMPILE) -c ./src/material.cpp -I ./include

reader.o: ./src/reader.cpp ./include/reader.h
    -$(PETSC_CXXCOMPILE) -c ./src/reader.cpp -I ./include

.PHONY: print_vars
print_vars:
    -@echo "PETSC_DIR: $(PETSC_DIR)"
    -@echo "CLINKER: $(CLINKER)"
    -@echo "CXX_COMPILE: $(PETSC_CXXCOMPILE)"
    -@echo "C_COMPILE: $(PETSC_COMPILE)"

You can see that I use the CLINKER and PETSC_CXXCOMPILE commands. These are defined in the petsc variables file that is included at the top of the makefile. I also have a .PHONY rule which lets me print out information about what the PETSc compile commands are actually doing.

CMAKE Approach

CMAKE is actually quite easy to use with PETSc, but there are some stumbling points for someone who is new to using CMAKE with a library that needs the mpi libraries to compile.

The PETSc FAQs suggest getting the FindPETSc.cmake module from Jed Brown's (a PETSc developer) cmake-modules repository, and to use the CMakeLists.txt from his dohp project an example. These are both excellent resources, although for a newcomer there is an extremely high learning curve.

I have included my CMakeLists.txt below, although the biggest stumbling point is not here...

If you have ever used PETSc you know that it uses MPI to perform computations in parallel. If you were going to compile a MPI library by hand you would use the MPICC/MPICXX wrappers to compile, but that's not the "right" way to do it in CMAKE is it?

It turns out that there seems to be some disagreement in the CMAKE community some say that utilities like FindMPI.cmake is only way to correctly compile a MPI livrary. Others including Jed Brown believe that "FindMPI.cmake is less reliable than the MPI wrappers". This sentiment can also be seen on the mailing list.

The final solution comes from the CMAKE FAQs...just define your c and c++ compiler as the mpicc/mpicxx scripts.

The following examples assumes that you have installed mpi along with PETSc.

Method 1

Set the CC and CXX variables before compiling with cmake.

export CC=$PETSC_DIR/$PETSC_ARCH/bin/mpicc
export CXX=$PETSC_DIR/$PETSC_ARCH/bin/mpicxx
mkdir build
cd build/
cmake ..

Method 2

Use the cmake -D flag to set the compiler options. Note that you only have to supply these command line options the first time you call cmake. All subsequent builds will know to use the appropriate compiler.

mkdir build
cd build/
cmake .. -D CMAKE_C_COMPILER=$PETSC_DIR/$PETSC_ARCH/bin/mpicc -D CMAKE_CXX_COMPILER=$PETSC_DIR/$PETSC_ARCH/bin/mpicxx

CMakeLists.txt

cmake_minimum_required (VERSION 2.6.3)
project (BEAM)

list (APPEND CMAKE_MODULE_PATH "${BEAM_SOURCE_DIR}/../cmake-modules")

find_package (PETSc REQUIRED)

message (STATUS "PETSC COMPILER ${PETSC_COMPILER}")

option (BEAM_ALL_WARNINGS       "Compile with all warnings (-Wall)"           ON)
option (BEAM_WERROR             "Treat warnings as errors (-Werror)"          OFF)

add_definitions (-std=c++0x)
message (STATUS "CPP COMPILER ${PETSC_COMPILER}")

if (BEAM_ALL_WARNINGS)
    add_definitions ("-Wall")
endif ()
if (BEAM_WERROR)
  add_definitions (-Werror)
endif ()

include_directories ("${BEAM_SOURCE_DIR}/include" "${BEAM_BINARY_DIR}/include" ${PETSC_INCLUDES})
add_definitions (${PETSC_DEFINITIONS})
add_definitions (-g)
message (STATUS "PETSC_DEFINITIONS ${PETSC_DEFINITIONS}")

set (BEAM_DEPENDENT_LIBRARIES "${PETSC_LIBRARIES}")

add_subdirectory (include)
add_subdirectory (src)

Conclusions

I hope that if you have stumbled upon this webpage that I have saved you a little bit of time. I spent a few days going around in circles trying to make cmake work with my project and PETSc. You can see from my pull request in FindPETSc.cmake that I originally thought I had to link the mpi libraries manually. This was only after FindMPI.cmake had failed me.

I have since changed my pull request to add a comment giving instructions on how to set the compiler in CMAKE. If you think that would be a useful addition to FindPETSC.cmake leave a comment on the pull request.

This process was definitely an eye opening experience as it was my first time dealing with CMAKE, and trying to commit to an open source project.