How to compile FVCOM4.3 (mpi mode)

This is a technical note on how to build and execute FVCOM 4.3 in parallel mode using Intel compilers on CHESS lab HPC at Hiroshima University. Suppose the source code exists in fvcom43/FVCOM_source, and fvcom43/Examples/Estuary is a selected test case.

Preparation for METIS

METIS library requires to be installed for building for MPI. First check the mpi-selector-menu, then select the mvapich-1.2.0 in the options:

$ mpi-selector-menu

Environmental variables for Intel C compiler should be made as follows:

exportCC=icc exportCC=icc export CPP="icc -E"

Move to the directory of fvcom43/METIS_source where the METIS source code exists and extracts the source codes:

$ tar xf metis.tgz
$ cd metis 

Open makefile and edit the MOPT at line 12 as follows:           

MOPT     = -O3 -no-prec-div -fp-model fast=2 -xHost

The above option of MOPT is the recommended value by the vendor. This makefile should read make.inc in fvcom43/FVCOM_source that was prepared for Series where FLAGs do not matter. For this purpose, include in this makefile should be corrected as follows:

include ../../FVCOM_source/make.inc

Edit make.inc

The file of make.inc for Series created in (here) requires slight modification for FLAG_4 and PARLIB as follows:           

            FLAG_1 =  -DDOUBLE_PRECISION
            FLAG_3 = -DWET_DRY
            FLAG_4 = -DMULTIPROCESSOR
            PARLIB = -lmetis  #-L/usr/local/lib -lmetis
            FLAG_8 =  -DLIMITED_NO
            FLAG_10  = -DGCN
            FLAG_14 = -DRIVER_FLOAT

Then comment out the setting for Intel compiler for Series in make.inc and edit for Intel MPI as follows:

#--------------------------------------------------------------------------
#  Intel/MPI Compiler Definitions (CHESS lab HPC)
#--------------------------------------------------------------------------
         CPP      = icc -E
         COMPILER = -DIFORT  
         CC       = mpiicc
         CXX      = mpiicpc
         CFLAGS   = -O3 -no-prec-div -fp-model fast=2 -xHost
         FC       = mpiifort
         DEBFLGS  = #-check all -traceback
         OPT      = -O3  -no-prec-div -fp-model fast=2 -xHost
#--------------------------------------------------------------------------

This setting is for FLAT MPI and the option is the recommended values by the vendor.

Building METIS

To build METIS, just make install in the source directory.

$ make install

makefile for building FVCOM4.3

In building for Seriesmod_esmf_nesting.F was deleted in makefile to avoid link errors. In building for MPI, this modified makefile is valid while the original one is also no problem.

Remove the line mod_esmf_nesting.o: ... in the makedepends file if not done yet.

Minor correction of a source file (FVCOM4.1)

Similarly in the case of Series introduced here, the comments at line 131 of wreal.F should be deleted to avoid warning during compilation:

#   endif       !!ice_embedding  yding

In FVCOM4.3, the line 131 is modified already and it does not need to correct it.

Build FVCOM4.3 for MPI

In the directory of FVCOM_source, build FVCOM4.3 for MPI as follows:

$ make

Taking some time and the executable file of fvcom should be created.

Compilation error

mod_par.o: In function 'mod_par_mp_add_to_mpi_type_':
mod_par.f90:(.text+0x67): undefined reference to 'mpi_type_get_extent_'

mod_par.o: In function 'mod_par_mp_create_mpi_time_':
mod_par.f90:(.text+0x282): undefined reference to 'mpi_type_get_extent_'

mod_par.o: In function 'mod_par_mp_init_mpi_env_':
mod_par.f90:(.text+0x58061): undefined reference to 'mpi_type_get_extent_'

/usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In function MPID_VIA_Finalize':
viainit.c:(.text+0x44b): undefined reference to 'ibv_close_xrc_domain'

/usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In function MPID_VIA_Init':
viainit.c:(.text+0x2b9e): undefined reference to 'ibv_open_xrc_domain'

/usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viapriv.o): In function create_srq':
viapriv.c:(.text+0xc8): undefined reference to 'ibv_create_xrc_srq'

make: *** [fvcom] Error 1

*** WaveWatch III MPI mode compile error ***

It produces the same error with the fvcom4.3 compilation message when compiling the mpi mode.

/usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In function `MPID_VIA_Finalize':
viainit.c:(.text+0x44b): undefined reference to `ibv_close_xrc_domain'
/usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In function `MPID_VIA_Init':
viainit.c:(.text+0x2b9e): undefined reference to `ibv_open_xrc_domain'
/usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viapriv.o): In function `create_srq':
viapriv.c:(.text+0xc8): undefined reference to `ibv_create_xrc_srq'

make: *** [/home2/hslee/MODELs/WW3/ww3_5.16/exe/ww3_grid] Error 1