This is a technical note on how to build and execute FVCOM 4.3 in parallel mode using Intel compilers on CHESS lab HPC at Hiroshima University. Suppose the source code exists in fvcom43/FVCOM_source, and fvcom43/Examples/Estuary is a selected test case.
Preparation for METIS
METIS library requires to be installed for building for MPI. First check the mpi-selector-menu, then select the mvapich-1.2.0 in the options:
$ mpi-selector-menu
Environmental variables for Intel C compiler should be made as follows:
exportCC=icc exportCC=icc export CPP="icc -E"
Move to the directory of fvcom43/METIS_source where the METIS source code exists and extracts the source codes:
$ tar xf metis.tgz $ cd metis
Open makefile and edit the MOPT at line 12 as follows:
MOPT = -O3 -no-prec-div -fp-model fast=2 -xHost
The above option of MOPT is the recommended value by the vendor. This makefile should read make.inc in fvcom43/FVCOM_source that was prepared for Series where FLAGs do not matter. For this purpose, include in this makefile should be corrected as follows:
include ../../FVCOM_source/make.inc
Edit make.inc
The file of make.inc for Series created in (here) requires slight modification for FLAG_4 and PARLIB as follows:
FLAG_1 = -DDOUBLE_PRECISION
FLAG_3 = -DWET_DRY
FLAG_4 = -DMULTIPROCESSOR
PARLIB = -lmetis #-L/usr/local/lib -lmetis
FLAG_8 = -DLIMITED_NO
FLAG_10 = -DGCN
FLAG_14 = -DRIVER_FLOAT
Then comment out the setting for Intel compiler for Series in make.inc and edit for Intel MPI as follows:
#-------------------------------------------------------------------------- # Intel/MPI Compiler Definitions (CHESS lab HPC) #-------------------------------------------------------------------------- CPP = icc -E COMPILER = -DIFORT CC = mpiicc CXX = mpiicpc CFLAGS = -O3 -no-prec-div -fp-model fast=2 -xHost FC = mpiifort DEBFLGS = #-check all -traceback OPT = -O3 -no-prec-div -fp-model fast=2 -xHost #--------------------------------------------------------------------------
This setting is for FLAT MPI and the option is the recommended values by the vendor.
Building METIS
To build METIS, just make install in the source directory.
$ make install
makefile for building FVCOM4.3
In building for Series, mod_esmf_nesting.F was deleted in makefile to avoid link errors. In building for MPI, this modified makefile is valid while the original one is also no problem.
Remove the line mod_esmf_nesting.o: ... in the makedepends file if not done yet.
Minor correction of a source file (FVCOM4.1)
Similarly in the case of Series introduced here, the comments at line 131 of wreal.F should be deleted to avoid warning during compilation:
# endif !!ice_embedding yding
In FVCOM4.3, the line 131 is modified already and it does not need to correct it.
Build FVCOM4.3 for MPI
In the directory of FVCOM_source, build FVCOM4.3 for MPI as follows:
$ make
Taking some time and the executable file of fvcom should be created.
Compilation error
mod_par.o: In function 'mod_par_mp_add_to_mpi_type_': mod_par.f90:(.text+0x67): undefined reference to'mpi_type_get_extent_' mod_par.o: In function 'mod_par_mp_create_mpi_time_': mod_par.f90:(.text+0x282): undefined reference to'mpi_type_get_extent_' mod_par.o: In function 'mod_par_mp_init_mpi_env_': mod_par.f90:(.text+0x58061): undefined reference to'mpi_type_get_extent_' /usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In functionMPID_VIA_Finalize': viainit.c:(.text+0x44b): undefined reference to'ibv_close_xrc_domain' /usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In functionMPID_VIA_Init': viainit.c:(.text+0x2b9e): undefined reference to'ibv_open_xrc_domain' /usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viapriv.o): In functioncreate_srq': viapriv.c:(.text+0xc8): undefined reference to'ibv_create_xrc_srq' make: *** [fvcom] Error 1
*** WaveWatch III MPI mode compile error ***
It produces the same error with the fvcom4.3 compilation message when compiling the mpi mode.
/usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In function `MPID_VIA_Finalize': viainit.c:(.text+0x44b): undefined reference to `ibv_close_xrc_domain' /usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viainit.o): In function `MPID_VIA_Init': viainit.c:(.text+0x2b9e): undefined reference to `ibv_open_xrc_domain' /usr/local/mvapich-1.2.0_intel-11.1.080/lib/libmpich.a(viapriv.o): In function `create_srq': viapriv.c:(.text+0xc8): undefined reference to `ibv_create_xrc_srq' make: *** [/home2/hslee/MODELs/WW3/ww3_5.16/exe/ww3_grid] Error 1
