{"id":2692,"date":"2019-11-21T20:12:52","date_gmt":"2019-11-21T11:12:52","guid":{"rendered":"http:\/\/home.hiroshima-u.ac.jp\/~leehs\/?p=2692"},"modified":"2019-11-22T20:24:14","modified_gmt":"2019-11-22T11:24:14","slug":"how-to-compile-fvcom-mpi-mode","status":"publish","type":"post","link":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/?p=2692","title":{"rendered":"How to compile FVCOM4.3 (mpi mode)"},"content":{"rendered":"\n

This is a technical note on how to build and execute FVCOM 4.3 in parallel mode using Intel compilers on CHESS lab HPC at Hiroshima University. Suppose the source code exists in fvcom43\/FVCOM_source<\/code>, and fvcom43\/Examples\/Estuary<\/code> is a selected test case. <\/p>\n\n\n\n

Preparation for METIS<\/h4>\n\n\n\n

METIS library requires to be installed for building for MPI. First check the mpi-selector-menu, then select the mvapich-1.2.0 in the options:<\/p>\n\n\n\n

$ mpi-selector-menu<\/pre>\n\n\n\n
Environmental variables for Intel C compiler should be made as follows:<\/p>\n\n\n\n
exportCC=icc exportCC=icc export CPP=\"icc -E\"<\/pre>\n\n\n\n
Move to the directory of fvcom43\/METIS_source<\/code> where the METIS<\/code> source code exists and extracts the source codes:<\/p>\n\n\n\n
$ tar xf metis.tgz\n$ cd metis <\/pre>\n\n\n\n
Open makefile<\/code> and edit the MOPT<\/code> at line 12 as follows:            <\/p>\n\n\n\n
MOPT     = -O3 -no-prec-div -fp-model fast=2 -xHost<\/pre>\n\n\n\n
The above option of\u00a0MOPT<\/code>\u00a0is the recommended value by the vendor. This\u00a0makefile<\/code>\u00a0should read\u00a0make.inc<\/code>\u00a0in\u00a0fvcom43\/FVCOM_source<\/code>\u00a0that was prepared for\u00a0Series<\/code>\u00a0where\u00a0FLAGs<\/code>\u00a0do not matter. For this purpose,\u00a0include<\/code>\u00a0in this\u00a0makefile<\/code>\u00a0should be corrected as follows:<\/p>\n\n\n\n
include ..\/..\/FVCOM_source\/make.inc<\/pre>\n\n\n\n
Edit make.inc<\/h4>\n\n\n\n
The file of\u00a0make.inc<\/code>\u00a0for\u00a0Series<\/code>\u00a0created in (here<\/a>) requires slight modification for\u00a0FLAG_4<\/code>\u00a0and\u00a0PARLIB<\/code>\u00a0as follows:\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/p>\n\n\n\n
            FLAG_1 =  -DDOUBLE_PRECISION
            FLAG_3 = -DWET_DRY
            FLAG_4 = -DMULTIPROCESSOR
            PARLIB = -lmetis  #-L\/usr\/local\/lib -lmetis
            FLAG_8 =  -DLIMITED_NO
            FLAG_10  = -DGCN
            FLAG_14 = -DRIVER_FLOAT<\/pre>\n\n\n\n
Then comment out the setting for Intel compiler for Series<\/code> in make.inc<\/code> and edit for Intel MPI<\/code> as follows:<\/p>\n\n\n\n
#--------------------------------------------------------------------------\n#  Intel\/MPI Compiler Definitions (CHESS lab HPC)\n#--------------------------------------------------------------------------\n         CPP      = icc -E\n         COMPILER = -DIFORT  \n         CC       = mpiicc\n         CXX      = mpiicpc\n         CFLAGS   = -O3 -no-prec-div -fp-model fast=2 -xHost\n         FC       = mpiifort\n         DEBFLGS  = #-check all -traceback\n         OPT      = -O3  -no-prec-div -fp-model fast=2 -xHost\n#--------------------------------------------------------------------------<\/pre>\n\n\n\n
This setting is for\u00a0FLAT MPI<\/code>\u00a0and the option is the recommended values by the vendor.<\/p>\n\n\n\n
Building METIS<\/h4>\n\n\n\n
To build METIS<\/code>, just make install<\/code> in the source directory.<\/p>\n\n\n\n
$ make install<\/pre>\n\n\n\n
makefile for building FVCOM4.3<\/h4>\n\n\n\n
In building for\u00a0Series<\/code>,\u00a0mod_esmf_nesting.F<\/code>\u00a0was deleted in\u00a0makefile<\/code>\u00a0to avoid link errors. In building for\u00a0MPI<\/code>, this modified makefile is valid while the original one is also no problem.<\/p>\n\n\n\n
Remove the line mod_esmf_nesting.o: ...<\/code> in the makedepends<\/code> file if not done yet.<\/p>\n\n\n\n
Minor correction of a source file (FVCOM4.1)<\/h4>\n\n\n\n
Similarly in the case of Series<\/code> introduced here<\/a>, the comments at line 131 of wreal.F<\/code> should be deleted to avoid warning during compilation:<\/p>\n\n\n\n
# \u00a0 endif \u00a0 \u00a0 \u00a0 !!ice_embedding \u00a0yding<\/pre>\n\n\n\n
In FVCOM4.3, the line 131 is modified already and it does not need to correct it.<\/p>\n\n\n\n
Build FVCOM4.3 for MPI<\/h4>\n\n\n\n
In the directory of FVCOM_source<\/code>, build FVCOM4.3<\/code> for MPI<\/code> as follows:<\/p>\n\n\n\n
$ make<\/pre>\n\n\n\n
Taking some time and the executable file of\u00a0fvcom<\/code>\u00a0should be created.<\/p>\n\n\n\n
Compilation error<\/h4>\n\n\n\n
mod_par.o: In function 'mod_par_mp_add_to_mpi_type_':\nmod_par.f90:(.text+0x67): undefined reference to<\/code> 'mpi_type_get_extent_'\n\nmod_par.o: In function 'mod_par_mp_create_mpi_time_':\nmod_par.f90:(.text+0x282): undefined reference to<\/code> 'mpi_type_get_extent_'\n\nmod_par.o: In function 'mod_par_mp_init_mpi_env_':\nmod_par.f90:(.text+0x58061): undefined reference to<\/code> 'mpi_type_get_extent_'\n\n\/usr\/local\/mvapich-1.2.0_intel-11.1.080\/lib\/libmpich.a(viainit.o): In function MPID_VIA_Finalize':\nviainit.c:(.text+0x44b): undefined reference to<\/code> 'ibv_close_xrc_domain'\n\n\/usr\/local\/mvapich-1.2.0_intel-11.1.080\/lib\/libmpich.a(viainit.o): In function MPID_VIA_Init':\nviainit.c:(.text+0x2b9e): undefined reference to<\/code> 'ibv_open_xrc_domain'\n\n\/usr\/local\/mvapich-1.2.0_intel-11.1.080\/lib\/libmpich.a(viapriv.o): In function create_srq':\nviapriv.c:(.text+0xc8): undefined reference to<\/code> 'ibv_create_xrc_srq'\n\nmake: *** [fvcom] Error 1<\/pre>\n\n\n\n
\n\n\n\n
***\u3000WaveWatch III MPI mode compile error ***<\/h4>\n\n\n\n
It produces the same error with the fvcom4.3 compilation message when compiling the mpi mode.<\/p>\n\n\n\n
\/usr\/local\/mvapich-1.2.0_intel-11.1.080\/lib\/libmpich.a(viainit.o): In function `MPID_VIA_Finalize':\nviainit.c:(.text+0x44b):\u00a0undefined reference to `ibv_close_xrc_domain'<\/strong>\n\/usr\/local\/mvapich-1.2.0_intel-11.1.080\/lib\/libmpich.a(viainit.o): In function `MPID_VIA_Init':\nviainit.c:(.text+0x2b9e):\u00a0undefined reference to `ibv_open_xrc_domain'<\/strong>\n\/usr\/local\/mvapich-1.2.0_intel-11.1.080\/lib\/libmpich.a(viapriv.o): In function `create_srq':\nviapriv.c:(.text+0xc8):\u00a0undefined reference to `ibv_create_xrc_srq'<\/strong>\n\nmake: *** [\/home2\/hslee\/MODELs\/WW3\/ww3_5.16\/exe\/ww3_grid] Error 1<\/pre>\n","protected":false},"excerpt":{"rendered":"
This is a technical note on how to build and execute FVCOM 4.3 in parallel mode using Intel compilers on CHESS lab HPC at Hiroshima University. Suppose the source code exists in fvcom43\/FVCOM_source, and fvcom43\/Examples\/Estuary is a selected test case. Preparation for METIS…
Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[],"class_list":["post-2692","post","type-post","status-publish","format-standard","hentry","category-tool"],"rttpg_featured_image_url":null,"rttpg_author":{"display_name":"hslee","author_link":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/?author=1"},"rttpg_comment":0,"rttpg_category":"Tool<\/a>","rttpg_excerpt":"This is a technical note on how to build and execute FVCOM 4.3 in parallel mode using Intel compilers on CHESS lab HPC at Hiroshima University. Suppose the source code exists in fvcom43\/FVCOM_source, and fvcom43\/Examples\/Estuary is a selected test case. Preparation for METIS...Read more","_links":{"self":[{"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=\/wp\/v2\/posts\/2692","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2692"}],"version-history":[{"count":14,"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=\/wp\/v2\/posts\/2692\/revisions"}],"predecessor-version":[{"id":2727,"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=\/wp\/v2\/posts\/2692\/revisions\/2727"}],"wp:attachment":[{"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2692"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2692"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/home.hiroshima-u.ac.jp\/~leehs\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2692"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}