Env_system is the file to set the environment variables that define your computer system, directory structures, etc. Most variables have a value set in the file as available from the HIRLAM export version); this is referred to as the 'default' value. The structure of this file is such that the further down in the file, the less likely you will have to change the defaults for your particular system, unless your system is complicated (e.g. a peculiar archiving system). The file is usually stored in a place where it can be used by all people running HIRLAM on your system. By convention, its name is Env_system_ where is its version number (e.g. 5.0.2). The sample file provided with the HIRLAM system contains sections that are valid for the computer system at ECMWF. Those sections are under the condition 'if [ $COMPCENTRE = ECMWF ] '. You should complete the `else' parts for your own situation. It is assumed that thare are n hosts, numbered from 0 to n-1. HOST0 is the host where all files are kept. Usually it is also the host on which mini-SMS is running. 0. This file version and full path ENV_SYS_VN is the version number of this file. ENV_SYS_HOST0 is its full path on HOST0. (As of version 5.1.1 ENV_SYS_HOST0 is not used anymore) 1. Identify and describe your system Unless you are at ECMWF, set COMPCENTRE to something local (in fact, why not use `local'?). 1.0 define hosts and the location of this file on each host Define the host names HOST0, ..., HOSTm of the machines that you are going to use for HIRLAM. Use the names with which the hosts are identified on your network. Because file Env_system will be used on all hosts, you should give its full path on each host. On HOSTn, that path is found in ENV_SYS_HOSTn. (As of version 5.1.1 ENV_SYS_HOSTn is not used anymore) On each host, there should be large volatile file system ($TEMP, $SCRATCH, or ...). In the sample Env_system this file system is identified by lvfshostn for HOSTn. (It does no harm if the file system is not volatile; and dependent on the way that HIRLAM is using the hosts, not all hosts need to have a large file system). (At ECMWF, code for vpp is cross-compiled on ecgate1; the directories to search for include files are set in this section). SMSNODE is the node on which a particular job, after submission by SMS, will be run. For those jobs it will be set by mini-SMS. But HIRLAM jbos that are not submitted by mini-SMS, including mini-SMS itself, have to know it in a different way. Set SMSNODE to this host (probably SMSNODE=${SMSNODE-`hostname`} is the proper solution). As from version 5.1.2, jobs that need knowledge about the file systems on remote hosts (in general jobs to copy in- or output files), will rerun Env_system with SMSNODE temporarily set to the remote host. To identify this temporary overriding of SMSNODE, the local variable bvn is then set to "5.1.2". The home file system on HOST0 must be NFS mounted on the other hosts. In the sample Env_system it is assumed that the NFS mount has the same directory structure, but with ws_prefix prefixed. 1.1 Set system hard/software The main usage of the variables CS (Compiling System) and OS (operating System) is by the makefiles to compile and load the HIRLAM libraries and executables. These makefiles can be extracted from the reference system, after installation, by typing in an experiment working directory (HL_WD, see below): $HL_SCR/Hirlam co scripts Makefile # for libraries $HL_SCR/Hirlam co scripts Makefile_x # for executables In here, $HL_SCR points to the HIRLAM scripts directory, (see below). If your operating system and compiling system have been implemented there, you should set OS and CS accordingly; if not, you should modify these makefiles to implement your system, see section 8.1 below. Currently, the following exist for CS: ALPHA, CONVEX, CRAY, FUJITSU, G77, SGI77, SGI90 For all, except GNU (G77), a proper default value for OS is set. LSERIAL and NHORPH: NHORPH defines the number of grid points to be processed in a single invocation of the physics routines. Large values of NHORPH make the physics package use much memory; small values may result in more efficient cache usage but at the expense of a large subroutine calling overhead because the physics routines will have to be called often. The following can be used as general guidelines for optimum performance: On vector machines: NHORPH should be set to a reasonable vector length; on most machines, the longer the better, but be aware that very large values may cost you memory. Try 1024 or 512. Set LSERIAL to .true. on vector machines. On non-vector machines, set LSERIAL to .false., and do not set NHORPH: if NHORPH is not set, HIRLAM will calculate a reasonable value for you, Set NBITS to the number of bits (precision with which to run HIRLAM). Choose from 32 or 64. Normally, 32 bits should suffice. Some computer systems do not support 32 bits calculations, so NBITS is set to 64 on those. If you decide to use 64 bits arithmetic for higher precision (although no case is known where this was needed), note the following: HIRLAM assumes that word lengths are equal for reals and integers. Not all computer systems support that through Fortran compiler options. On such a system, you are limited in choice of word length. The HIRLAM system assumes by default that your computer system is big-endian. You can modify this by the option `ENDIAN'. Set it to `LITTLE' e.g. on PCs with Linux. However, do not do so on ALPHA if used with its native compiler, because that compiler is invoked with an option to simulate big-endian IO. Beware: A wrong choice may go unnoticed until you start exchanging files with other computers! If your computer configuration is not simple you may profit from using the variable COMPCENTRE to isolate specific changes in the scripts you had to make for your own computer configuration. For example COMPCENTRE=ECMWF is used to isolate the peculiarities required to access the ECMWF data bases. 1.2 Root directories of the HIRLAM system The variables that point to the root directories of the HIRLAM system are set. These directories are created and maintained by your local HIRLAM system manager. (S)He should be the only one with write access in them. See the description under 1.3 for the several file systems involved. 1.3 Root directories of HIRLAM experiments The variables that point to the root directories for your experiments with the HIRLAM system are set. These directories are assumed to be writable by you. Most variables are assigned some default value. These defaults are based on the assumption that there are three or four different file systems on your computer system: FPFS: A fast permanent file system. This system contains frequently used files, for example the HIRLAM scripts to be included in your search path; or files that are precious to you, for example a Fortran routine developed by you. This file system may be small. For files owned by you, the default is your home directory. For the files from the HIRLAM system, there is no reasonable default, but it could for example be the home directory of your local HIRLAM system manager. LPFS: A large permanent file system, possibly slow. This may be a system on virtual disc, or even on a separate archiving configuration. This system is used to store the HIRLAM system, and to store the (usually large) HIRLAM output files. Default is the same as the fast permanent file system, but if that system is small, you should expect to run into storage problems if you do not change this default. FVFS: A fast volatile file system. This file system is used as a working directory by HIRLAM. It may also be used to keep a quickly accessible copy of files in the large permanent file system. Default is a directory $TEMP for your own files. If the fast permanent file system is big enough to keep the HIRLAM system, there is no need to take copies of the HIRLAM system files. Hence, in accordance with the default for the large permanent file system, default for the HIRLAM system on the large, fast, volatile file system is the fast permanent file system, and no copies are taken. The fourth file system is intended to hold the html documentation of the HIRLAM system. This may be a unix system. But because the html documentation system operates independently from the HIRLAM system itself, it may also be installed on a different system (where, of course, a browser should be available). Default is the same as the fast permanent file system. In the sequel at several locations the defaults used by this script assume that the different file systems can all be accessed by standard unix methods. This may not always be the case (e.g. if the large permanent file system is remote, or if the html documentation is on a PC-like system). Comments indicate where you should deviate from the defaults for this reason. But otherwise, in the sequel, you are advised to follow the defaults, and this description assumes you do indeed. 2. Set HIRLAM variables A range of environment variables is set, based on the root directories as set in the previous section. 2.1 HIRLAM system location HL_REF_FS is the actual root of the HIRLAM system. It contains subdirectories for the different types of files in the HIRLAM system (e.g. src for sources, dat for data). Because of its fairly big size it is in LPFS. When running the system, unix accessible copies are made to the volatile system HL_REF_CP. If, however, LPFS is unix-accessible and fast, there is no need for separate copies. In that case, HL_REF_CP may point to the same directory as HL_REF_FS, which will avoid the creation of copies. Other files of the HIRLAM system need to be permanently on unix-accessible, fast file systems. Notably: HL_UTIL contains in its sole subdirectory bin the GNU utilities for make and RCS. It is assumed that they have been installed before installing the HIRLAM system. Set HL_UTIL to point to their directory. If there are more hosts, HL_UTIL may be host-specific, because it may contain binaries. It is used to set the path (see section 3 below, so make sure that the path there points to the directory valid for the current host). It is also used in a few other places, notably when installing the utilities using HIRLAM facilities to do so; but this was never needed at ECMWF, and that is why there is no host-dependency in the provided sample of Env_system. The local HIRLAM system manager will also have to install in this directory the program to add/subtract dates and times, mandtg. The HIRLAM scripts are in HL_SCR, default ~hirlam/hl_scr, where ~hirlam is the HIRLAM root directory in FPFS. HIRLAM maintains a set of resource files in HL_RESOURCES. For each file in the possibly remote directory HL_REF_FS there is a resources file to record the last-change date of the corresponding file in HL_REF_FS, which is used by the make process to decide on (re)compilation of libraries. Also, HL_RESOURCES contains HIRLAM help files, in subdirectory doc. But you can change the default location of these files by setting HL_SDOC. HIRLAM standard output and error files often are in HTML format, containing links to relevant sections of the documentation available over the HIRLAM User Group Extranet. the address of this Extranet is given in HL_DOC. For the decoding of observations, HIRLAM uses BUFR tables. The unix-accessible copies are kept in BUFRTAB_DIR. Note that the system requires that the last two directory components in BUFRTAB_DIR are dat/bufrtables. There is no way to avoid this requirement. If you have BUFR tables somewhere on your system, but in a directory that does not match this requirement, it is suggested you make a symbolic link from /dat/bufrtables to the directory containing the BUFR tables, or you copy the BUFR tables to such a directory (you only need to copy the ASCII tables). The HTML documentation provided with the HIRLAM sources is supposed to go to the directory pointed to by HL_HTML. 2.3 Experiment identifier and directories You should identify your HIRLAM experiments by a (preferably) 3 character identifier, environment variable EXP. The script Env_system uses this to define a number of variables pointing to directories private to your experiment. Hence, EXP must have been set before Env_system is invoked. The directories are: HL_WD: Your working directory when setting up your HIRLAM experiment. It should contain modified sources and scripts. Because you may have spent a lot of time to create these, these files should be in permanent storage, hence FPFS. HL_WD may be needed on all hosts, and should hence be available on all hosts as NFS mount. In the provided sample file on Env_system, HL_WD=$home/$EXP, where $home contains a host dependency through ws_prefix. HL_DATA: The working directory for the HIRLAM experiment. It contains the large files (e.g. executables) to be preserved during an experiment, but which can be reconstructed fairly simply. Hence, in LVFS. There must be a HL_DATA on each host. HL_DATA_HOSTn is the full path of HL_DATA on HOSTn. If executing on HOSTn, HL_DATA is set to HL_DATA_HOSTn. (as of version 5.1.2, HL_DATA_HOSTn is not used anymore. In stead, HL_DATA on a remote host is obtained by rerunning Env_system with SMSNODE temporarily set to the remote host HOSTn). HL_LIB: (Introduced with version 5.1.2; older versions used HL_DATA) The directory to hold the scripts, objects and executables for an experiment. Also HL_LIB is sometimes needed on a remote host, and is then obtained by rerunning Env_system with a temporary override of SMSNODE. HL_EXP: The directory for the experiment files that you want to store for longer time, eg. for archiving forecasts. So HL_EXP is in LPFS. It is only used from HOST0. In this section, also the local variable hl_hs is set; it points to a directory of scripts that you want to use in preference over the reference system files, for all your experiments. So this directory is private to you but not to your experiments. Probably you will need it on all hosts, but if $home is set properly on each host, this should be OK. 2.4 Pseudo-environment file Most HIRLAM scripts start with the in-line execution of the file with full pathname SETENV, default $HL_WD/ENVIRONMENT. This mechanism makes the variables defined by SETENV or by the files invoked by SETENV available to those scripts. The file is needed on all hosts, ubt if HL_WD is set properly, this should be no problem. 3. Path The following components are appended to your path, in this order: HL_WD/scripts, hl_hs, HL_LIB/scripts, HL_UTIL/bin, and HL_SCR. For a local installation, it may be useful to have an additional entry in the path, to point at local scripts that should replace scripts from the HIRLAM reference system. The directory of those scripts should be in between hl_hs and HL_DATA/scripts, so as to allow the user to override the local alternatives by placing scripts in HL_WD/scripts. or hl_hs. You may find it convenient to have some of the HIRLAM directories in your login search path. If you do so, you may disturb the order in which the directories should be searched. To avoid that, you might want to replace the `$PATH" on the right hand side by something more restrictive, as you see illustrated in the ECMWF sections. 4. Utilities 4.1 for the main unix platform Because HIRLAM uses several GNU extensions to make, and because many native RCS installations do not check out (co) Fortran files properly, variables MAKE and CO define make and co with their full path, and these should be GNU compatible. Not all systems support the recurrence flag '-p' on mkdir. On those systems, choose an alternative MKDIR. As a last resort, you may try the script Mkdir available from the HIRLAM export file (make it executable, and store it in $HL_UTIL/bin). If you do so, set MKDIR to $HL_UTIL/bin/Mkdir. Similarly, not all systems support 'whence' to locate executables in your search path. On such systems you may want to use the script whence from the HIRLAM export file (make it executable, and store it in $HL_UTIL/bin). If you do so, set WHENCE to $HL_UTIL/bin/whence. The HIRLAM script whence is slow but it is not called often so it should not impact performance substantially. 4.2 For (mini-)SMS (Introduced in version 5.1.2) SMSMETER should identify smsmeter in the search path. If you use mini-SMS, you will have obtained a script smsmeter; put it e.g. in HL_SCR and set SMSMETER to $HL_SCR/smsmeter. If you are using full SMS, chances are that the smsmeter that goes with it is in a directory that is searched before the HIRLAM ones (e.g. in /usr/local/bin). In that case, it suffices to set SMSMETER to smsmeter. 4.3 For the large permanent file system If LPFS is not unix-accessible, you may need a different utility to create directories there. This utility is used in the next section. 5. Preparative actions 5.1 Make directories (note: may be different on non-'unix' systems) Create the directories needed by your experiment. As of version 5.1.2, this has been moved to the jobs running HIRLAM. Because of the re-invocation of Env_system with a temporarily redefined value for SMSNODE, from version 5.1.2 HL_DATA may not be a valid directory name on the current host (remeber, it is a directory on SMSNODE). That is why creation of this directory is then inhibit by a test on the local variable bvn. 6. Operating system dependants For portability, several variables are used by HIRLAM to select for different operating systems. 6.1 To run scripts NAMLIS_E is the string to end namelists. On most Fortran-77 systems this is the string '&end', but Fortran-90 compilers may require '/'. 6.2 System libraries On some systems, performance is increased by using BLAS or EISPACK routines. To invoke those, you will have to set cpp define options during compilation, and make the libraries known to the loader by defining them in the variable SCILIB (space separated). At ECMWF, ECLIB and EMOSLIB are loaded but I doubt they are required at all. 6.3 HDF HIRLAM is in the process of converting to HDF as data base format. HIRLAM 4.6.3 and before do not use HDF, so for those versions there is no need to install or use HDF. The first step to move to HDF is the climate file generation process. If you can skip this step because you have the climate files for your area already from an earlier HIRLAM run, then again there is no need to install ot use HDF. If HDF is required, specify the location of the HDF libraries in environment veriable HDFLIBDIR and that of the HDF include files in HDFINCDIR; specify the additional loader flags in HDFLIBS; export those variables. 6.4 MPI Starting from version 4.4.0, HIRLAM has preparations to run on massively parallel machines. In principle, several parallellisation models have been implemented, but only MPI and SGI/Cray SH_MEM (on T3E) have been tried so far. If you want to use MPI, you should make the additional command line arguments for the loader known to the system through the environment variable MPILIB, and the location of the MPI include files through MPIINC. Note that MPILIB may also be used to specify additional loader flags. To use MPI, obviously, MPI must be available on your system. If it is not, you should look for either a vendor-specific implementation (probably most efficient) or a more general one, e.g. LAM (http://www.mpi.nd.edu/lam). Some implementations of MPI may require additional environment variables, or extensions to the search PATH. As an example the requirements for LAM-MPI have been included. For other implementations probably you would also have to modify Makefile and/or Makefile_x (see section 8.1 below). If you would want to play with other parallellisation models (e.g. PVM), please do so, good luck! On shared-memory or single-processor machines, or on SGI using SH_MEM, nothing has to be done for MPI. Here, parallellisation is organised through compiler and loader flags, in Makefile. 6.5 mini-SMS Mini-SMS is available automatically when the HIRLAM scripts have been installed. When it starts up, it tries to invoke a graphical interface, which requires the Perl-Tk toolkit. Ask your system manager to install this toolkit in the default Perl library path; but if (s)he cannot do this as quickly as you need it, you may want to install it in a private directory; and then you should set PERLTKDIR to that directory. 8. Derived environment variables Environment variables that can be derived from those above are set here. It is most undesirable that you change this section during installation. 8.1 Makefile and Makefile_x HIRLAM libraries and executables are constructed by Makefile and Makefile_x, resp. These are treated just as scripts, so the Makefile and Makefile_x that are found first in your path (see section 3 above) will be used. If your architecture is not coded for in either of these files, you should add the appropriate code here. This may be a lot of work, especially if you want to (and you should!) find the optimum compiler flags. This is typically a task for the local system manager, who should put these files somewhere in your path (as per section 3 above). # 9. Obsolescing Some older HIRLAM scripts still use obsolescing variables.