\documentclass[a4paper]{article}

                                % Packages

\usepackage[utf8]{inputenc}
\usepackage{ifpdf}

\ifpdf
\else
\usepackage[active]{srcltx}
\fi

\usepackage{amsmath}

\usepackage[T1]{fontenc}
\usepackage{lmodern}

\usepackage{geometry}

\usepackage[np]{numprint}

\usepackage{hyperref}
\ifpdf
\hypersetup{pdftitle={Tutorial XIOS}, pdfauthor={LMDZ team},
  hypertexnames=false, pdfstartview=FitBH}
\fi

\author{LMDZ team}
\title{LMDZ tutorial: XIOS}
\date{November 22nd, 2021}


\geometry{top=2cm,bottom=2cm,left=2.5cm,right=2.5cm}

\begin{document}

\maketitle

This tutorial focuses on setting up, compiling 
and running LMDZ with XIOS.\\

This document can be downloaded as a pdf file:
\begin{verbatim}
wget http://lmdz.lmd.jussieu.fr/pub/Training/Tutorials/Tutorial_XIOS.pdf
\end{verbatim}
which should ease any copy/paste of command lines to issue.\\

%\def\infol#1{#1}
%\def\infol#1{\verb+ #1 +}
\def\infol#1{{\bf #1}}
%\newcommand{\infol}[1]{\verb\+ #1 \+}

\def\resol#1#2#3{#1$\times$#2-L#3} %


This tutorial is for users who want to learn the basic
steps needed to be able to run LMDZ with the XIOS input-output library
on their computer.
Note that this implies the prerequisite that you can run in parallel
on your machine (i.e. that you have already completed the tutorial
on running LMDZ in parallel).\\

\section{Running the install\_lmdz.sh script}
The \infol{install\_lmdz.sh} script can download and compile the necessary libraries (NetCDF, IOIPSL, XIOS) and programs (ORCHIDEE and LMDZ), and runs a test simulation. All that is required is to specify the \verb+-xios+ and \verb+-parallel mpi_omp+ (XIOS is designed to be used in parallel) options:
\begin{verbatim}
wget http://www.lmd.jussieu.fr/~lmdz/pub/install_lmdz.sh
chmod +x install_lmdz.sh
./install_lmdz.sh -parallel mpi_omp -cosp none -xios -d 32x32x39 -v 20211102.trunk
\end{verbatim}

As with previous automated installations, you are encouraged to browse through
the contents of subdirectory \infol{LMDZ} (e.g. the \infol{compile.sh} script) and
\infol{BENCH32x32x39}.\newline

Note that to compile LMDZ with XIOS, one must use the \verb+-io xios+ flag, e.g.:
\begin{verbatim}
./makelmdz_fcm -arch local -parallel mpi -io xios -d 32x32x39 -rad rrtm -cosp true -j 8 gcm
\end{verbatim}
As can be seen in the \infol{compile.sh} script in the \infol{LMDZ} directory.\newline

Take the time to browse through the \infol{xml} and \infol{histday.nc} 
files in \infol{BENCH32x32x39} to identify definitions and settings that were
used. Note that the automated processing in the
install script has renamed the executable \infol{gcm.e}, but you may still find it in 
\infol{LMDZ/bin}.

%{\bf Important}: on the laptops provided for the training, the compilation
%may fail due to a wrong ordering of \infol{xios} and \infol{libc++} in the
%\infol{makelmdz\_fcm} script. You should modify line 393 from:
%\begin{verbatim}
%-l${LIBPREFIX}stdc++ -l${LIBPREFIX}xios"
%\end{verbatim}
%to
%\begin{verbatim}
%-l${LIBPREFIX}xios -l${LIBPREFIX}stdc++"
%\end{verbatim}
%Moreover, because the \infol{makelmdz\_fcm} script failed, you will need to
%recompile (use the \infol{compile.sh} script in \infol{LMDZ} directory
%and then move the executable to the \infol{BENCH32x32x39} directory
%\begin{verbatim}
%mv bin/gcm_32x32x39_phylmd_para_mem_orch.e BENCH32x32x39/gcm.e
%\end{verbatim}
%and then go to the \infol{BENCH32x32x39} directory and run the
%\infol{bench.sh} script to run the model.

\section{Running a first simulation with XIOS}
Make a new simulation directory (e.g. by copying over all input \infol{.def} %\infol{.txt}\footnote{These are required for COSP.}
and \infol{.nc} files from the \infol{BENCH32x32x39}
directory\footnote{As in this example Orchidee is not used, the flag
\infol{VEGET} in \infol{config.def} should be set to \infol{n}}), along with the executable \verb+gcm_32x32x39_phylmd_para_mem.e+ from \infol{LMDZ/bin}.
In addition you will need the input \infol{.xml} files to manage XIOS outputs, which you should also copy over from \infol{BENCH32x32x39}.\\

\noindent
For this first simulation, we will use XIOS in attached mode (i.e.
embedded in LMDZ), so the \infol{using\_server} variable in 
\infol{iodef.xml} must be set to \infol{false} (which should already be the case
in the current setup).\\
Moreover, to enable outputs via XIOS in LMDZ, note that the following flag:
\begin{verbatim}
ok_all_xml = .true.
\end{verbatim}
(or equivalently \verb+ok_all_xml = y+)
must be set in the \infol{config.def} file (which should also already be the case
in the current setup).\\

\noindent
As can be seen in the \infol{context\_lmdz.xml} file, many
predefined output files (mimicking what is done via the output.def file when
using IOIPSL) are defined.
By default only \infol{histday.nc} is enabled.
%By default \infol{histday.nc} and \infol{histdayCOSP.nc} are enabled. Start by modifying file \infol{file\_def\_histdayCOSP\_lmdz.xml} to not output file \infol{histdayCOSP.nc}, simply by changing its \infol{enabled} field to \infol{.false.}:
%\begin{verbatim}
%<file id="histdayCOSP" name="histdayCOSP" output_freq="1d" output_level="5" 
%type="one_file" enabled=".false." compression_level="0" sync_freq="1d">
%\end{verbatim}
Let's imagine that instead we want output file \infol{histhf.nc} to be outputed.
Modify file \infol{file\_def\_histhf\_lmdz.xml} by setting
\begin{verbatim}
<file id="histhf" name="histhf" output_freq="3h" output_level="5"
type="one_file" enabled=".true." compression_level="0" sync_freq="3h" >
\end{verbatim}
so that the \infol{histhf.nc} file will also be generated (as a single file
over the entire domain) when the model is run.\\

\noindent
%The model can now be run.
%Note that because the NetCDF libraries are
%dynamically loaded at run time, the path to these must be included in
%the \infol{LD\_LIBRARY\_PATH} environment variable\footnote{Rather than having
%to have to add the path to \infol{LD\_LIBRARY\_PATH} in each new shell before
%running the model, it makes sense to add that command in your
%\infol{\$HOME/.bashrc} file so that it is always automatically done.}:
%\begin{verbatim}
%export LD_LIBRARY_PATH=\home\util1\local\lib:$LD_LIBRARY_PATH
%\end{verbatim}
Then run the model "as usual", e.g. in MPI mode using 4 processes\footnote{Or using a script
such as \infol{bench.sh} or \infol{bench\_parallel.sh}, which you might need to adapt to your settings}:
\begin{verbatim}
mpirun -np 4 gcm_32x32x39_phylmd_para_mem.e > listing 2>&1
\end{verbatim}
And check the contents of the generated \infol{histday.nc} and \infol{histhf.nc} files. Note the use of \verb#operation="average"#, \verb#operation="instant"#, \verb#operation="once"# , etc. attributes in the xml files to trigger designated operations on the outputted fields.

\section{Exercise: Defining an additional output domain and grid}
One can output only a selected subset of the global domain\footnote{Some XIOS terminology: a "domain" refers to a horizontal (2D) domain, an "axis" is 1D (e.g. altitude, or number of tracers) and a "grid" is a (0D, 1D, 2D or 3D) geometry along which outputs will be made.} by specifying
the appropriate \infol{domain} attributes in the
\infol{context\_lmdz.xml} file. For example to ontput a 2x3 subdomain starting
at (rectilinear) grid indexes $i=20$,$j=15$ (C convention: index begining at $0$):
\begin{verbatim}
<domain_definition>
    <domain id="dom_glo" data_dim="2" />
    <domain id="domain_zoom" domain_ref="dom_glo">
        <zoom_domain ni="2" ibegin="20" nj="3" jbegin="15" />
    </domain>
....
</domain_definition>
\end{verbatim}
And also, still in the \infol{context\_lmdz.xml} file, define the corresponding
grid in the \verb+<grid_definition>+ section:
\begin{verbatim}
<!-- Define Scalar grid for GHG, orbital parameters and solar constants -->
  <grid_definition>
    <grid id="grid_scalar">
    <scalar/>
    </grid>
    <grid id="grid_zoom" >
      <domain domain_ref="domain_zoom" />
    </grid>
  </grid_definition>

  <!-- Define groups of vertical axes -->
  <axis_definition>
....
  </axis_definition>

  <grid_definition>
....
     <grid id="grid_glo">
        <domain domain_ref="dom_glo" />
     </grid>
     <grid id="grid_out">
        <domain domain_ref="dom_out" />
     </grid>

     <grid id="grid_zoom_presnivs">
         <domain id="domain_zoom" />
         <axis id="presnivs" />
     </grid>
....
\end{verbatim}

\noindent
To test implementing this setup, let's assume you want to output at only
one grid point, corresponding to Paris %(longitude 49N ,latitude 2E)
to compare model ouput to station records.\\

\noindent
The first thing to do is to identify the grid coordinates that will
have to be specified in the \infol{domain} attributes. This can be done
by inspecting the \infol{lat} and \infol{lon} values in the
\infol{histday.nc} file from the previous run, either via your favorite
visualization software, or simply using the \infol{ncdump} utility:
\begin{verbatim}
ncdump -fc -v lon histday.nc
ncdump -fc -v lat histday.nc
\end{verbatim}
And adapt the \infol{context\_lmdz.xml} file accordingly.

\noindent
Since we are interested in instantaneous values of for instance
the \infol{t2m} (temperature at 2m), \infol{precip} (precipitation
rates), \infol{psol} (surface pressure) and \infol{temp} (temperature profile)
in the zoomed grid, it makes sense to define a new output file.
One could either adapt the current \infol{histins} file,
or define a new one e.g. a \infol{file\_def\_histinsParis\_lmdz.xml} file:
\begin{verbatim}
<file_definition>
    <file_group id="defile">
        <file id="histinsParis" name="histinsParis"
               output_freq="1ts" output_level="5"
               type="one_file" enabled=".TRUE.">
            
            <!-- VARS 2D -->
            <field_group operation="instant" grid_ref="grid_zoom">
                <field field_ref="t2m" level="5" />
                <field field_ref="precip" level="5" />
                <field field_ref="psol" level="5" />
            </field_group>
            
            <!-- VARS 3D -->
            <field_group operation="instant"
                         grid_ref="grid_zoom_presnivs">
                <field field_ref="temp" level="5" />
            </field_group>
        </file>
    </file_group>
</file_definition>
\end{verbatim}
And add this new definition file to the others specified in \infol{context\_lmdz.xml}:
\begin{verbatim}
    <file_definition src="./file_def_histinsParis_lmdz.xml"/>
\end{verbatim}

\noindent
Run the model and check the produced \infol{histinsParis.nc} file.

\section{Exercise: Changing XIOS output files and verbosity level}
Parameters specific to dictate XIOS behavior are defined in the \infol{iodef.xml} file within the XIOS context:
\begin{verbatim}
	<context  id="xios">
....
	</context>
\end{verbatim}
The \infol{info\_level} variable sets the degree of verbosity of XIOS (0: low, 100: high). There are other additionnal variables than the one present in the current \infol{iodef.xml}, for instance \infol{print\_file}, which can be set to true to have XIOS issue its output and error messages to dedicated text files (one of each for each MPI process), or false (default behavious) to send these messages to the same standard output and error streams as LMDZ. When debugging, another useful variable is \infol{xios\_stack} (true/false; default value is false) to have a more complete traceback or encountered errors.\\

\noindent
Assume you want to rerun the previous experiment with higher XIOS verbosity and dedicated output files; adapt the \infol{iodef.xml} and rerun to check the generated \infol{xios\_client\_*.err} and \infol{xios\_client\_*.out} files.

\section{Exercise: Running in client-server mode}
When running on a small number of cores, it is advised to use XIOS in "attached" mode, as done so far.
In multicore environments (i.e. more than 32), when outputing a lot of data, it can be more efficient to run in client-server mode and dedicate some cores to the XIOS server.\\

\noindent
To test this setup, make a new directory where  to
run and copy over input files from previous simulation.
Start by copying over the XIOS server
\infol{xios\_server.exe} from \infol{XIOS/bin}. Then adapt the
\infol{iodef.xml} to switch to client-server mode
by setting the \infol{using\_server} variable to \infol{true}.
The executables may now be run, where the number of processes
allocated to each is set via the \infol{mpirun} command,
for instance to run LMDZ on 3 processes and XIOS on 1:
\begin{verbatim}
mpirun -np 3 gcm_32x32x39_phylmd_para_mem.e > listing 2>&1 : -np 1 xios_server.exe
\end{verbatim}
And check that you get the same output files as before.

\end{document}
