Commit 2050fd49 authored by Cristina Sanz Sanz's avatar Cristina Sanz Sanz
parents e079ba26 a33a2c6b
Pipeline #4591 passed with stages
in 1 minute and 58 seconds
......@@ -64,6 +64,7 @@ The following modules connected to the DL_MESO_DPD code have been produced so fa
./modules/DL_MESO_DPD_onGPU/fftw/readme
./modules/DL_MESO_DPD/check_dlmeso_dpd/readme
./modules/DL_MESO_DPD/tetra_dlmeso_dpd/readme
./modules/DL_MESO_DPD_onGPU/multi_gpu/readme
./modules/DL_MESO_DPD/sionlib_dlmeso_dpd/readme
ESPResSo++
......@@ -119,6 +120,9 @@ This modules are connected to the Adaptive Resolution Simulation implementation
./modules/GC-AdResS/AdResS_RDF/readme
./modules/GC-AdResS/Abrupt_Adress_forcecap/readme
./modules/GC-AdResS/AdResS_TF/readme
./modules/GC-AdResS/LocalThermostat_AdResS/readme
./modules/GC-AdResS/Analyse_Tools/readme
./modules/GC-AdResS/Analyse_VACF/readme
.. _ALL_background:
......
################################
Multi-GPU version of DL_MESO_DPD
################################
.. sidebar:: Software Technical Information
The information in this section describes the DL_MESO_DPD GPU versions as a whole.
Language
Fortran/CUDA-C (cuda toolkit 7.5)
Documentation Tool
ReST files
Application Documentation
See the `DL_MESO Manual <http://www.scd.stfc.ac.uk/SCD/resources/PDF/USRMAN.pdf>`_
Relevant Training Material
See `DL_MESO webpage <http://www.scd.stfc.ac.uk/SCD/support/40694.aspx>`_
Licence
BSD, v. 2.7 or later
.. contents:: :local:
Authors: Jony Castagna
This module implements the first version of the D\_MESO\_DPD code with multiple NVidia Graphical Processing Units (GPUs). More details about it can be found in the following sections.
Purpose of Module
_________________
.. Give a brief overview of why the module is/was being created.
In this module the main framework of a multi-GPU version of the DL\_MESO\_DPD code has been developed. The exchange of data between GPUs overlaps with the computation of the forces
for the internal cells of each partition (a domain decomposition approach based on the MPI parallel version of DL\_MESO\_DPD has been followed).
The current implementation is a proof of concept only and relies on slow transfers of data from the GPU to the host and vice-versa. Faster implementations will be explored in future modules.
In particular, the transfer of data occurs in 3 steps: x-y planes first, x-z planes with halo data (i.e. the values which will fill the ghost cells) from
the previous swap and finally the y-z planes with all halos. This avoid the problems of the corner cells, which usually requires a separate communication
reducing the number of send/receive calls from 14 to 6.The multi-GPU version has been currently tested with 8 GPUs and successfully reproduce the same results as a
single GPU within machine accuracy resolution.
Future plans include benchmarking of the code with different data transfer implementations other than the current (trivial) GPU-host-GPU transfer mechanism.
These are: of Peer To Peer communication within a node, CUDA-aware MPI, and CUDA-aware MPI with Direct Remote Memory Access (DRMA).
.. references would be nice here...
Background Information
______________________
This module is part of the DL\_MESO\_DPD code. Full support and documentation is available at:
* https://www.scd.stfc.ac.uk/Pages/DL_MESO.aspx
* https://www.scd.stfc.ac.uk/Pages/USRMAN.pdf
To download the DL\_MESO\_DPD code you need to register at https://gitlab.stfc.ac.uk. Please contact Dr. Micheal Seaton at Daresbury Laboratory (STFC) for further details.
Testing
_______
The DL\_MESO code is developed using git version control. Currently the GPU version is under a branch named ``add_gpu_version``. After downloading the code, checkout the GPU branch and look into the ``DPD/gpu_version`` folder, i.e:
.. code-block:: bash
git clone https://gitlab.stfc.ac.uk/dl_meso.git
cd dl_meso
git checkout gpu_version
cd ./DPD/gpu_version
make all
To compile and run the code you need to have installed the CUDA-toolkit (>=8.0) and have a CUDA enabled GPU device (see http://docs.nvidia.com/cuda/#axzz4ZPtFifjw). For the MPI library the OpenMPI 3.1.0 has been used.
The current version has been tested ONLY for the ``Mixture_Large`` test case available in the ``DEMO/DPD`` folder. To run the case, compile the code using the ``make all`` command from the ``bin`` directory, copy the ``FIELD`` and ``CONTROL`` files in this directory and run ``./dpd_gpu.exe``.
Attention: the ``HISTORY`` file produced is currently NOT compatible with the serial version, because this is written in the C binary data format (Fortran files are organised in records,
while C are not. See https://scipy.github.io/old-wiki/pages/Cookbook/FortranIO.html).
However, you can compare the ``OUTPUT`` and the ``export`` files to verify your results. For more details see the ``README.rst`` file in the ``gpu_version`` folder.
Performance
___________
A test case a two phase mixture separation with 1.8 billion particles has been used and run for 100 time steps without IO operations.A weak scaling efficiency (:math:`\eta`) plot up to 512 GPUs (1.2 billion particles) is presented below. This plot is obtained by taking the ratio between the wall time for the GPU count and a reference walltime of two GPUs (the singleGPU version uses a non-scalable, faster, alternative implementation which would skew the results). As can be seen, the result (:math:`\eta*GPUs`) oscillates near perfect scalability.
.. image:: ./DL_MESO_GPU_WeakScaling.png
:width: 90 %
:align: center
Strong scaling results are obtained using 1.8 billion particles for 256 to 2048 GPUs. Results show very good scaling, with efficiency always above 89% for 2048 GPUs (note that 2048 P100 GPUs on PizDaint is equivalent to almost 10 Petaflops of raw double precision compute performance).
.. image:: ./DL_MESO_GPU_StrongScaling.png
:width: 90 %
:align: center
Examples
________
See the ``Mixture_Large`` case in the DL\_MESO manual.
Source Code
___________
.. link the source code
This module has been merged into DL\_MESO code. It is composed of the
following commits (you need to be registered as collaborator):
* https://gitlab.stfc.ac.uk/dl_meso/dl_meso/commit/7f3e7abe7bb1c8010dd6a5baa0de4907ffe2f003
.. IF YOUR MODULE IS A SEPARATE REPOSITORY
.. The source code for this module can be found in: URL.
.. CLOSING MATERIAL -------------------------------------------------------
.. Here are the URL references used
.. _nose: http://nose.readthedocs.io/en/latest/
......@@ -289,7 +289,6 @@ To apply the patch:
.. Remember to change the reference "patch" for something unique in your patch file subpage or you will have
cross-referencing problems
In this module we also include a test scenario for GROMACS version 5.1.5 with a possible CG potential and all necessary input files. To run it simply run *gmx grompp -f grompp.mdp -c conf.gro -p topol.top -n index.ndx -maxwarn 5; gmx mdrun* using the patched version of GROMACS version 5.1.5 (see above).
When *gmx mdrun* finished normally (with the above mentioned setup), we have several mandatory checks to see if the simulation was successful or not.
......@@ -314,5 +313,3 @@ When *gmx mdrun* finished normally (with the above mentioned setup), we have sev
The files for the water example can be found here:
:download:`spc-example.tar.gz <spc-example.tar.gz>`
.. In ReStructured Text (ReST) indentation and spacing are very important (it is how ReST knows what to do with your
document). For ReST to understand what you intend and to render it correctly please to keep the structure of this
template. Make sure that any time you use ReST syntax (such as for ".. sidebar::" below), it needs to be preceded
and followed by white space (if you see warnings when this file is built they this is a common origin for problems).
.. Firstly, let's add technical info as a sidebar and allow text below to wrap around it. This list is a work in
progress, please help us improve it. We use *definition lists* of ReST_ to make this readable.
.. sidebar:: Software Technical Information
Name
Tools for AdResS.
Language
C/C++, Python, Fortran, BASH, AWK
Licence
Opensource
Application Documentation
See GROMACS web page: `<http://www.gromacs.org>`_. For analysis tools and thermodynamic force calculation see VOTCA web page: `<http://www.votca.org/home>`_. Visualize molecular dynamics with `VMD <http://www.ks.uiuc.edu/Research/vmd/>`_.
Documentation Tool
none
Relevant Training Material
none
.. In the next line you have the name of how this module will be referenced in the main documentation (which you can
reference, in this case, as ":ref:`example`"). You *MUST* change the reference below from "example" to something
unique otherwise you will cause cross-referencing errors. The reference must come right before the heading for the
reference to work (so don't insert a comment between).
################
Tools for AdResS
################
.. Let's add a local table of contents to help people navigate the page
.. contents:: :local:
.. Add an abstract for a *general* audience here. Write a few lines that explains the "helicopter view" of why you are
creating this module. For example, you might say that "This module is a stepping stone to incorporating XXXX effects
into YYYY process, which in turn should allow ZZZZ to be simulated. If successful, this could make it possible to
produce compound AAAA while avoiding expensive process BBBB and CCCC."
Purpose of Module
_________________
One purpose of our project is to promote GC-AdResS as a method. It is an advanced method, for people with experience, and once the simulation is done there are several properties and checks to consider to make sure that the simulation was successful.
This module provides little tools to make working with AdResS easier.
Content:
1) how to mask the configuration (output from a full atomistic simulation run) to generate the double resolution configuration.
2) Quick and dirty: get the reference coordinate from the GROMACS input file
3) Checks for the density (for both geometries currently implemented in GROMACS version 5.1.5)
4) Check the temperature on the fly
5) A short fortran code to calculate the distribution of the angles in slab-like AdResS simulation.
.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
Background Information
______________________
.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
Source Code
___________
.. Notice the syntax of a URL reference below `Text <URL>`_
Quick and fast data grab from the configuration file:
::
tail conf.gro | awk '(NF==3){print $1/2.0,$2/2.0,$3/2.0}'
How to mask the configuration for setting it up for the AdResS simulation. A straigh forward way is using `VOTCA <http://www.votca.org/tutorials>`_ :
::
csg_map --cg mapping_scheme.xml --hybrid --trj input_file.gro --out output_file.gro --top atomistic_run/topol.tpr
Check temperature on the fly from the output md.log:
::
#!/bin/bash
grep -A 1 --no-group-separator Lambda md.log | grep -v Step | awk '{print $1}' > mdlogging1
grep -A 1 --no-group-separator Temp md.log | grep -v Temp | awk '{print $2}' > mdlogging2
paste mdlogging1 mdlogging2
paste mdlogging1 mdlogging2 >temperature
rm mdlogging1 mdlogging2
Quick grab of the density in the xsplit (slab like) configuration. One way is using the tool from `GROMACS <http://www.gromacs.org>`_:
::
gmx density -d X -f trajectory_file.xtc -sl 50
Quick grab of the density in the sphere configuration. We use `VOTCA <http://www.votca.org/tutorials>`_ for it:
::
csg_density -- axis r -- rmax 10. --ref [x_ref,y_ref,z_ref] --trj trajectory_input.xtc --top topol.tpr --out SOL.dens.out
Collect the p(N) data and combine them in one file. For that we use `VMD <http://www.ks.uiuc.edu/Research/vmd/>`_, it includes a script called topotools, which is used to handle the trajectory. This script can be run from the command line directly:
::
vmd -dispdev text -e extract_coord.tcl
grep -B1 "Frame" WCG.xyz > a
sed '/Frame/ {$!N;d;}' a > column2
grep -B0 "Frame" WCG.xyz > a
sed -i s/Frame// a
sed -i s/--// a
sed -i s/:// a
sed '/^$/d' a > column1
paste column1 column2|awk '{print $1, $2}' > dat.3nm.pn.WCG.dat
And the corresponding extract_coord.tcl:
::
package require topotools 1.2
mol new conf.gro
mol addfile traj_comp.xtc type xtc waitfor all first 0 last -1 step 1
topo writevarxyz WCG.xyz selmod "name WCG and (x>285 and x<315)"
exit
All the small scripts are available as files:
:download:`analysis tools source code <./analysis.tools.tar.gz>`
.. In ReStructured Text (ReST) indentation and spacing are very important (it is how ReST knows what to do with your
document). For ReST to understand what you intend and to render it correctly please to keep the structure of this
template. Make sure that any time you use ReST syntax (such as for ".. sidebar::" below), it needs to be preceded
and followed by white space (if you see warnings when this file is built they this is a common origin for problems).
.. Firstly, let's add technical info as a sidebar and allow text below to wrap around it. This list is a work in
progress, please help us improve it. We use *definition lists* of ReST_ to make this readable.
.. sidebar:: Software Technical Information
Name
Velocity-Velocity autocorrelation function for AdResS.
Language
C/C++
Licence
Opensource
Documentation Tool
none
Application Documentation
http://www.ks.uiuc.edu/Research/vmd/current/docs.html
Relevant Training Material
http://www.ks.uiuc.edu/Research/vmd/current/docs.html
.. In the next line you have the name of how this module will be referenced in the main documentation (which you can
reference, in this case, as ":ref:`example`"). You *MUST* change the reference below from "example" to something
unique otherwise you will cause cross-referencing errors. The reference must come right before the heading for the
reference to work (so don't insert a comment between).
#####################################################
Velocity-Velocity autocorrelation function for AdResS
#####################################################
.. Let's add a local table of contents to help people navigate the page
.. contents:: :local:
.. Add an abstract for a *general* audience here. Write a few lines that explains the "helicopter view" of why you are
creating this module. For example, you might say that "This module is a stepping stone to incorporating XXXX effects
into YYYY process, which in turn should allow ZZZZ to be simulated. If successful, this could make it possible to
produce compound AAAA while avoiding expensive process BBBB and CCCC."
Purpose of Module
_________________
One purpose of our project is to promote GC-AdResS as a method. It is an advanced method, for people with experience, and once the simulation is done there are several properties and checks to consider to make sure that the simulation was successful.
This module provides the code to run a velocity velocity autocorrelation function on the current geometries available in the Abrupt AdResS implementation. The paper
`Ref. <http://iopscience.iop.org/article/10.1088/1367-2630/17/8/083042>`_ describes the correlation functions and why they can be used in AdResS. This code is based on that theory and has been developed to check the dynamics of the local thermostat GC-AdResS simulations presented in the paper cited above.
Source Code
___________
.. Notice the syntax of a URL reference below `Text <URL>`_
Files are stored here: `<https://gitlab.e-cam2020.eu/krekeler/analyze.energy>`_. The source code for the velocity autocorrelation function can be found here: `<https://gitlab.e-cam2020.eu/krekeler/analyze.energy/tree/master/app/cal_vel_acc_adr.cpp>`_
The installation instruction can be found `<https://gitlab.e-cam2020.eu:10443/krekeler/analyze.energy#installation-instructions>`_.
Usage:
::
cal_vel_acc_adr:
options:
-h , "print this message")
-b start time
-e end time (=number of MD steps)
--x0 lower bound of the interval
--x1 upper bound of the interval (--x1 0, use the whole box = atomistic)
--frame length of correlation
--acc breaks
--total number of frames
--tf Output Frequency (=Delta_t)
-m type of simulation to analyze (adress or atom)
-f input .xtc file
-o output file
It is important to have the XDR files and setup in the same directory as they have to be specified in the Makefile. The XDR files can be found via the GROMACS web page, see `<http://www.gromacs.org/Developer_Zone/Programming_Guide/XTC_Library>`_ or `<ftp://ftp.gromacs.org/pub/contrib/xdrfile-1.1.4.tar.gz>`_.
.. _localT_abrupt_adress:
####################################
Patch file for module: Abrupt AdResS
####################################
The patch for the abrupt AdResS code is:
.. literalinclude:: ./localT_abrupt_adress.patch
:linenos:
:download:`Downloadable version of patch file <localT_abrupt_adress.patch>`
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment