Commit a0633388 by Alan O'Cais

### Merge branch 'master' into Analyse_Energy

parents 23d8f3ba 01ab8150
 ######## FFTXlib ######## .. sidebar:: Software Technical Information Language Fortran 1995 Documentation Tool Sphinx, ReStructuredText Application Documentation Doc mirror _ Relevant Training Material See usage examples in the examples directory of the source code. Licence GNU Lesser General Public License v3.0 Author of Module Emine Kucukbenli .. contents:: :local: Purpose of Module _________________ FFTXlib module is a collection of driver routines for complex 3D fast Fourier transform (FFT) libraries to be used within planewave-based electronic structure calculation software. Generally speaking, FFT algorithm requires a data array to act on, a clear description of the input-output sequence and transform domains. In the context of planewave based electronic structure calculations, the data array may hold elements such as electronic wavefunction :math:\psi or charge density :math:\rho or their functions. The transform domains are direct (real) and reciprocal space, the discretization in real space is represented as a uniform grid of the unit cell and the discretization of the reciprocal space is in the basis of planewaves whose wavevectors are multiples of reciprocal space vectors :math:(\mathbf G) . To understand the main motivation behind FFTXlib routines we need to clarify the differences between the representation of wavefunction and charge density in planewave based codes: In these codes, the expansion of wavefunction in planewave basis is truncated at a cut-off wave-vector :math:\mathbf G_{max}. Since density is the norm-square of the wavefunction, the expansion that is consistent with the one of wavefunctions requires a cut-off wavevector twice that of wavefunctions: :math:2 \mathbf G_{max}. Meanwhile, the real space FFT domain is often only defined by one uniform grid of the unit cell, so the array sizes of both :math:\rho and :math:\psi in their real space representation are the same. Therefore, to boost optimization and to reduce numerical noise, the library implements two possible options while performing FFT: in one ( 'Wave') the wavevectors beyond :math:\mathbf G_{max} are ignored, in the other ( 'Rho' ) no such assumption is made. Another crucial feature of FFTXlib is that some approximations in the electronic structure calculations (such as usage of non-normconserving pseudopotentials) require that density is not just norm-square of wavefunctions, but has spatially localized extra components. In that case, these localized contributions may require higher G-vector components than the ones needed for density (:math:> 2 \mathbf G_{max}). Hence, in such systems, the density array in reciprocal space has more elements than the norm-conserving case (in other words a finer resolution or a denser grid is needed in real space) while the resolution needed to represent wavefunctions are left unchanged. To accommodate for these different requirements of grid size, and to be able to make Fourier transforms back and forth between them, the FFTXlib routines explicitly require descriptor arguments which define the grids to be used. For example, if potential is obtained from density, the FFT operations on it should use the denser grid; while FFT on wavefunctions should use the smoother grid (corresponding to :math:2\mathbf G_{max} as defined before). When the Hamiltonian's action on wavefunctions are being calculated, the potential should be brought from dense to smooth grid. But when the density is being calculated, wavefunction normsquare should be carried from smooth to dense grid. A final important feature of FFTXlib is the index mapping. In the simple case of no parallelization, as a choice, the reciprocal space arrays are ordered in increasing order of :math:|G|^2 while the real space arrays are sorted in column major order. Therefore for FFT to be performed, a map between these two orders must be known. This index map is created and preserved by the FFTXlib. In summary, FFTXlib allows the user to perform complex 3D fast Fourier transform (FFT) in the context of plane wave based electronic structure software. It contains routines to initialize the array structures, to calculate the desired grid shapes. It imposes underlying size assumptions and provides correspondence maps for indices between the two transform domains. Once this data structure is constructed, forward or inverse in-place FFT can be performed. For this purpose FFTXlib can either use a local copy of an earlier version of FFTW (a commonly used open source FFT library), or it can also serve as a wrapper to external FFT libraries via conditional compilition using pre-processor directives. It supports both MPI and OpenMP parallelization technologies. FFTXlib is currently employed within Quantum Espresso package, a widely used suite of codes for electronic structure calculations and materials modeling in the nanoscale, based on planewave and pseudopotentials. FFTXlib is also interfaced with "miniPWPP" module that solves the Kohn Sham equations in the basis of planewaves and soon to be released as a part of E-CAM Electronic Structure Library. Background Information ______________________ FFTXlib is mainly a rewrite and optimization of earlier versions of FFT related routines inside Quantum ESPRESSO pre-v6; and finally their replacement. This may shed light on some of the variable name choices, as well as the default of :math:2\mathbf G_{max} cut-off for the expansion of the smooth part of the charge density, and the required format for lattice parameters in order to build the FFT domain descriptor. Despite many similarities, current version of FFTXlib dramatically changes the FFT strategy in the parallel execution, from 1D+2D FFT performed in QE pre v6 to a 1D+1D+1D one; to allow for greater flexibility in parallelization. Building and Testing ______________________________ A stable version of the module can be downloaded using this link _ .. when fftxlib has its own repo, this link can be moved there. Current installation and testing are done with gfortran compiler, version 4.4.7. The configuration uses GNU Autoconf 2.69. The commands for installation are:: $./configure$ make libfftx As a result, the library archive "libfftx.a" is produced in src directory, and symbolicly linked to a "lib" directory. .. To test whether the library is working as expected, run:: .. $make FFTXtest .. Besides the PASS/FAIL status of the test, by changing the bash script in the tests directory, you can perform your custom tests. Read the README.test documentation in the tests subdirectory for further details about the tests. To see how the library works in a realistic case scenario of an electronic structure calculation, run::$make FFTXexamples .. Besides the PASS/FAIL status of the example, by changing the bash script in the examples directory, you can create your custom examples. A mini-app will be compiled in src directory and will be symbolicly copied into bin directory. The mini-app simulates an FFT scenario with a test unit cell, and plane wave expansion cutoff. It creates the FFT structures and tests forward and backward transform on sample array and reports timings. Read the README.examples documentation in the examples subdirectory for further details. Source Code ____________ The FFTXlib bundle corresponding to the stable release can be downloaded from this link _ The source code itself can be found under the subdirectory src. The development is ongoing. The version that corresponds to the one of examples and tests can be obtained with SHA 31a6f4ecbb7ce474b0c87702c716713758f99a0a. This will soon be replaced with a version tag. Further Information ____________________ This documentation can be found inside the docs subdirectory. The FFTXlib is developed with the contributions of C. Cavazzoni, S. de Gironcoli, P. Giannozzi, F. Affinito, P. Bonfa', Martin Hilgemans, Guido Roma, Pascal Thibaudeau, Stephane Lefranc, Nicolas Lacorne, Filippo Spiga, Nicola Varini, Jason Wood, Emine Kucukbenli.
 .. In ReStructured Text (ReST) indentation and spacing are very important (it is how ReST knows what to do with your document). For ReST to understand what you intend and to render it correctly please to keep the structure of this template. Make sure that any time you use ReST syntax (such as for ".. sidebar::" below), it needs to be preceded and followed by white space (if you see warnings when this file is built they this is a common origin for problems). .. Firstly, let's add technical info as a sidebar and allow text below to wrap around it. This list is a work in progress, please help us improve it. We use *definition lists* of ReST_ to make this readable. .. sidebar:: Software Technical Information Name DL_MESO (DPD). Language Fortran, CUDA-C. Licence BSD _, v. 2.7 or later Documentation Tool ReST files Application Documentation See the DL_MESO Manual _ Relevant Training Material See DL_MESO webpage _ Software Module Developed by Jony Castagna .. In the next line you have the name of how this module will be referenced in the main documentation (which you can reference, in this case, as ":ref:example"). You *MUST* change the reference below from "example" to something unique otherwise you will cause cross-referencing errors. The reference must come right before the heading for the reference to work (so don't insert a comment between). .. _dl_meso_dpd_gpu_fftw: ################################# SPME on DL_MESO_DPD (GPU version) ################################# .. Let's add a local table of contents to help people navigate the page .. contents:: :local: .. Add an abstract for a *general* audience here. Write a few lines that explains the "helicopter view" of why you are creating this module. For example, you might say that "This module is a stepping stone to incorporating XXXX effects into YYYY process, which in turn should allow ZZZZ to be simulated. If successful, this could make it possible to produce compound AAAA while avoiding expensive process BBBB and CCCC." The electrostatic force calculation usually represents the main computational costs in systems where even a small amount of charged particles are present (>1%). The Smooth Particle Mesh Ewald [SPME]_ splits the electrostatic forces in two parts: a short range, solved in the real space, and a long range, solved in the Fourier space. An error weight function combines the two contributions. For the long range force the electrical charges are spread on a virtual particle mesh using a B-spline interpolation function. Porting the full short and long range interactions to GPUs allowed us to achieve a speedup factor of 4x when compared to a traditional 12-core Intel CPU. One of the main applications which includes electrical charges are the simulations of plasma. Purpose of Module _________________ .. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment The Ewald summation method scales with :math:N^{1.5} at best, where N is the number of charged particles. The SPME method allows for improved scaling, :math:N*log(N), but requires a stencil domain decomposition (i.e. decomposing the domain along one direction only) to allow the FFTW library to scale with more than 1 core. If this is not used, as in the current master version of DL\_MESO\_DPD, FFTW rapidly becomes a bottleneck for scaling across several nodes. On the other hand, the porting to a single GPU does not need domain decomposition and the same speedup factor (4x compared to 12-core Intel) is maintained. Background Information ______________________ .. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment This module is part of the DL\_MESO\_DPD code. Full support and documentation is available at: * https://www.scd.stfc.ac.uk/Pages/DL_MESO.aspx * https://www.scd.stfc.ac.uk/Pages/USRMAN.pdf To download the DL\_MESO\_DPD code you need to register at https://gitlab.stfc.ac.uk/dl_meso/dl_meso. Please contact Dr. Micheal Seaton at Daresbury Laboratory (STFC) for further details. Building and Testing ____________________ .. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment The DL\_MESO code is developed using git version control. Currently the GPU version is under a branch named "add\_gpu\_version". After downloading the code, checkout the GPU branch and look into the "DPD/gpu\_version" folder, i.e: * git clone DL_MESO_repository_path * cd dl_meso * git checkout gpu_version * cd /DPD/gpu_version * make all To compile and run the code you need to have installed the CUDA-toolkit and have a CUDA enabled GPU device (see http://docs.nvidia.com/cuda/#axzz4ZPtFifjw). To run the case, compile the code using the "make all" command from the "bin" directory, copy the "FIELD" and "CONTROL" files in this directory and run "./dpd_gpu.exe". Source Code ___________ .. Notice the syntax of a URL reference below Text _ the backticks matter! This module has been merged into DL_MESO code. It is composed of the following commits (you need to be register as developer): * https://gitlab.stfc.ac.uk/dl_meso/dl_meso/commit/34a652fe62cadbac5e8a037b57ee9be64dcf4187 .. [SPME] J. Chem. Phys. 103, 8577 (1995) .. _ReST: http://www.sphinx-doc.org/en/stable/rest.html .. _Sphinx: http://www.sphinx-doc.org/en/stable/markup/index.html
 ################################ Multi-GPU version of DL_MESO_DPD ################################ .. sidebar:: Software Technical Information The information in this section describes the DL_MESO_DPD GPU versions as a whole. Language Fortran/CUDA-C (cuda toolkit 7.5) Documentation Tool ReST files Application Documentation See the DL_MESO Manual _ Relevant Training Material See DL_MESO webpage _ Licence BSD, v. 2.7 or later .. contents:: :local: Authors: Jony Castagna This module implements the first version of the D\_MESO\_DPD code with multiple NVidia Graphical Processing Units (GPUs). More details about it can be found in the following sections. Purpose of Module _________________ .. Give a brief overview of why the module is/was being created. In this module the main framework of a multi-GPU version of the DL\_MESO\_DPD code has been developed. The exchange of data between GPUs overlaps with the computation of the forces for the internal cells of each partition (a domain decomposition approach based on the MPI parallel version of DL\_MESO\_DPD has been followed). The current implementation is a proof of concept only and relies on slow transfers of data from the GPU to the host and vice-versa. Faster implementations will be explored in future modules. In particular, the transfer of data occurs in 3 steps: x-y planes first, x-z planes with halo data (i.e. the values which will fill the ghost cells) from the previous swap and finally the y-z planes with all halos. This avoid the problems of the corner cells, which usually requires a separate communication reducing the number of send/receive calls from 14 to 6.The multi-GPU version has been currently tested with 8 GPUs and successfully reproduce the same results as a single GPU within machine accuracy resolution. Future plans include benchmarking of the code with different data transfer implementations other than the current (trivial) GPU-host-GPU transfer mechanism. These are: of Peer To Peer communication within a node, CUDA-aware MPI, and CUDA-aware MPI with Direct Remote Memory Access (DRMA). .. references would be nice here... Background Information ______________________ This module is part of the DL\_MESO\_DPD code. Full support and documentation is available at: * https://www.scd.stfc.ac.uk/Pages/DL_MESO.aspx * https://www.scd.stfc.ac.uk/Pages/USRMAN.pdf To download the DL\_MESO\_DPD code you need to register at https://gitlab.stfc.ac.uk. Please contact Dr. Micheal Seaton at Daresbury Laboratory (STFC) for further details. Testing _______ The DL\_MESO code is developed using git version control. Currently the GPU version is under a branch named add_gpu_version. After downloading the code, checkout the GPU branch and look into the DPD/gpu_version folder, i.e: .. code-block:: bash git clone https://gitlab.stfc.ac.uk/dl_meso.git cd dl_meso git checkout gpu_version cd ./DPD/gpu_version make all To compile and run the code you need to have installed the CUDA-toolkit (>=8.0) and have a CUDA enabled GPU device (see http://docs.nvidia.com/cuda/#axzz4ZPtFifjw). For the MPI library the OpenMPI 3.1.0 has been used. The current version has been tested ONLY for the Mixture_Large test case available in the DEMO/DPD folder. To run the case, compile the code using the make all command from the bin directory, copy the FIELD and CONTROL files in this directory and run ./dpd_gpu.exe. Attention: the HISTORY file produced is currently NOT compatible with the serial version, because this is written in the C binary data format (Fortran files are organised in records, while C are not. See https://scipy.github.io/old-wiki/pages/Cookbook/FortranIO.html). However, you can compare the OUTPUT and the export files to verify your results. For more details see the README.rst file in the gpu_version folder. Performance ___________ A test case a two phase mixture separation with 1.8 billion particles has been used and run for 100 time steps without IO operations.A weak scaling efficiency (:math:\eta) plot up to 512 GPUs (1.2 billion particles) is presented below. This plot is obtained by taking the ratio between the wall time for the GPU count and a reference walltime of two GPUs (the singleGPU version uses a non-scalable, faster, alternative implementation which would skew the results). As can be seen, the result (:math:\eta*GPUs) oscillates near perfect scalability. .. image:: ./DL_MESO_GPU_WeakScaling.png :width: 90 % :align: center Strong scaling results are obtained using 1.8 billion particles for 256 to 2048 GPUs. Results show very good scaling, with efficiency always above 89% for 2048 GPUs (note that 2048 P100 GPUs on PizDaint is equivalent to almost 10 Petaflops of raw double precision compute performance). .. image:: ./DL_MESO_GPU_StrongScaling.png :width: 90 % :align: center Examples ________ See the Mixture_Large case in the DL\_MESO manual. Source Code ___________ .. link the source code This module has been merged into DL\_MESO code. It is composed of the following commits (you need to be registered as collaborator): * https://gitlab.stfc.ac.uk/dl_meso/dl_meso/commit/7f3e7abe7bb1c8010dd6a5baa0de4907ffe2f003 .. IF YOUR MODULE IS A SEPARATE REPOSITORY .. The source code for this module can be found in: URL. .. CLOSING MATERIAL ------------------------------------------------------- .. Here are the URL references used .. _nose: http://nose.readthedocs.io/en/latest/
 ... ... @@ -95,8 +95,7 @@ _________________ .. : .. [CIT2009] A citation (as often used in journals). The original idea of our proposal: to work on a general implementation of AdResS in class. MD packages. The current implementation of GC- AdResS in GROMACS has several performance problems. We know that the main performance loss of AdResS simulations in GROMACS is in the neighboring list search and the generic serial force kernel, linking the atomistic (AT) and coarse grained (CG) forces together via a smooth weighting function. Thus, to get rid of the bottleneck with respect to performance and a hindrance regarding the easy/general implementation into other codes and thus get rid of the not optimized force kernel used in GROMACS we had to change the neighborlist search. This lead to a considerable speed up of the code. Furthermore it decouples the method directly from the core of any MD code, which does not hinder the performance and makes the scheme hardware independent. For the theory, application and tests see _ or _. The main performance loss of AdResS simulations in GROMACS is in the neighboring list search and the generic serial force kernel, linking the atomistic (AT) and coarse grained (CG) forces together via a smooth weighting function. Thus, to get rid of the bottleneck with respect to performance and a hindrance regarding the easy/general implementation into other codes and thus get rid of the not optimized force kernel used in GROMACS we had to change the neighborlist search. This lead to a considerable speed up of the code. Furthermore it decouples the method directly from the core of any MD code, which does not hinder the performance and makes the scheme hardware independent. For the theory, application and tests see _ or _. .. The interface between the regions is more fluctuating and needs a more responsive thermodynamic force but it works reasonably well. ... ... @@ -277,7 +276,11 @@ ___________ .. Notice the syntax of a URL reference below Text _ To apply the patch: (:ref:abrupt_adress_patch) The patch file for Abrupt GC-Adress is: .. literalinclude:: ./abrupt_adress.patch To apply the patch: 1) copy into the main directory (gromacs/) ... ... @@ -286,7 +289,6 @@ To apply the patch: (:ref:abrupt_adress_patch) .. Remember to change the reference "patch" for something unique in your patch file subpage or you will have cross-referencing problems In this module we also include a test scenario for GROMACS version 5.1.5 with a possible CG potential and all necessary input files. To run it simply run *gmx grompp -f grompp.mdp -c conf.gro -p topol.top -n index.ndx -maxwarn 5; gmx mdrun* using the patched version of GROMACS version 5.1.5 (see above). When *gmx mdrun* finished normally (with the above mentioned setup), we have several mandatory checks to see if the simulation was successful or not. ... ... @@ -311,5 +313,3 @@ When *gmx mdrun* finished normally (with the above mentioned setup), we have sev The files for the water example can be found here: :download:spc-example.tar.gz
 .. In ReStructured Text (ReST) indentation and spacing are very important (it is how ReST knows what to do with your document). For ReST to understand what you intend and to render it correctly please to keep the structure of this template. Make sure that any time you use ReST syntax (such as for ".. sidebar::" below), it needs to be preceded and followed by white space (if you see warnings when this file is built they this is a common origin for problems). .. Firstly, let's add technical info as a sidebar and allow text below to wrap around it. This list is a work in progress, please help us improve it. We use *definition lists* of ReST_ to make this readable. .. sidebar:: Software Technical Information Name GC-AdResS -Abrupt scheme- Force Capping Language Implemented in GROMACS version 5.1.5 Licence See GROMACS web page: _ Documentation Tool Application Documentation See GROMACS web page: _ Relevant Training Material See GROMACS web page: _ .. In the next line you have the name of how this module will be referenced in the main documentation (which you can reference, in this case, as ":ref:example"). You *MUST* change the reference below from "example" to something unique otherwise you will cause cross-referencing errors. The reference must come right before the heading for the reference to work (so don't insert a comment between). _example: