Commit fccd459c authored by Valerio Vitale's avatar Valerio Vitale

Merge branch 'master' into 'W90_non_orthogonal_cube'

# Conflicts:
#   Electronic-Structure-Modules/index.rst
parents 7304f274 0820dab2
......@@ -3,7 +3,7 @@ image: cloudcompass/docker-rtdsphinx
spelling:
script:
- pip3 install codespell
- codespell --skip=".git,_static,_build,Diff*,*.patch" --quiet-level=2 --ignore-words-list="adress"
- codespell --skip=".git,_static,_build,Diff*,*.patch,*.f90" --quiet-level=2 --ignore-words-list="adress,catalogue,noe"
only:
- master
- merge_requests
......@@ -13,7 +13,7 @@ orphans:
# Report all the orphans but ignore the exit code
- find ./ -name "*.rst"|xargs -i grep -H orphan {} || true
# Now handle the error code
- if [ $(find ./ -name "*.rst"|xargs -i grep -H orphan {}|wc -l) -gt "1" ]; then $(exit 1); else $(exit 0); fi
- if [ $(find ./ -name "*.rst"|xargs -i grep -H orphan {}|wc -l) -gt "2" ]; then $(exit 1); else $(exit 0); fi
only:
- master
......
......@@ -173,7 +173,10 @@ separate projects. The modules that were incorporated into the core are:
./modules/OpenPathSampling/ops_channel_analysis/readme
./modules/OpenPathSampling/ops_new_tis_analysis/readme
./modules/OpenPathSampling/ops_resampling_statistics/readme
./modules/OpenPathSampling/ops_gromacs_engine/readme
./modules/OpenPathSampling/ops_visit_all_states/readme
./modules/OpenPathSampling/ops_interface_shooting/readme
The modules that are based on OPS, but remain separate, are:
.. toctree::
......@@ -244,4 +247,14 @@ August 2017. The following modules have been produced:
./modules/OpenPathSampling/ops_plumed_wrapper/readme
./modules/OpenPathSampling/ops_s_shooting/readme
The third ESDW for the Classical MD workpackage was held in Turin, Italy in July
2018. The following have been produced as a result:
.. toctree::
:glob:
:maxdepth: 1
./modules/HTC/decorators/readme
./modules/pybop/readme
.. _E-CAM: https://www.e-cam2020.eu/
.. In ReStructured Text (ReST) indentation and spacing are very important (it is how ReST knows what to do with your
document). For ReST to understand what you intend and to render it correctly please to keep the structure of this
template. Make sure that any time you use ReST syntax (such as for ".. sidebar::" below), it needs to be preceded
and followed by white space (if you see warnings when this file is built they this is a common origin for problems).
.. Firstly, let's add technical info as a sidebar and allow text below to wrap around it. This list is a work in
progress, please help us improve it. We use *definition lists* of ReST_ to make this readable.
.. sidebar:: Software Technical Information
Name
``jobqueue_features``
Language
Python
Licence
`MIT <https://opensource.org/licenses/mit-license>`_
Documentation Tool
In-source documentation
Application Documentation
Not currently available.. Example usage provided.
Relevant Training Material
Not currently available.
Software Module Developed by
Adam Włodarczyk (Wrocław Centre of Networking and Supercomputing),
Alan O'Cais (Juelich Supercomputing Centre)
.. In the next line you have the name of how this module will be referenced in the main documentation (which you can
reference, in this case, as ":ref:`example`"). You *MUST* change the reference below from "example" to something
unique otherwise you will cause cross-referencing errors. The reference must come right before the heading for the
reference to work (so don't insert a comment between).
.. _htc:
#######################################
E-CAM High Throughput Computing Library
#######################################
.. Let's add a local table of contents to help people navigate the page
.. contents:: :local:
.. Add an abstract for a *general* audience here. Write a few lines that explains the "helicopter view" of why you are
creating this module. For example, you might say that "This module is a stepping stone to incorporating XXXX effects
into YYYY process, which in turn should allow ZZZZ to be simulated. If successful, this could make it possible to
produce compound AAAA while avoiding expensive process BBBB and CCCC."
E-CAM is interested in the challenge
of bridging timescales. To study molecular dynamics with atomistic detail, timesteps must be used on
the order of a femtosecond. Many problems in biological chemistry, materials science, and other
fields involve events that only spontaneously occur after a millisecond or longer (for example,
biomolecular conformational changes, or nucleation processes). That means that around :math:`10^{12}` time
steps would be needed to see a single millisecond-scale event. This is the problem of "rare
events" in theoretical and computational chemistry.
Modern supercomputers are beginning to make it
possible to obtain trajectories long enough to observe some of these processes, but to fully
characterize a transition with proper statistics, many examples are needed. In order to obtain many
examples the same application must be run many thousands of times with varying inputs. To manage
this kind of computation a task scheduling high throughput computing (HTC) library is needed. The main elements of mentioned
scheduling library are: task definition, task scheduling and task execution.
While traditionally an HTC workload is looked down upon in the HPC
space, the scientific use case for extreme-scale resources exists and algorithms that require a
coordinated approach make efficient libraries that implement
this approach increasingly important in the HPC space. The 5 Petaflop booster technology of `JURECA <http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JURECA/JURECA_node.html>`_
is an interesting concept with respect to this approach since the offloading approach of heavy
computation marries perfectly to the concept outlined here.
Purpose of Module
_________________
.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
This module is the first in a sequence that will form the overall capabilities of the library. In particular this module
deals with creating a set of decorators to wrap around the `Dask-Jobqueue <https://jobqueue.dask.org/en/latest/>`_
Python library, which aspires to make the development time cost of leveraging it lower for our use cases.
Background Information
______________________
.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
The initial motivation for this library is driven by the ensemble-type calculations that are required in many scientific
fields, and in particular in the materials science domain in which the E-CAM Centre of Excellence operates. The scope
for parallelisation is best contextualised by the `Dask <https://dask.org/>`_ documentation:
A common approach to parallel execution in user-space is task scheduling. In task scheduling we break our program
into many medium-sized tasks or units of computation, often a function call on a non-trivial amount of data. We
represent these tasks as nodes in a graph with edges between nodes if one task depends on data produced by another.
We call upon a task scheduler to execute this graph in a way that respects these data dependencies and leverages
parallelism where possible, multiple independent tasks can be run simultaneously.
Many solutions exist. This is a common approach in parallel execution frameworks. Often task scheduling logic hides
within other larger frameworks (Luigi, Storm, Spark, IPython Parallel, and so on) and so is often reinvented.
Dask is a specification that encodes task schedules with minimal incidental complexity using terms common to all
Python projects, namely dicts, tuples, and callables. Ideally this minimum solution is easy to adopt and understand
by a broad community.
While we were attracted by this approach, Dask did not support *task-level* parallelisation (in particular
multi-node tasks). We researched other options (including Celery, PyCOMPSs, IPyParallel and others) and organised a
workshop that explored some of these (see https://www.cecam.org/workshop-0-1650.html for further details).
Building and Testing
____________________
.. Keep the helper text below around in your module by just adding ".. " in front of it, which turns it into a comment
The library is a Python module and can be installed with
::
python setup.py install
More details about how to install a Python package can be found at, for example, `Install Python packages on the
research computing systems at IU <https://kb.iu.edu/d/acey>`_
To run the tests for the decorators within the library, you need the ``pytest`` Python package. You can run all the
relevant tests from the ``jobqueue_features`` directory with
::
pytest tests/test_decorators.py
Examples of usage can be found in the ``examples`` directory.
Source Code
___________
The latest version of the library is available on the `jobqueue_features GitHub repository
<https://github.com/E-CAM/jobqueue_features>`_, the file specific to this module
is `decorators.py <https://github.com/E-CAM/jobqueue_features/blob/master/jobqueue_features/decorators.py>`_.
(The code that was originally created for this module can be seen in the specific commit `4590a0e427112f
<https://gitlab.e-cam2020.eu/adam/jobqueue_features/tree/4590a0e427112fbf51edff6116e34c90e765baf0>`_
which can be found in the original private repository of the code.)
.. _ops_gromacs_engine:
##################################
Gromacs engine in OpenPathSampling
##################################
.. sidebar:: Software Technical Information
The information in this section describes OpenPathSampling as a whole.
Information specific to the additions in this module are in subsequent
sections.
Language
Python (2.7, 3.6, 3.7)
Documentation Tool
Sphinx, numpydoc format (ReST)
Application Documentation
http://openpathsampling.org
Relevant Training Material
http://openpathsampling.org/latest/examples/
Licence
LGPL, v. 2.1 or later
Authors
David W.H. Swenson
.. contents:: :local:
This module adds support for Gromacs as an engine for OpenPathSampling.
Purpose of Module
_________________
.. Give a brief overview of why the module is/was being created.
Different molecular dynamics (MD) codes have developed to serve different
communities. Gromacs is one of the major MD codes for the biomolecular
community, and even though much of its functionality can be reproduced by
other MD codes, such as OpenMM, there are still some extensions that are
built on top of Gromacs that haven't been ported to other codes. For
example, the MARTINI coarse-grained model is not available other codes such
as OpenMM.
Additionally, people who are familiar with a given MD package will prefer to
continue to work with that. Therefore codes that wrap around MD packages, as
OpenPathSampling does, can expand their reach by adding ways to interface
with other MD packages.
This module adds the Gromacs engine for OpenPathSampling. It is the first
practical test of the external engine API of OPS.
Specific functionality in this module includes:
* ``GromacsEngine``: the OPS dynamics engine, based on the
``ExternalEngine``, that runs Gromacs as an external tool. Option on
initialization allow the user to customize the path to the Gromacs
executable.
* ``ExternalMDSnapshot``: an OPS snapshot for external MD engines, which
contains coordinates, velocities, and box vectors. Requires that the
engine implement a ``read_frame_data`` method to load from a specific MD
trajectory.
* ``snapshot_from_gro``: a function that creates an OPS snapshot from a
Gromacs ``.gro`` file.
Background Information
______________________
This module builds on OpenPathSampling, a Python package for path sampling
simulations. To learn more about OpenPathSampling, you might be interested in
reading:
* OPS documentation: http://openpathsampling.org
* OPS source code: http://github.com/openpathsampling/openpathsampling
Testing
_______
.. IF YOUR MODULE IS IN OPS CORE:
.. This module has been included in the OpenPathSampling core. Its tests can
.. be run by setting up a developer install of OpenPathSampling and running
.. the command ``py.test`` from the root directory of the repository.
This module is in a development branch of OpenPathSampling. If you have
conda installed, this branch of OPS can be installed by downloading the
``conda_ops_dev_install.sh`` script and running it with the command:
.. code:: bash
source conda_ops_dev_install.sh dwhswenson gromacs_engine
This will download a new copy of the OPS git repository, select the
``gromacs_engine`` branch from the ``dwhswenson`` fork, install the
requirements, and create an editable install of OPS. If you would like to do
this in a new conda environment, set the environment variable ``OPS_ENV``,
and it will install in a new environment with the name ``$OPS_ENV``.
To run tests, you may need ``pytest``, which can be installed with ``conda
install pytest``.
The entire OPS test suite can be run with run with ``py.test --pyargs
openpathsampling``. Tests specific to the Gromacs engine can be run with
``py.test --pyargs openpathsampling.tests.test_gromacs_engine``.
.. IF YOUR MODULE IS IN A SEPARATE REPOSITORY
.. The tests for this module can be run by downloading its source code,
.. installing its requirements, and running the command ``nosetests`` from the
.. root directory of the repository.
Examples
________
* An example can be found here: https://github.com/dwhswenson/openpathsampling/tree/gromacs_engine/examples/gromacs
Source Code
___________
.. link the source code
.. IF YOUR MODULE IS IN OPS CORE
.. This module has been merged into OpenPathSampling. It is composed of the
.. following pull requests:
This module is contained in the following pull request:
* https://github.com/openpathsampling/openpathsampling/pull/819
.. IF YOUR MODULE IS A SEPARATE REPOSITORY
.. The source code for this module can be found in: URL.
.. CLOSING MATERIAL -------------------------------------------------------
.. Here are the URL references used
.. _nose: http://nose.readthedocs.io/en/latest/
.. _ops_interface_shooting:
##################################################
Interface-Constrained Shooting in OpenPathSampling
##################################################
.. sidebar:: Software Technical Information
The information in this section describes OpenPathSampling as a whole.
Information specific to the additions in this module are in subsequent
sections.
Language
Python
Documentation Tool
Sphinx, numpydoc format (ReST)
Application Documentation
http://openpathsampling.org
Relevant Training Material
http://openpathsampling.org/latest/examples/
Licence
LGPL, v. 2.1 or later
Software module developed by
Peter G. Bolhuis and David W.H. Swenson
.. contents:: :local:
This module adds interface-constrained shooting to OpenPathSampling.
Interface-constrained shooting is a technique that can improve the
efficiency of transition interface sampling.
Purpose of Module
_________________
In transition interface sampling (TIS), one defines stable states (volumes in
phase space) and interfaces (surfaces in phase space). For a trajectory to
be accepted in TIS, it must begin with exactly one frame in a given initial
state, cross the interface, and end with exactly one frame in any state
volume (including the initial state).
New trajectories are generated with the shooting move, which selects a point
along an initial trajectory from which new frames can be made. In one-way
shooting, the dynamics only needs to run in one direction (with the
stochastic nature of the dynamics ensuring that a new trajectory is
generated).
However, if the interfaces are far from the initial state and if all frames
are equally likely to be used for shooting, it can be very likely for the
shooting point to come before the trajectory has crossed the interface. This
can then lead to shooting moves that usually generate trajectories that
don't cross the interface, and therefore must be rejected. This uses a lot
of simulation effort without generating useful new trajectories.
Interface-constrained shooting (also called "constrained interface
shooting") [1]_ is an approach to solve this problem. Instead of selecting
from anywhere along the trajectory, only the first point after crossing the
interface is allowed as a shooting point. This ensures that every
trajectory that is generated will be valid (will cross the interface). In
addition, because the first crossing is still the first crossing in the new
trajectory, this leads to the Metropolis acceptance probability also being
1. Therefore, every trial trajectory is accepted.
In practice, this must be combined with the path reversal move in order to
sample all of trajectory space. The result is an approach with very high
acceptance, although decorrelation of the trajectory is a little slower.
This module implements interface-constrained shooting using:
* ``ForwardShootingStrategy``: An OPS ``MoveStrategy`` to do forward-only
shooting. The interface-constrained shooting approach uses forward-only
stochastic dynamics, counting on path reversal to handle the backward-time
dynamics.
* ``InterfaceConstrainedSelector``: A ``ShootingPointSelector`` that selects
the first point outside the given volume (the boundary of which defines
the interface).
.. [1] Bolhuis, P. G. (2008). Rare events via multiple reaction channels
sampled by path replica exchange. The Journal of Chemical Physics,
129(11), 114108. https://doi.org/10.1063/1.2976011
Background Information
______________________
This module builds on OpenPathSampling, a Python package for path sampling
simulations. To learn more about OpenPathSampling, you might be interested in
reading:
* OPS documentation: http://openpathsampling.org
* OPS source code: http://github.com/openpathsampling/openpathsampling
Testing
_______
Tests in OpenPathSampling use `pytest`_.
.. IF YOUR MODULE IS IN OPS CORE:
This module has been included in the OpenPathSampling core. Its tests can
be run by installing pytest and OPS (with commit ``c340818``, which will be
part of release 0.9.6 and later), and running the command ``py.test
--pyargs openpathsampling``.
.. IF YOUR MODULE IS IN A SEPARATE REPOSITORY
.. The tests for this module can be run by downloading its source code,
.. installing its requirements, and running the command ``py.test`` from the
.. root directory of the repository.
Examples
________
An example of this is in the following notebook:
* https://github.com/openpathsampling/openpathsampling/blob/7e157661dd8633690ebfcae4a8265fc14e31c5b9/examples/toy_model_mstis/toy_mstis_A4_constrained_shooting.ipynb
Source Code
___________
.. link the source code
.. IF YOUR MODULE IS IN OPS CORE
This module has been merged into OpenPathSampling. It is composed of the
following pull requests:
* https://github.com/openpathsampling/openpathsampling/pull/788
* https://github.com/openpathsampling/openpathsampling/pull/790
* https://github.com/openpathsampling/openpathsampling/pull/800
.. IF YOUR MODULE IS A SEPARATE REPOSITORY
.. The source code for this module can be found in: URL.
.. CLOSING MATERIAL -------------------------------------------------------
.. Here are the URL references used
.. _pytest: http://pytest.org/
......@@ -25,9 +25,10 @@ OPS-based modules
Licence
LGPL, v. 2.1 or later
.. contents:: :local:
Software module developed by
YOUR NAME(S) HERE
Authors: Alan O'Cais
.. contents:: :local:
This is the template for an E-CAM *module* based on OpenPathSampling (OPS). Several
sections are already pre-filled with the details of OPS. Please fill out the
......@@ -66,18 +67,19 @@ reading:
Testing
_______
Tests in OpenPathSampling use the `nose`_ package.
Tests in OpenPathSampling use `pytest`_.
.. IF YOUR MODULE IS IN OPS CORE:
.. This module has been included in the OpenPathSampling core. Its tests can
.. be run by setting up a developer install of OpenPathSampling and running
.. the command ``nosetests`` from the root directory of the repository.
.. be run by installing pytest and OPS (with commit ????????, which will be
.. part of release ??? and later), and running the command ``py.test
.. --pyargs openpathsampling``.
.. IF YOUR MODULE IS IN A SEPARATE REPOSITORY
.. The tests for this module can be run by downloading its source code,
.. installing its requirements, and running the command ``nosetests`` from the
.. installing its requirements, and running the command ``py.test`` from the
.. root directory of the repository.
Examples
......@@ -105,5 +107,5 @@ ___________
.. Here are the URL references used
.. _nose: http://nose.readthedocs.io/en/latest/
.. _pytest: http://pytest.org/
.. _ops_visit_all_states:
#############################
OPS Visit All States Ensemble
#############################
.. sidebar:: Software Technical Information
The information in this section describes OpenPathSampling as a whole.
Information specific to the additions in this module are in subsequent
sections.
Language
Python (2.7, 3.6, 3.7)
Documentation Tool
Sphinx, numpydoc format (ReST)
Application Documentation
http://openpathsampling.org
Relevant Training Material
http://openpathsampling.org/latest/examples/
Licence
LGPL, v. 2.1 or later
Software module developed by
David W.H. Swenson
.. contents:: :local:
This module adds a convenient new OpenPathSampling ensemble that allows
trajectories to continue until they have visited all the states in the
system. In addition, it provides real-time reporting about the progress.
Purpose of Module
_________________
.. Give a brief overview of why the module is/was being created.
One of the ways to get initial trajectories for path sampling is to use
dynamics that aren't physical for the ensemble of interest, such as using an
increased temperature. If a trajectory has a frame in every state, then it
must have subtrajectories that connect from each state to another one, and
therefore it has all the information to start a MSTIS simulation. The
ensemble definition tools in OPS make it easy to create such custom
sequential ensembles [1]_.
However, users often want to do simulation setup in an interactive mode,
such as in a Jupyter notebook, or at a minimum want to have a sense of the
progress made on a long trajectory such as this. The default OPS ensemble
gives no output and therefore no sense of how much progress has been made.
This module provides a custom OPS ensemble that gives such output during its
simulation. It outputs the length of the trajectory so far, as well as the
states that have and have not already been visited. This gives a much better
sense of how long the simulation will take to run.
This module includes:
* ``default_state_progress_report``: A function to create the progress
report string. This can be replaced by another function to customize the
output.
* ``VisitAllStatesEnsemble``: A class that wraps an OPS
``SequentialEnsemble``. The ``can_append`` method, called while generating
the dynamics, can output information about the progress. The behavior of
this output can be set with the initialization variable ``progress``,
which can take the values of ``default`` for the default output,
``silent`` for no output, or can take a 2-tuple of callables where the
first determines what to write, and the second determined how to emit the
information.
.. [1] D.W.H. Swenson, J.-H. Prinz, F. Noe, J.D. Chodera, and P.G. Bolhuis.
J. Chem. Theory Comput. **15**, 837 (2019);
http://doi.org/10.1021/acs.jctc.8b00627
Background Information
______________________
This module builds on OpenPathSampling, a Python package for path sampling
simulations. To learn more about OpenPathSampling, you might be interested in
reading:
* OPS documentation: http://openpathsampling.org
* OPS source code: http://github.com/openpathsampling/openpathsampling
Testing
_______
Tests in OpenPathSampling use `pytest`_.
.. IF YOUR MODULE IS IN OPS CORE:
This module has been included in the OpenPathSampling core. Its tests can
be run by installing pytest and OPS (with commit 8e767872, which will be
part of release 1.0 and later), and running the command ``py.test --pyargs
openpathsampling``.
.. IF YOUR MODULE IS IN A SEPARATE REPOSITORY
.. The tests for this module can be run by downloading its source code,
.. installing its requirements, and running the command ``py.test`` from the
.. root directory of the repository.
Examples
________
This module is used in the OPS `alanine dipeptide MSTIS example <https://github.com/openpathsampling/openpathsampling/blob/master/examples/alanine_dipeptide_mstis/AD_mstis_1_setup.ipynb>`_, during the creation of initial trajectories.
Source Code
___________
.. link the source code
.. IF YOUR MODULE IS IN OPS CORE
This module has been merged into OpenPathSampling. It is composed of the
following pull requests:
* https://github.com/openpathsampling/openpathsampling/pull/826
.. IF YOUR MODULE IS A SEPARATE REPOSITORY
.. The source code for this module can be found in: URL.
.. CLOSING MATERIAL -------------------------------------------------------