• DP-GEN Manual
  • Table of Contents
  • About DP-GEN
    • Highlighted features
    • Code structure and interface
    • License and credits
  • Download and Install
  • Init: Preparing Initial Data
    • Init_bulk
    • Init_surf
  • Run: Main Process of Generator
  • Test: Auto-test for Deep Generator
  • Set up machine
  • Troubleshooting
  • License

DP-GEN (Deep Generator) is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is dependent on DeepMD-kit. With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results.

If you use this software in any publication, please cite:

Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E, DP-GEN: A concurrent learning platform for the generation of reliable deep learning based potential energy models, Computer Physics Communications, 2020, 107206.

• Accurate and efficient: DP-GEN is capable to sample more than tens of million structures and select only a few for first principles calculation. DP-GEN will finally obtain a uniformly accurate model.
• User-friendly and automatic: Users may install and run DP-GEN easily. Once succusefully running, DP-GEN can dispatch and handle all jobs on HPCs, and thus there's no need for any personal effort.
• Highly scalable: With modularized code structures, users and developers can easily extend DP-GEN for their most relevant needs. DP-GEN currently supports for HPC systems (Slurm, PBS, LSF and cloud machines ), Deep Potential interface with DeePMD-kit, MD interface with LAMMPS, Gromacs and ab-initio calculation interface with VASP, PWSCF, CP2K, SIESTA and Gaussian, Abacus, PWMAT, etc . We're sincerely welcome and embraced to users' contributions, with more possibilities and cases to use DP-GEN.

• dpgen:

  • data: source codes for preparing initial data of bulk and surf systems.

  • generator: source codes for main process of deep generator.

  • auto_test : source code for undertaking materials property analysis.

  • remote and dispatcher : source code for automatically submiting scripts,maintaining job queues and collecting results. Notice this part hase been integrated into dpdispatcher

  • database : source code for collecting data generated by DP-GEN and interface with database.

• examples : providing example JSON files.

• tests : unittest tools for developers.

One can easily run DP-GEN with :

dpgen TASK PARAM MACHINE

where TASK is the key word, PARAM and MACHINE are both JSON files.

Options for TASK:

• init_bulk : Generating initial data for bulk systems.
• init_surf : Generating initial data for surface systems.
• run : Main process of Deep Generator.
• test: Auto-test for Deep Potential.
• db: Collecting data from DP-GEN.

Here are examples you can refer to. You should make sure that provide a correct JSON file. You can use following command to check your JSON file.

import json
#Specify machine parameters in machine.json
json.load(open("machine.json"))

One can download the source code of dpgen by

git clone https://github.com/deepmodeling/dpgen.git

then you may install DP-GEN easily by:

cd dpgen
pip install --user .

With this command, the dpgen executable is install to $HOME/.local/bin/dpgen. You may want to export the PATH by

export PATH=$HOME/.local/bin:$PATH

To test if the installation is successful, you may execute

dpgen -h

You may prepare initial data for bulk systems with VASP by:

dpgen init_bulk PARAM [MACHINE]

The MACHINE configure file is optional. If this parameter exists, then the optimization tasks or MD tasks will be submitted automatically according to MACHINE.json.

Basically init_bulk can be devided into four parts , denoted as stages in PARAM:

1. Relax in folder 00.place_ele
2. Pertub and scale in folder 01.scale_pert
3. Run a shor AIMD in folder 02.md
4. Collect data in folder 02.md.

All stages must be in order. One doesn't need to run all stages. For example, you may run stage 1 and 2, generating supercells as starting point of exploration in dpgen run.

If MACHINE is None, there should be only one stage in stages. Corresponding tasks will be generated, but user's intervention should be involved in, to manunally run the scripts.

Following is an example for PARAM, which generates data from a typical structure hcp.

{
    "stages" : [1,2,3,4],
    "cell_type":    "hcp",
    "latt":     4.479,
    "super_cell":   [2, 2, 2],
    "elements":     ["Mg"],
    "potcars":      ["....../POTCAR"],
    "relax_incar": "....../INCAR_metal_rlx",
    "md_incar" : "....../INCAR_metal_md",
    "scale":        [1.00],
    "skip_relax":   false,
    "pert_numb":    2,
    "md_nstep" : 5,
    "pert_box":     0.03,
    "pert_atom":    0.01,
    "coll_ndata":   5000,
    "type_map" : [ "Mg", "Al"],
    "_comment":     "that's all"
}

If you want to specify a structure as starting point for init_bulk, you may set in PARAM as follows.

"from_poscar":	true,
"from_poscar_path":	"....../C_mp-47_conventional.POSCAR",

init_bulk support both VASP and ABACUS for first-principle calculation. You can choose the software by specifying the key init_fp_style. If init_fp_style is not specified, the default software will be VASP.

When using ABACUS for init_fp_style, the keys of the paths of INPUT files for relaxation and MD simulations are the same as INCAR for VASP, which are relax_incar and md_incar respectively. You have to additionally specify relax_kspacing and md_kspacing for k points spacing, and dpgen will automatically generate KPT files according to them. You may also use relax_kpt and md_kpt instead of them for the relative path for KPT files of relaxation and MD simulations. However, either relax_kspacing and md_kspacing, or relax_kpt and md_kpt is needed. If from_poscar is set to false, you have to specify atom_masses in the same order as elements.

The following table gives explicit descriptions on keys in PARAM.

The bold notation of key (such as Elements) means that it's a necessary key.

You may prepare initial data for surface systems with VASP by:

dpgen init_surf PARAM [MACHINE]

The MACHINE configure file is optional. If this parameter exists, then the optimization tasks or MD tasks will be submitted automatically according to MACHINE.json.

Basically init_surf can be devided into two parts , denoted as stages in PARAM:

1. Build specific surface in folder 00.place_ele
2. Pertub and scale in folder 01.scale_pert

All stages must be in order.

Following is an example for PARAM, which generates data from a typical structure hcp.

{
  "stages": [
    1,
    2
  ],
  "cell_type": "fcc",
  "latt": 4.034,
  "super_cell": [
    2,
    2,
    2
  ],
  "layer_numb": 3,
  "vacuum_max": 9,
  "vacuum_resol": [
    0.5,
    1
  ],
  "mid_point": 4.0,
  "millers": [
    [
      1,
      0,
      0
    ],
    [
      1,
      1,
      0
    ],
    [
      1,
      1,
      1
    ]
  ],
  "elements": [
    "Al"
  ],
  "potcars": [
    "....../POTCAR"
  ],
  "relax_incar": "....../INCAR_metal_rlx_low",
  "scale": [
    1.0
  ],
  "skip_relax": true,
  "pert_numb": 2,
  "pert_box": 0.03,
  "pert_atom": 0.01,
  "_comment": "that's all"
}

Another example is from_poscar method. Here you need to specify the POSCAR file.

{
  "stages": [
    1,
    2
  ],
  "cell_type": "fcc",
  "from_poscar":	true,
  "from_poscar_path":	"POSCAR",
  "super_cell": [
    1,
    1,
    1
  ],
  "layer_numb": 3,
  "vacuum_max": 5,
  "vacuum_resol": [0.5,2],
  "mid_point": 2.0,
  "millers": [
    [
      1,
      0,
      0
    ]
  ],
  "elements": [
    "Al"
  ],
  "potcars": [
    "./POTCAR"
  ],
  "relax_incar" : "INCAR_metal_rlx_low",
  "scale": [
    1.0
  ],
  "skip_relax": true,
  "pert_numb": 5,
  "pert_box": 0.03,
  "pert_atom": 0.01,
  "coll_ndata": 5000,
  "_comment": "that's all"
}

The following table gives explicit descriptions on keys in PARAM.

The bold notation of key (such as Elements) means that it's a necessary key.

You may call the main process by: dpgen run PARAM MACHINE.

The whole process of generator will contain a series of iterations, succussively undertaken in order such as heating the system to certain temperature.

In each iteration, there are three stages of work, namely, 00.train 01.model_devi 02.fp.

• 00.train: DP-GEN will train several (default 4) models based on initial and generated data. The only difference between these models is the random seed for neural network initialization.

• 01.model_devi : represent for model-deviation. Model-deviation engine in 01.model_devi can be chosen between Molecular Dynamics(LAMMPS and GROMACS) or Structures Prediction(CALYPSO). DP-GEN will use models obtained from 00.train to run Molecular Dynamics or to run structure optimization with ASE in CALYPSO. Larger deviation for structure properties (default is force of atoms) means less accuracy of the models. Using this criterion, a few structures will be selected and put into next stage 02.fp for more accurate calculation based on First Principles.

• 02.fp : Selected structures will be calculated by first principles methods(default VASP). DP-GEN will obtain some new data and put them together with initial data and data generated in previous iterations. After that a new training will be set up and DP-GEN will enter next iteration!

DP-GEN identifies the current stage by a record file, record.dpgen, which will be created and upgraded by codes.Each line contains two number: the first is index of iteration, and the second ,ranging from 0 to 9 ,records which stage in each iteration is currently running.

0,1,2 correspond to make_train, run_train, post_train. DP-GEN will write scripts in make_train, run the task by specific machine in run_train and collect result in post_train. The records for model_devi and fp stage follow similar rules.

In PARAM, you can specialize the task as you expect.

{
  "type_map": [
    "H",
    "C"
  ],
  "mass_map": [
    1,
    12
  ],
  "init_data_prefix": "....../init/",
  "init_data_sys": [
    "CH4.POSCAR.01x01x01/02.md/sys-0004-0001/deepmd"
  ],

  "sys_configs_prefix": "....../init/",
  "sys_configs": [
    [
      "CH4.POSCAR.01x01x01/01.scale_pert/sys-0004-0001/scale*/00000*/POSCAR"
    ],
    [
      "CH4.POSCAR.01x01x01/01.scale_pert/sys-0004-0001/scale*/00001*/POSCAR"
    ]
  ],
 
  "_comment": " that's all ",
  "numb_models": 4,
  "default_training_param": {
     "model": {
            "type_map": [
                "H",
                "C"
            ],
            "descriptor": {
                "type": "se_a",
                "sel": [
                    16,
                    4
                ],
                "rcut_smth": 0.5,
                "rcut": 5,
                "neuron": [
                    120,
                    120,
                    120
                ],
                "resnet_dt": true,
                "axis_neuron": 12,
                "seed": 1
            },
            "fitting_net": {
                "neuron": [
                    25,
                    50,
                    100
                ],
                "resnet_dt": false,
                "seed": 1
            }
        },
        "learning_rate": {
            "type": "exp",
            "start_lr": 0.001,
            "decay_steps": 100,
            "decay_rate": 0.95
        },
        "loss": {
            "start_pref_e": 0.02,
            "limit_pref_e": 2,
            "start_pref_f": 1000,
            "limit_pref_f": 1,
            "start_pref_v": 0.0,
            "limit_pref_v": 0.0
        },
        "training": {
            "set_prefix": "set",
            "stop_batch": 2000,
            "batch_size": 1,
            "disp_file": "lcurve.out",
            "disp_freq": 1000,
            "numb_test": 4,
            "save_freq": 1000,
            "save_ckpt": "model.ckpt",
            "load_ckpt": "model.ckpt",
            "disp_training": true,
            "time_training": true,
            "profiling": false,
            "profiling_file": "timeline.json",
            "_comment": "that's all"
        }
  },
  "model_devi_dt": 0.002,
  "model_devi_skip": 0,
  "model_devi_f_trust_lo": 0.05,
  "model_devi_f_trust_hi": 0.15,
  "model_devi_clean_traj": true,
  "model_devi_jobs": [
    {
      "sys_idx": [
        0
      ],
      "temps": [
        100
      ],
      "press": [
        1.0
      ],
      "trj_freq": 10,
      "nsteps": 300,
      "ensemble": "nvt",
      "_idx": "00"
    },
    {
      "sys_idx": [
        1
      ],
      "temps": [
        100
      ],
      "press": [
        1.0
      ],
      "trj_freq": 10,
      "nsteps": 3000,
      "ensemble": "nvt",
      "_idx": "01"
    }
  ],
  "fp_style": "vasp",
  "shuffle_poscar": false,
  "fp_task_max": 20,
  "fp_task_min": 1,
  "fp_pp_path": "....../methane/",
  "fp_pp_files": [
    "POTCAR"
  ],
  "fp_incar": "....../INCAR_methane"
}

The following table gives explicit descriptions on keys in PARAM.

The bold notation of key (such aas type_map) means that it's a necessary key.

One can choose the model-deviation engine by specifying the key model_devi_engine. If model_devi_engine is not specified, the default model-deviation engine will be LAMMPS.

There are some new keys needed to be added into param and machine if CALYPSO as model-deviation engine.

The bold notation of key (such as calypso_path) means that it's a necessary key.

Converting cp2k input is very simple as dictionary used to dpgen input. You just need follow some simple rule:

• kind section parameter must be provide
• replace keyword in cp2k as keyword in dict.
• replace keyword parameter in cp2k as value in dict.
• replace section name in cp2k as keyword in dict. . The corresponding value is a dict.
• repalce section parameter in cp2k as value with dict. keyword "_"
• repeat section in cp2k just need to be written once with repeat parameter as list.

If you want to use your own paramter, just write a corresponding dictionary. The COORD section will be filled by dpgen automatically, therefore do not include this in dictionary. The OT or Diagonalization section is require for semiconductor or metal system. For specific example, have a look on example directory.

Here are examples for setting:

#minimal information you should provide for input
#other we have set other parameters in code, if you want to
#use your own paramter, just write a corresponding dictionary
"user_fp_params":   {
    "FORCE_EVAL":{
        "DFT":{
            "BASIS_SET_FILE_NAME": "path",
            "POTENTIAL_FILE_NAME": "path",
            "SCF":{
                "OT":{ "keyword":"keyword parameter", "keyword2":"keyword parameter" }
            }
        }
        "SUBSYS":{
            "KIND":{
                "_": ["N","C","H"],
                "POTENTIAL": ["GTH-PBE-q5","GTH-PBE-q4", "GTH-PBE-q1"],
                "BASIS_SET": ["DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH","DZVP-MOLOPT-GTH"]
            }
        }
    }
}

See Full example template.inp and dpgen input parameter file in

tests/generator/cp2k_make_fp_files/exinput/template.inp and tests/generator/param-mgo-cp2k-exinput.json

Here is example for provide external input

    {
    "_comment":     " 02.fp ",
     "fp_style":     "cp2k",
     "shuffle_poscar":   false,
     "fp_task_max":  100,
     "fp_task_min":  10,
     "fp_pp_path":   ".",
     "fp_pp_files":  [],
     "external_input_path": "./cp2k_make_fp_files/exinput/template.inp",
     "_comment":     " that's all 
     }

the following essential section should be provided in user template

 &FORCE_EVAL
   # add this line if you need to fit virial
   STRESS_TENSOR ANALYTICAL
   &PRINT
     &FORCES ON
     &END FORCES
     # add this line if you need to fit virial
     &STRESS_TENSOR ON
     &END FORCES
   &END PRINT
   &SUBSYS
     &CELL
       ABC LEFT FOR DPGEN
     &END CELL
     &COORD
     @include coord.xyz
     &END COORD
   &END SUBSYS
&END FORCE_EVAL

At this step, we assume that you have prepared some graph files like graph.*.pb and the particular pseudopotential POTCAR.

The main code of this step is

dpgen test PARAM MACHINE

where PARAM and MACHINE are both json files. MACHINE is the same as above.

The whole program contains a series of tasks shown as follows. In each task, there are three stages of work, generate, run and compute.

• 00.equi:(default task) the equilibrium state

• 01.eos: the equation of state

• 02.elastic: the elasticity like Young's module

• 03.vacancy: the vacancy formation energy

• 04.interstitial: the interstitial formation energy

• 05.surf: the surface formation energy

Dpgen auto_test will auto make dir for each task it tests, the dir name is the same as the dir name. And the test results will in a plain text file named result. For example cat ./01.eos/Al/std-fcc/deepmd/result

We take Al as an example to show the parameter settings of param.json. The first part is the fundamental setting for particular alloy system.

    "_comment": "models",
    "potcar_map" : {
	"Al" : "/somewhere/POTCAR"
    },
    "conf_dir":"confs/Al/std-fcc",
    "key_id":"API key of Material project",
    "task_type":"deepmd",
    "task":"eos",

You need to add the specified paths of necessary POTCAR files in "potcar_map". The different POTCAR paths are separated by commas. Then you also need to add the folder path of particular configuration, which contains POSCAR file.

"confs/[element or alloy]/[std-* or mp-**]"
std-*: standard structures, * can be fcc, bcc, hcp and so on.
mp-**: ** means Material id from Material Project.

Usually, if you add the relative path of POSCAR as the above format, dpgen test will check the existence of such file and automatically downloads the standard and existed configurations of the given element or alloy from Materials Project and stores them in confs folder, which needs the API key of Materials project.

• task_type contains 3 optional types for testing, i.e. vasp, deepmd and meam.
• task contains 7 options, equi, eos, elastic, vacancy, interstitial, surf and all. The option all can do all the tasks.

It is worth noting that the subsequent tasks need to rely on the calculation results of the equilibrium state, so it is necessary to give priority to the calculation of the equilibrium state while testing. And due to the stable consideration, we recommand you to test the equilibrium state of vasp before other tests.

The second part is the computational settings for vasp and lammps. According to your actual needs， you can choose to add the paths of specific INCAR or use the simplified INCAR by setting vasp_params. The priority of specified INCAR is higher than using vasp_params. The most important setting is to add the folder path model_dir of deepmd model and supply the corresponding element type map. Besides, dpgen test also is able to call common lammps packages, such as meam.

"relax_incar":"somewhere/relax_incar",
"scf_incar":"somewhere/scf_incar",
"vasp_params":	{
	"ecut":		650,
	"ediff":	1e-6,
	"kspacing":	0.1,
	"kgamma":	false,
	"npar":		1,
	"kpar":		1,
	"_comment":	" that's all "
    },
    "lammps_params":    {
        "model_dir":"somewhere/example/Al_model",
        "type_map":["Al"],
        "model_name":false,
        "model_param_type":false
    },

The last part is the optional settings for various tasks mentioned above. You can change the parameters according to actual needs.

param.json in a dictionary.

The keys in param["vasp_params"] is shown below.

the keys in param["lammps_params"].

    "_comment":"00.equi",
    "store_stable":true,

• store_stable:(boolean) whether to store the stable energy and volume

param.json.

test results

conf_dir:        EpA(eV)  VpA(A^3)
confs/Al/std-fcc  -3.7468   16.511

    "_comment": "01.eos",
    "vol_start":	12,
    "vol_end":		22,
    "vol_step":		0.5,

• vol_start, vol_end and vol_step determine the volumetric range and accuracy of the eos.

test results

conf_dir:confs/Al/std-fcc
VpA(A^3)  EpA(eV)
15.500   -3.7306
16.000   -3.7429
16.500   -3.7468
17.000   -3.7430

    "_comment": "02.elastic",
    "norm_deform":	2e-2,
    "shear_deform":	5e-2,

• norm_deform and shear_deform are the scales of material deformation. This task uses the stress-strain relationship to calculate the elastic constant.

test results

conf_dir:confs/Al/std-fcc
130.50   57.45   54.45    4.24    0.00    0.00
57.61  130.31   54.45   -4.29   -0.00   -0.00
54.48   54.48  133.32   -0.00   -0.00   -0.00
4.49   -4.02   -0.89   33.78    0.00   -0.00
-0.00   -0.00   -0.00   -0.00   33.77    4.29
0.00   -0.00   -0.00   -0.00    4.62   36.86
# Bulk   Modulus BV = 80.78 GPa
# Shear  Modulus GV = 36.07 GPa
# Youngs Modulus EV = 94.19 GPa
# Poission Ratio uV = 0.31

    "_comment":"03.vacancy",
    "supercell":[3,3,3],

• supercell:(list of integer) the supercell size used to generate vacancy defect and interstitial defect

test result

conf_dir:confs/Al/std-fcc
Structure:      Vac_E(eV)  E(eV) equi_E(eV)
struct-3x3x3-000:   0.859  -96.557 -97.416

    "_comment":"04.interstitial",
    "insert_ele":["Al"],
    "reprod-opt":false,

• insert_ele:(list of string) the elements used to generate point interstitial defect
• repord-opt:(boolean) whether to reproduce trajectories of interstitial defect

test result

conf_dir:confs/Al/std-fcc
Insert_ele-Struct: Inter_E(eV)  E(eV) equi_E(eV)
struct-Al-3x3x3-000:   3.919  -100.991 -104.909
struct-Al-3x3x3-001:   2.681  -102.229 -104.909

    "_comment": "05.surface",
    "min_slab_size":	10,
    "min_vacuum_size":	11,
    "_comment": "pert xz to work around vasp bug...",
    "pert_xz":		0.01,
    "max_miller": 2,
    "static-opt":false,
    "relax_box":false,

• min_slab_size and min_vacuum_size are the minimum size of slab thickness and the vacuume width.
• pert_xz is the perturbation through xz direction used to compute surface energy.
• max_miller (integer) is the maximum miller index
• static-opt:(boolean) whether to use atomic relaxation to compute surface energy. if false, the structure will be relaxed.
• relax_box:(boolean) set true if the box is relaxed, otherwise only relax atom positions.

test result

conf_dir:confs/Al/std-fcc
Miller_Indices:         Surf_E(J/m^2) EpA(eV) equi_EpA(eV)
struct-000-m1.1.1m:        0.673     -3.628   -3.747
struct-001-m2.2.1m:        0.917     -3.592   -3.747

To know what actually will dpgen autotest do, including the lammps and vasp script, the input file and atom configuration file auto_test will generate, please refer to https://hackmd.io/@yeql5ephQLaGJGgFgpvIDw/rJY1FO92B

When you have a dataset containing lots of repeated data, this step will help you simplify your dataset. The workflow contains three stages: train, model_devi, and fp. The train stage and the fp stage are as the same as the run step, and the model_devi stage will calculate model deviations of the rest data that has not been confirmed accurate. Data with small model deviations will be confirmed accurate, while the program will pick data from those with large model deviations to the new dataset.

Use the following script to start the workflow:

dpgen simplify param.json machine.json

Here is an example of param.json for QM7 dataset:

{
    "type_map": [
        "C",
        "H",
        "N",
        "O",
        "S"
    ],
    "mass_map": [
        12.011,
        1.008,
        14.007,
        15.999,
        32.065
    ],
    "pick_data": "/scratch/jz748/simplify/qm7",
    "init_data_prefix": "",
    "init_data_sys": [],
    "sys_batch_size": [
        "auto"
    ],
    "numb_models": 4,
    "default_training_param": {
        "model": {
            "type_map": [
                "C",
                "H",
                "N",
                "O",
                "S"
            ],
            "descriptor": {
                "type": "se_a",
                "sel": [
                    7,
                    16,
                    3,
                    3,
                    1
                ],
                "rcut_smth": 1.00,
                "rcut": 6.00,
                "neuron": [
                    25,
                    50,
                    100
                ],
                "resnet_dt": false,
                "axis_neuron": 12
            },
            "fitting_net": {
                "neuron": [
                    240,
                    240,
                    240
                ],
                "resnet_dt": true
            }
        },
        "learning_rate": {
            "type": "exp",
            "start_lr": 0.001,
            "decay_steps": 10,
            "decay_rate": 0.99
        },
        "loss": {
            "start_pref_e": 0.02,
            "limit_pref_e": 1,
            "start_pref_f": 1000,
            "limit_pref_f": 1,
            "start_pref_v": 0,
            "limit_pref_v": 0,
            "start_pref_pf": 0,
            "limit_pref_pf": 0
        },
        "training": {
            "set_prefix": "set",
            "stop_batch": 10000,
            "disp_file": "lcurve.out",
            "disp_freq": 1000,
            "numb_test": 1,
            "save_freq": 1000,
            "save_ckpt": "model.ckpt",
            "load_ckpt": "model.ckpt",
            "disp_training": true,
            "time_training": true,
            "profiling": false,
            "profiling_file": "timeline.json"
        },
        "_comment": "that's all"
    },
    "fp_style": "gaussian",
    "shuffle_poscar": false,
    "fp_task_max": 1000,
    "fp_task_min": 10,
    "fp_pp_path": "/home/jzzeng/",
    "fp_pp_files": [],
    "fp_params": {
        "keywords": "mn15/6-31g** force nosymm scf(maxcyc=512)",
        "nproc": 28,
        "multiplicity": 1,
        "_comment": " that's all "
    },
    "init_pick_number":100,
    "iter_pick_number":100,
    "e_trust_lo":1e10,
    "e_trust_hi":1e10,
    "f_trust_lo":0.25,
    "f_trust_hi":0.45,
    "_comment": " that's all "
}

Here pick_data is the directory to data to simplify where the program recursively detects systems System with deepmd/npy format. init_pick_number and iter_pick_number are the numbers of picked frames. e_trust_lo, e_trust_hi mean the range of the deviation of the frame energy, and f_trust_lo and f_trust_hi mean the range of the max deviation of atomic forces in a frame. fp_style can only be gaussian currently. Other parameters are as the same as those of generator.

dpdispatcher Update Note: dpdispatcher has updated and the api of machine.json is changed. dpgen will use new dpdispatcher if the key api_version in dpgen's machine.json's value is equal or large than 1.0.

And dpgen will use old dpdispatcher if the key api_version is not specified in machine.json or the api_version is smaller than 1.0. This gurantees that the old machine.jsons still work.

And for now dpdispatcher is maintained on a seperate repo. The repo link: https://github.com/deepmodeling/dpdispatcher

The api of new dpdispatcher is close to old one except for a few changes.

The new machine.json examples can be seen here

And Here are the explanations of the keys in machine resources.

Here is a example machine.json for dpgen's new dpdispatcher. Please check the documents for more information about new dpdispatcher.

an example of new dpgen's machine.json

{
  "api_version": "1.0",
  "train":
    {
      "command": "dp",
      "machine": {
        "batch_type": "PBS",
        "context_type": "SSHContext",
        "local_root": "./",
        "remote_root": "/home/user1234/work_path_dpdispatcher_test",
        "remote_profile": {
            "hostname": "39.xxx.xx.xx",
            "username": "user1234"
        }
      },
      "resources": {
        "number_node": 1,
        "cpu_per_node": 4,
        "gpu_per_node": 1,
        "queue_name": "T4_4_15",
        "group_size": 1,
        "custom_flags":["#SBATCH --mem=32G"],
        "strategy": {"if_cuda_multi_devices": true},
        "para_deg": 3,
        "source_list": ["/home/user1234/deepmd.1.2.4.env"]
      }
    },
  "model_devi":
    {
      "command": "lmp",
      "machine":{
        "batch_type": "PBS",
        "context_type": "SSHContext",
        "local_root": "./",
        "remote_root": "/home/user1234/work_path_dpdispatcher_test",
        "remote_profile": {
          "hostname": "39.xxx.xx.xx",
          "username": "user1234"
        }
      },
      "resources": {
        "number_node": 1,
        "cpu_per_node": 4,
        "gpu_per_node": 1,
        "queue_name": "T4_4_15",
        "group_size": 5,
        "source_list": ["/home/user1234/deepmd.1.2.4.env"]
      }
    },
  "fp":
    {
      "command": "vasp_std",
      "machine":{
        "batch_type": "PBS",
        "context_type": "SSHContext",
        "local_root": "./",
        "remote_root": "/home/user1234/work_path_dpdispatcher_test",
        "remote_profile": {
          "hostname": "39.xxx.xx.xx",
          "username": "user1234"
        }
      },
      "resources": {
        "number_node": 1,
        "cpu_per_node": 32,
        "gpu_per_node": 0,
        "queue_name": "G_32_128",
        "group_size": 1,
        "source_list": ["~/vasp.env"]
      }
    }
}

note1: the key "local_root" in dpgen's machine.json is always ./

When switching into a new machine, you may modifying the MACHINE, according to the actual circumstance. Once you have finished, the MACHINE can be re-used for any DP-GEN tasks without any extra efforts.

An example for MACHINE is:

{
  "train":
    {
      "machine": {
        "batch": "slurm",
        "hostname": "localhost",
        "port": 22,
        "username": "Angus",
        "work_path": "....../work"
      },
      "resources": {
        "numb_node": 1,
        "numb_gpu": 1,
        "task_per_node": 4,
        "partition": "AdminGPU",
        "exclude_list": [],
        "source_list": [
          "....../train_tf112_float.env"
        ],
        "module_list": [],
        "time_limit": "23:0:0",
        "qos": "data"
      },
      "command": "USERPATH/dp"
    },
  "model_devi":
    {
      "machine": {
        "batch": "slurm",
        "hostname": "localhost",
        "port": 22,
        "username": "Angus",
        "work_path": "....../work"
      },
      "resources": {
        "numb_node": 1,
        "numb_gpu": 1,
        "task_per_node": 2,
        "partition": "AdminGPU",
        "exclude_list": [],
        "source_list": [
          "......./lmp_tf112_float.env"
        ],
        "module_list": [],
        "time_limit": "23:0:0",
        "qos": "data"
      },
      "command": "lmp_serial",
      "group_size": 1
    },
  "fp":
    {
      "machine": {
        "batch": "slurm",
        "hostname": "localhost",
        "port": 22,
        "username": "Angus",
        "work_path": "....../work"
      },
      "resources": {
        "task_per_node": 4,
        "numb_gpu": 1,
        "exclude_list": [],
        "with_mpi": false,
        "source_list": [],
        "module_list": [
          "mpich/3.2.1-intel-2017.1",
          "vasp/5.4.4-intel-2017.1",
          "cuda/10.1"
        ],
        "time_limit": "120:0:0",
        "partition": "AdminGPU",
        "_comment": "that's All"
      },
      "command": "vasp_gpu",
      "group_size": 1
    }
}

Following table illustrates which key is needed for three types of machine: train,model_devi and fp. Each of them is a list of dicts. Each dict can be considered as an independent environmnet for calculation.

The following table gives explicit descriptions on keys in param.json.

1. The most common problem is whether two settings correspond with each other, including:

   • The order of elements in type_map and mass_map and fp_pp_files.
   • Size of init_data_sys and init_batch_size.
   • Size of sys_configs and sys_batch_size.
   • Size of sel_a and actual types of atoms in your system.
   • Index of sys_configs and sys_idx

2. Please verify the directories of sys_configs. If there isnt's any POSCAR for 01.model_devi in one iteration, it may happen that you write the false path of sys_configs.

3. Correct format of JSON file.

4. In 02.fp, total cores you require through task_per_node should be devided by npar times kpar.

5. The frames of one system should be larger than batch_size and numb_test in default_training_param. It happens that one iteration adds only a few structures and causes error in next iteration's training. In this condition, you may let fp_task_min be larger than numb_test.

The project dpgen is licensed under GNU LGPLv3.0.