[Feature] V-D4RL (#1756)

pytorch · Dec 21, 2023 · 6d217c6 · 6d217c6 · github-actions · Dec 21, 2023
1 parent d125dd9
commit 6d217c6
Show file tree

Hide file tree

Showing 32 changed files with 1,077 additions and 42 deletions.
diff --git a/.github/unittest/linux_libs/scripts_vd4rl/environment.yml b/.github/unittest/linux_libs/scripts_vd4rl/environment.yml
@@ -0,0 +1,22 @@
+channels:
+  - pytorch
+  - defaults
+dependencies:
+  - pip
+  - pip:
+    - hypothesis
+    - future
+    - cloudpickle
+    - pytest
+    - pytest-cov
+    - pytest-mock
+    - pytest-instafail
+    - pytest-rerunfailures
+    - pytest-error-for-skips
+    - expecttest
+    - pyyaml
+    - scipy
+    - hydra-core
+    - huggingface_hub
+    - tqdm
+    - h5py
diff --git a/.github/unittest/linux_libs/scripts_vd4rl/install.sh b/.github/unittest/linux_libs/scripts_vd4rl/install.sh
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+unset PYTORCH_VERSION
+# For unittest, nightly PyTorch is used as the following section,
+# so no need to set PYTORCH_VERSION.
+# In fact, keeping PYTORCH_VERSION forces us to hardcode PyTorch version in config.
+apt-get update && apt-get install -y git wget gcc g++
+#apt-get update && apt-get install -y git wget freeglut3 freeglut3-dev
+
+set -e
+
+eval "$(./conda/bin/conda shell.bash hook)"
+conda activate ./env
+
+if [ "${CU_VERSION:-}" == cpu ] ; then
+    version="cpu"
+else
+    if [[ ${#CU_VERSION} -eq 4 ]]; then
+        CUDA_VERSION="${CU_VERSION:2:1}.${CU_VERSION:3:1}"
+    elif [[ ${#CU_VERSION} -eq 5 ]]; then
+        CUDA_VERSION="${CU_VERSION:2:2}.${CU_VERSION:4:1}"
+    fi
+    echo "Using CUDA $CUDA_VERSION as determined by CU_VERSION ($CU_VERSION)"
+    version="$(python -c "print('.'.join(\"${CUDA_VERSION}\".split('.')[:2]))")"
+fi
+
+
+# submodules
+git submodule sync && git submodule update --init --recursive
+
+printf "Installing PyTorch with %s\n" "${CU_VERSION}"
+if [ "${CU_VERSION:-}" == cpu ] ; then
+    # conda install -y pytorch torchvision cpuonly -c pytorch-nightly
+    # use pip to install pytorch as conda can frequently pick older release
+#    conda install -y pytorch cpuonly -c pytorch-nightly
+    pip3 install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cpu
+else
+    pip3 install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu121
+fi
+
+# install tensordict
+pip install git+https://github.com/pytorch/tensordict.git
+
+# smoke test
+python -c "import functorch;import tensordict"
+
+printf "* Installing torchrl\n"
+python setup.py develop
+
+# smoke test
+python -c "import torchrl"
diff --git a/.github/unittest/linux_libs/scripts_vd4rl/post_process.sh b/.github/unittest/linux_libs/scripts_vd4rl/post_process.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+set -e
+
+eval "$(./conda/bin/conda shell.bash hook)"
+conda activate ./env
Benchmark suite	Current: `6d217c6`	Previous: `d125dd9`	Ratio
`benchmarks/test_objectives_benchmarks.py::test_dqn_speed`	`118.6655178190302` iter/sec (`stddev: 0.008177005011271726`)	`612.5660643897461` iter/sec (`stddev: 0.00036650169250447654`)	`5.16`
`benchmarks/test_objectives_benchmarks.py::test_ddpg_speed`	`62.96666856661682` iter/sec (`stddev: 0.0035689927738724973`)	`268.6158002699967` iter/sec (`stddev: 0.0006092554852675664`)	`4.27`
`benchmarks/test_objectives_benchmarks.py::test_sac_speed`	`32.22077638486197` iter/sec (`stddev: 0.001485809986045899`)	`96.76345907912162` iter/sec (`stddev: 0.0008145697363435426`)	`3.00`
`benchmarks/test_objectives_benchmarks.py::test_td3_speed`	`46.00036175797715` iter/sec (`stddev: 0.0012374778327018637`)	`95.85938294170964` iter/sec (`stddev: 0.0008614043119327969`)	`2.08`
`benchmarks/test_objectives_benchmarks.py::test_cql_speed`	`11.13572372637063` iter/sec (`stddev: 0.002633927797375412`)	`24.786932552619238` iter/sec (`stddev: 0.0014965218503312572`)	`2.23`
`benchmarks/test_objectives_benchmarks.py::test_a2c_speed`	`35.618530863743274` iter/sec (`stddev: 0.001469944119404085`)	`122.20455492444225` iter/sec (`stddev: 0.0007606526948799705`)	`3.43`
`benchmarks/test_objectives_benchmarks.py::test_ppo_speed`	`34.95654662870257` iter/sec (`stddev: 0.0016405182281246735`)	`117.45764223487465` iter/sec (`stddev: 0.0007633809455745516`)	`3.36`
`benchmarks/test_objectives_benchmarks.py::test_reinforce_speed`	`37.731461753137836` iter/sec (`stddev: 0.0003893965041641839`)	`137.02676781723414` iter/sec (`stddev: 0.000803828721691577`)	`3.63`
Benchmark suite	Current: `6d217c6`	Previous: `d125dd9`	Ratio
`benchmarks/test_objectives_benchmarks.py::test_dqn_speed`	`132.43734783149543` iter/sec (`stddev: 0.00025272888708596493`)	`672.0861593230526` iter/sec (`stddev: 0.00008079168709534294`)	`5.07`
`benchmarks/test_objectives_benchmarks.py::test_ddpg_speed`	`62.509311647182926` iter/sec (`stddev: 0.010210026280507493`)	`295.61924243910386` iter/sec (`stddev: 0.0001717703687373524`)	`4.73`
`benchmarks/test_objectives_benchmarks.py::test_sac_speed`	`33.74354390394762` iter/sec (`stddev: 0.0001481905097606889`)	`105.46581902608443` iter/sec (`stddev: 0.0001389888318879231`)	`3.13`
`benchmarks/test_objectives_benchmarks.py::test_redq_speed`	`28.18623431385413` iter/sec (`stddev: 0.0003695834530375335`)	`59.17962027154041` iter/sec (`stddev: 0.00013803768739019626`)	`2.10`
`benchmarks/test_objectives_benchmarks.py::test_td3_speed`	`50.16031248912739` iter/sec (`stddev: 0.0009876355919231584`)	`102.81568011106538` iter/sec (`stddev: 0.000968408833367249`)	`2.05`
`benchmarks/test_objectives_benchmarks.py::test_cql_speed`	`11.846214364156348` iter/sec (`stddev: 0.0008803835841211234`)	`29.73765776723217` iter/sec (`stddev: 0.000348129879197689`)	`2.51`
`benchmarks/test_objectives_benchmarks.py::test_a2c_speed`	`37.00614709188471` iter/sec (`stddev: 0.00016750548219076437`)	`138.84230371597474` iter/sec (`stddev: 0.00020649052314402418`)	`3.75`
`benchmarks/test_objectives_benchmarks.py::test_ppo_speed`	`36.70858821200184` iter/sec (`stddev: 0.00030782047581809395`)	`132.6776826823408` iter/sec (`stddev: 0.000162087903086183`)	`3.61`
`benchmarks/test_objectives_benchmarks.py::test_reinforce_speed`	`37.96610146797272` iter/sec (`stddev: 0.000264569881035583`)	`161.83660167541046` iter/sec (`stddev: 0.00019594841932310132`)	`4.26`
`benchmarks/test_objectives_benchmarks.py::test_iql_speed`	`17.296038885387688` iter/sec (`stddev: 0.0004534155499948286`)	`36.427269307510045` iter/sec (`stddev: 0.000409049603048344`)	`2.11`