update

javyduck · Jun 17, 2024 · abb9862 · abb9862
1 parent 431454d
commit abb9862
Show file tree

Hide file tree

Showing 1,760 changed files with 787,601 additions and 3 deletions.
diff --git a/LICENSE b/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020: Safebench Team.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/README.md b/README.md
@@ -1,5 +1,155 @@
-# ChatScene
+<h3 align="center"><a href="https://arxiv.org/abs/2405.14062" style="color:#9C276A">
+ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles</a></h3>
 
-Here is the repository for the CVPR 2024 paper: ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles.
+## Installation
+
+This repository utilizes [Safebench](https://github.com/trust-ai/SafeBench) and [Scenic](https://github.com/BerkeleyLearnVerify/Scenic), we acknowledge and thank these projects for their contributions!
+
+**Recommended system: Ubuntu 20.04 or 22.04**
+
+### 1. Local Installation
+
+Step 1: Setup conda environment
+
+```
+conda create -n chatscene python=3.8
+conda activate chatscene
+```
+
+Step 2: Clone this git repo in an appropriate folder
+
+```
+git clone git@github.com:javyduck/ChatScene.git
+```
+
+Step 3: Enter the repo root folder and install the packages:
+
+```
+cd ChatScene
+pip install -r requirements.txt
+pip install -e .
+```
+
+Step 4: Install the Scenic package:
+
+```
+cd Scenic
+python -m pip install -e .
+```
+
+Step 5: Download our [CARLA_0.9.13](https://drive.google.com/file/d/139vLRgXP90Zk6Q_du9cRdOLx7GJIw_0v/view?usp=sharing) and extract it to your folder.
+
+Step 6: Run `sudo apt install libomp5` as per this [git issue](https://github.com/carla-simulator/carla/issues/4498).
+
+Step 7: Add the python API of CARLA to the ```PYTHONPATH``` environment variable. You can add the following commands to your `~/.bashrc`:
+
+```
+export CARLA_ROOT={path/to/your/carla}
+export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla/dist/carla-0.9.13-py3.8-linux-x86_64.egg
+export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla/agents
+export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla
+export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI
+```
+
+Then, do `source ~/.bashrc` to update the environment variable.
+
+## CARLA Setup
+
+### 1. Desktop Users
+
+Enter the CARLA root folder, launch the CARLA server and run our platform with
+
+```
+# Launch CARLA
+./CarlaUE4.sh -prefernvidia -windowed -carla-port=2000
+```
+
+### 2. Remote Server Users
+
+Enter the CARLA root folder, launch the CARLA server with headless mode, and run our platform with
+
+```
+# Launch CARLA
+./CarlaUE4.sh -prefernvidia -RenderOffScreen -carla-port=2000
+```
+
+(Optional) You can also visualize the pygame window using [TurboVNC](https://sourceforge.net/projects/turbovnc/files/).
+First, launch CARLA with headless mode, and run our platform on a virtual display.
+
+```
+# Launch CARLA
+./CarlaUE4.sh -prefernvidia -RenderOffScreen -carla-port=2000
+
+# Run a remote VNC-Xserver. This will create a virtual display "8".
+/opt/TurboVNC/bin/vncserver :8 -noxstartup
+```
+
+You can use the TurboVNC client on your local machine to connect to the virtual display.
+
+```
+# Use the built-in SSH client of TurboVNC Viewer
+/opt/TurboVNC/bin/vncviewer -via user@host localhost:n
+
+# Or you can manually forward connections to the remote server by
+ssh -L fp:localhost:5900+n user@host
+# Open another terminal on local machine
+/opt/TurboVNC/bin/vncviewer localhost::fp
+```
+
+where `user@host` is your remote server, `fp` is a free TCP port on the local machine, and `n` is the display port specified when you started the VNC server on the remote server ("8" in our example).
+
+## ChatScene
+
+In ChatScene, we ensure a fair comparison with the baselines by using the same eight scenarios, sampling five behaviors for each scenario from the database. The corresponding generated comeplete Scenic files, with some modifications, have been provided in `safebench/scenario/scenario_data/scenic_data` (with some manual modifications to use the same fixed 10 routes for the ego agent to ensure fair comparison with the baselines).
+
+The ego agent is controlled by a default RL model, while the surrounding adversarial agent is controlled by Scenic.
+
+The agent configuration is provided in `safebench/agent/config/adv_scenic.yaml`. By default, it loads a pretrained RL model from Safebench-v1.
+
+### Modes in ChatScene:
+
+1. **train_scenario**: Select the most challenging scenes for the same behavior under the same scenario.
+
+   Configuration can be found in `safebench/scenario/config/train_agent_scenic.yaml`.
+
+   The `sample_num = 50, opt_step = 10, select_num = 2` settings in the file mean we sample 50 scenes and select the 2 most challenging ones for evaluation. The default setting is to choose scenes that lead to a collision of the ego agent and provide the lowest overall score. We optimize the range of parameters, like speed, every 10 steps based on collision statistics from previously sampled scenes.
+
+   Example command for optimizing the scene:
+
+   ```
+   python scripts/run_train.py --agent_cfg=adv_scenic.yaml --scenario_cfg=train_scenario_scenic.yaml --mode train_scenario --scenario_id 1
+   ```
+
+   Use the following command if you are using a TurboVNC client on your local machine to connect to the virtual display:
+
+   ```
+   DISPLAY=:8 python scripts/run_train.py --agent_cfg=adv_scenic.yaml --scenario_cfg=train_scenario_scenic.yaml --mode train_scenario --scenario_id 1
+   ```
+
+   The IDs for the final selected scenes will be stored in `safebench/scenario/scenario_data/scenic_data/scenario_1/scenario_1.json`.
+
+2. **train_agent**: Train the agent based on the selected challenging scenes:
+
+   ```
+   python scripts/run_train.py --agent_cfg=adv_scenic.yaml --scenario_cfg=train_agent_scenic.yaml --mode train_agent --scenario_id 1
+   ```
+
+   We have a total of 10 routes for each scenario. We use the first 8 for training and the last 2 for testing (route IDs: `[0,1,2,3,4,5,6,7]`). The configuration, including `scenario_1.json`, will train the agent based on the most challenging scenes (the ones leading to a collision of the ego agent).
+
+3. **eval**: Evaluate the trained agent on the last 2 routes (route IDs: `[8,9]`), the `test_epoch` is for loading a finetuned model after a specific training epoch:
+
+   ```
+   python scripts/run_eval.py --agent_cfg=adv_scenic.yaml --scenario_cfg=eval_scenic.yaml --mode eval --scenario_id 1 --test_epoch -1
+   ```
+
+The `-1` here is for loading our provided fine-tuned agent in each scenario based on our Scenic scenarios in `safebench/agent/model_ckpt/adv_train/sac/scenic/scenario_1/model.sac.-001.torch`.
+
+## Coming Soon: Dynamic Mode
+
+The above part ensures using the same scenario and routes for fair comparison with baselines. However, ChatScene can generate scenarios and scenes freely without any constraints. Simply provide a text description, such as "*The ego vehicle is driving on a straight road; the adversarial pedestrian suddenly crosses the road from the right front and suddenly stops in front of the ego.*" is enough for the training. We are currently integrating our database with GPT-4o for generating more diverse scenarios based on our pre-built retrieval database, and will upload both soonly.
+
+- [ ] Integrate GPT-4o with our retrieval database and commit the dynamic mode. ETA: within one week. :-)
+- [ ] Finetune an LLM for generating snippets end-to-end based on the data constructed from our database.
+
+If you have any questions, please open an issue or email [jiaweiz7@illinois.edu](mailto:jiaweiz7@illinois.edu). We aim to resolve your issues as soon as possible!
 
-We will provide two branches: one for finetuning the ego agent based on the scenarios chosen in paper, and another for generating safety-critical scenarios based on natural descriptions soon.
diff --git a/Scenic/.coveragerc b/Scenic/.coveragerc
@@ -0,0 +1,22 @@
+# .coveragerc to control coverage.py
+[run]
+omit =
+    src/scenic/simulators/webots/WBT*.py
+
+[report]
+# Regexes for lines to exclude from consideration
+exclude_lines =
+    # Have to re-enable the standard pragma
+    pragma: no cover
+
+    # Don't complain about missing debug-only code:
+    def __repr__
+
+    # Don't complain if tests don't hit defensive assertion code:
+    raise AssertionError
+    raise NotImplementedError
+
+    # Don't complain about abstract methods, they aren't run:
+    @(abc\.)?abstractmethod
+
+ignore_errors = True
diff --git a/Scenic/.github/workflows/run-tests.yml b/Scenic/.github/workflows/run-tests.yml
@@ -0,0 +1,32 @@
+name: tests
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: true
+      matrix:
+        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Update pip
+      run: |
+        python -m pip install --upgrade pip
+    - name: Install Scenic and dependencies
+      run: |
+        python -m pip install -e ".[test]"
+    - name: Run pytest
+      run: |
+        pytest
diff --git a/Scenic/.gitignore b/Scenic/.gitignore
@@ -0,0 +1,133 @@
+# Autogenerated documentation
+docs/modules
+
+# Poetry lock file
+poetry.lock
+
+# Scenic cache files
+*.snet
+
+# Webots temporary files
+.*.wbproj
+
+# OS X junk
+.DS_Store
+
+# Sublime Text files
+*.sublime-project
+*.sublime-workspace
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+docs/_autosummary/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+*.cproject
diff --git a/Scenic/.readthedocs.yaml b/Scenic/.readthedocs.yaml
@@ -0,0 +1,27 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.10"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Optionally build your docs in additional formats such as PDF and ePub
+formats: all
+
+# Optionally declare the Python requirements required to build your docs
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - dev