-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
two separate instructions for dn-splatter and grounded-sam
- Loading branch information
Showing
3 changed files
with
112 additions
and
106 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
**Note:** Because our major dependencies, `Nerfstudio` and `Grounded-SAM-2`, officially support two different CUDA versions (11.8 vs. 12.1), we will have to create two separate environments. We hope to resolve this in the future when `Nerfstudio` bumps its official CUDA support version. | ||
|
||
|
||
Clone our repo. Make sure to clone the submodule as well by using `--recursive`. | ||
```sh | ||
git clone --recursive https://github.com/ai4ce/FusionSense.git | ||
``` | ||
|
||
Create the environment. | ||
```sh | ||
cd FusionSense | ||
``` | ||
```sh | ||
conda env create -f config.yml | ||
``` | ||
```sh | ||
conda activate fusionsense | ||
``` | ||
|
||
Install compatible **PyTorch** and **cuda-toolkit** version: | ||
|
||
```sh | ||
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118 | ||
``` | ||
```sh | ||
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit | ||
``` | ||
|
||
Install **mmcv**: | ||
|
||
```sh | ||
pip install mmcv | ||
``` | ||
|
||
Install **tinycudann**: | ||
|
||
```sh | ||
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch | ||
``` | ||
|
||
Build the environment | ||
```sh | ||
pip install -e . | ||
``` | ||
Note that this part of our codebase is largely modified from `dn-splatter`, so we did not modify their project name out of respect. | ||
|
||
This environment is largely a mix of `dn-splatter`([doc](https://github.com/maturk/dn-splatter?tab=readme-ov-file#installation)) and `Metric3D`([doc](https://github.com/YvanYin/Metric3D/tree/main)). If you encounter any installation problem, in addition to posting an issue in this repo, you are welcome to checkout their repos as well. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
**Note 1:** Because our major dependencies, `Nerfstudio` and `Grounded-SAM-2`, officially support two different CUDA versions (11.8 vs. 12.1), we will have to create two separate environments. We hope to resolve this in the future when `Nerfstudio` bumps its official CUDA support version. | ||
|
||
**Note 2:** We use `Grounded-SAM-2` for segmenting the foreground and background. Please make sure to use our modified submodule. | ||
|
||
We recommend starting a separate Conda environment since `Grounded-SAM-2` requires CUDA 12.1, which is not yet officially supported by `Nerfstudio`. | ||
|
||
Starting from the root of this repo, first download the checkpoints needed. | ||
```sh | ||
cd Grounded-SAM2-for-masking | ||
``` | ||
```sh | ||
cd checkpoints | ||
``` | ||
```sh | ||
bash download_ckpts.sh | ||
``` | ||
```sh | ||
cd ../gdino_checkpoints | ||
``` | ||
```sh | ||
bash download_ckpts.sh | ||
``` | ||
|
||
Then we create an environment for this part. | ||
```sh | ||
conda create -n G-SAM-2 python=3.10 | ||
``` | ||
```sh | ||
conda activate G-SAM-2 | ||
``` | ||
|
||
We then install `PyTorch 2.3.1` and its friends | ||
```sh | ||
conda install pytorch==2.3.1 torchvision==0.18.1 pytorch-cuda=12.1 -c pytorch -c nvidia | ||
``` | ||
```sh | ||
pip install opencv-python supervision transformers | ||
``` | ||
and then `CUDA 12.1` development kit as we will need it to compile `Deformable Attention` operator used in `Grounded-SAM-2`. | ||
```sh | ||
conda install -c "nvidia/label/cuda-12.1.0" cuda-toolkit | ||
``` | ||
Use `which nvcc`to check that the installation is successful. The result should look like | ||
``` | ||
/home/irving/miniconda3/envs/G-SAM-2/bin/nvcc | ||
``` | ||
Then, the `CUDA_HOME` should be set to | ||
```sh | ||
export CUDA_HOME=/home/irving/miniconda3/envs/G-SAM-2/ | ||
``` | ||
Install Segment Anything 2 | ||
```sh | ||
pip install -e . | ||
``` | ||
Install Grounding DINO. Yes this is not a typo. Grounding DINO is needed to run `Grounded-SAM-2`. | ||
```sh | ||
pip install --no-build-isolation -e grounding_dino | ||
``` | ||
|
||
If you encounter any problem, you can check out `Grounded-SAM2-for-masking`'s official [installation guide](https://github.com/IDEA-Research/Grounded-SAM-2#installation). |