[v2 BUG]: "TypeError in Chemprop hpopt: torch.cat() got invalid arguments during metric computation" #1121

TianYaoCSU · 2024-12-07T08:01:56Z

Describe the bug

I encountered a TypeError while running Chemprop's hyperparameter optimization (hpopt) command. The error occurs during the evaluation phase when calling torch.cat() in the compute() method of metrics.py. It seems that the variable self.TP is not of the expected type (a list or tuple of tensors) but is instead a single tensor or an incompatible object. This issue prevents the training process from completing.

Example(s)

Here is the exact command I used to run the hyperparameter optimization:

chemprop hpopt \
  -i test1126_filter1.csv \
  --task-type classification \
  --output-dir cpu_hpopt_test \
  --hpopt-save-dir cpu_test_hp_results \
  --log cpu_hpopt_test.log \
  --split RANDOM \
  --metrics roc prc accuracy f1 binary-mcc \
  --search-parameter-keywords basic \
  --batch-size 32

The error occurs during the evaluation phase of the training process when computing the metrics.

Expected behavior

The expected behavior is that the training and evaluation processes complete successfully, with the specified metrics (roc, prc, accuracy, f1, binary-mcc) being computed correctly. The torch.cat() function should concatenate a list or tuple of tensors, but it seems self.TP is not being updated or initialized correctly.

Error Stack Trace

Here is the full stack trace of the error:

TypeError: cat() received an invalid combination of arguments - got (Tensor, dim=int), but expected one of:
 * (tuple of Tensors tensors, int dim = 0, *, Tensor out = None)
 * (tuple of Tensors tensors, name dim, *, Tensor out = None)

File "/gpfs/hpc/home/caodongsh/tianyao/miniconda3/envs/chemprop/lib/python3.11/site-packages/chemprop/nn/metrics.py", line 303, in compute
    TP = torch.cat(self.TP, dim=0).sum(0, keepdim=True)
         ^^^^^^^^^^^^^^^^^^^^^^^^^

This error is raised during the evaluation phase when computing metrics. From the error message, it seems that self.TP is either a single tensor or has not been properly initialized as a list of tensors.

The error occurs in the following context:

TP = torch.cat(self.TP, dim=0).sum(0, keepdim=True)

Environment

Python version: 3.11
Chemprop version:2.1
PyTorch version: 2.5.1
Lightning version: 2.4.0
Ray version: 2.40.0
OS: HPC cluster (Linux-based)
Hardware: CPU but when training in GPU it also comes this bug.

Checklist

Package Version

aimsim_core 2.2.2
aiohappyeyeballs 2.4.4
aiohttp 3.11.9
aiosignal 1.3.1
alembic 1.14.0
astartes 1.3.0
attrs 24.2.0
certifi 2024.8.30
charset-normalizer 3.4.0
chemprop 2.1.0
click 8.1.7
cloudpickle 3.1.0
colorlog 6.9.0
ConfigArgParse 1.7
descriptastorus 2.8.0
dill 0.3.9
filelock 3.16.1
frozenlist 1.5.0
fsspec 2024.10.0
future 1.0.0
greenlet 3.1.1
hyperopt 0.2.7
idna 3.10
Jinja2 3.1.4
joblib 1.4.2
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
lightning 2.4.0
lightning-utilities 0.11.9
Mako 1.3.7
markdown-it-py 3.0.0
MarkupSafe 3.0.2
mdurl 0.1.2
mhfp 1.9.6
mordredcommunity 2.0.6
mpmath 1.3.0
msgpack 1.1.0
multidict 6.1.0
multiprocess 0.70.17
networkx 3.4.2
numpy 1.26.4
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
optuna 4.1.0
packaging 24.2
padelpy 0.1.16
pandas 2.2.3
pandas-flavor 0.6.0
pillow 11.0.0
pip 24.2
propcache 0.2.1
protobuf 5.29.1
psutil 6.1.0
py4j 0.10.9.7
pyarrow 18.1.0
Pygments 2.18.0
python-dateutil 2.9.0.post0
pytorch-lightning 2.4.0
pytz 2024.2
PyYAML 6.0.2
ray 2.40.0
rdkit 2024.3.2
referencing 0.35.1
requests 2.32.3
rich 13.9.4
rpds-py 0.22.3
scikit-learn 1.5.2
scipy 1.14.1
setuptools 75.1.0
six 1.17.0
SQLAlchemy 2.0.36
sympy 1.13.1
tabulate 0.9.0
tensorboardX 2.6.2.2
threadpoolctl 3.5.0
torch 2.5.1
torchmetrics 1.6.0
tqdm 4.67.1
triton 3.1.0
typing_extensions 4.12.2
tzdata 2024.2
urllib3 2.2.3
wheel 0.44.0
xarray 2024.11.0
yarl 1.18.3

packages in environment at /gpfs/hpc/home/caodongsh/tianyao/miniconda3/envs/chemprop:

Thank you for your time and support. Please let me know if additional information is needed to debug this issue.

The text was updated successfully, but these errors were encountered:

KnathanM · 2024-12-20T23:29:26Z

@TianYaoCSU thank you very much for raising this and sorry it took me so long to get a solution. If you have time, I would appreciate if you could checkout my branch (in the PR above) that includes a fix to test if it resolves the bug for you.

TianYaoCSU added the bug Something isn't working label Dec 7, 2024

KnathanM self-assigned this Dec 18, 2024

KnathanM linked a pull request Dec 20, 2024 that will close this issue

Fix MCC for DDP and multitask #1131

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v2 BUG]: "TypeError in Chemprop hpopt: torch.cat() got invalid arguments during metric computation" #1121

[v2 BUG]: "TypeError in Chemprop hpopt: torch.cat() got invalid arguments during metric computation" #1121

TianYaoCSU commented Dec 7, 2024

KnathanM commented Dec 20, 2024

[v2 BUG]: "TypeError in Chemprop hpopt: torch.cat() got invalid arguments during metric computation" #1121

[v2 BUG]: "TypeError in Chemprop hpopt: torch.cat() got invalid arguments during metric computation" #1121

Comments

TianYaoCSU commented Dec 7, 2024

packages in environment at /gpfs/hpc/home/caodongsh/tianyao/miniconda3/envs/chemprop:

Name Version Build Channel

KnathanM commented Dec 20, 2024