Disable cache in downloading dataset (#20)

The cluster seems to have a corrupted copy of ag_news, so to ensure that no problem happens I'll disable the cache (it would be better to fix the problem, but it's a solo project)
Markel · May 2, 2024 · bd08601 · bd08601
1 parent d18abc4
commit bd08601
Showing 1 changed file with 2 additions and 1 deletion.
diff --git a/src/dataset/base_dataset.py b/src/dataset/base_dataset.py
@@ -40,7 +40,8 @@ def __init__(self, dataset_name: str) -> None:
         local_dir = "./downloads/datasets/"+dataset_name
         if not os.path.isdir(local_dir):
             logger.info("Downloading the dataset from the internet.")
-            snapshot_download(dataset_name, local_dir=local_dir, repo_type="dataset")
+            snapshot_download(dataset_name, local_dir=local_dir,
+                              repo_type="dataset", local_dir_use_symlinks=False)
             logger.debug("Dataset downloaded successfully.")
 
     def __str__(self):