Tensorboard logger check if lightning_logs directory exists (#1377)

* tensorboard logger version if root_dir not exist * update changelog * resolve comments Co-authored-by: Alexander Reshytko <areshytko@Alexanders-MacBook-Pro.local> Co-authored-by: William Falcon <waf2107@columbia.edu>
Lightning-AI · Apr 7, 2020 · 495ffbd · 495ffbd
1 parent b8ff9bc
commit 495ffbd
Show file tree

Hide file tree

Showing 2 changed files with 7 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -83,6 +83,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed running `on_validation_end` only on main process in DDP ([#1125](https://github.com/PyTorchLightning/pytorch-lightning/pull/1125))
 - Fixes `use_amp` issue ([#1145](https://github.com/PyTorchLightning/pytorch-lightning/pull/1145))
 - Fixes using deprecated `use_amp` attribute ([#1145](https://github.com/PyTorchLightning/pytorch-lightning/pull/1145))
+- Fixed Tensorboard logger error: lightning_logs directory not exists in multi-node DDP on nodes with rank != 0 ([#1375](https://github.com/PyTorchLightning/pytorch-lightning/issues/1375)).
 - Fixed `Unimplemented backend XLA` error on TPU ([#1387](https://github.com/PyTorchLightning/pytorch-lightning/pull/1387))
 
 ## [0.7.1] - 2020-03-07

diff --git a/pytorch_lightning/loggers/tensorboard.py b/pytorch_lightning/loggers/tensorboard.py
@@ -9,6 +9,7 @@
 from torch.utils.tensorboard import SummaryWriter
 
 from pytorch_lightning.loggers.base import LightningLoggerBase, rank_zero_only
+from pytorch_lightning import _logger as log
 
 
 class TensorBoardLogger(LightningLoggerBase):
@@ -163,6 +164,11 @@ def version(self) -> int:
 
     def _get_next_version(self):
         root_dir = os.path.join(self.save_dir, self.name)
+
+        if not os.path.isdir(root_dir):
+            log.warning('Missing logger folder: %s', root_dir)
+            return 0
+
         existing_versions = []
         for d in os.listdir(root_dir):
             if os.path.isdir(os.path.join(root_dir, d)) and d.startswith("version_"):