Enable more frequent log flush to log.txt

Summary: :Facebook When using manifold to write log.txt , the logs will buffer until 10MB in size and only then the training log is dumped. Or if the training finishes and buffer <10MB, the log will be dumped. The issue with this is that most of our training logs are like 2-5MB and we can't monitor an ongoing training if the logs are not flushed during the training. It used to be the case that logs would flush with 1KB buffer but was changed to 10MB, for our trainings in vissl, 10KB is a good buffer size so for a 2KB training log, we will flush to manifold 200 times through training. Reviewed By: sujitoc Differential Revision: D26341684 fbshipit-source-id: 22ea39a97c9d43515ba4cf6746d99f69816513d5
yangsenwxy · Feb 9, 2021 · 4345df7 · 4345df7
1 parent cf65346
commit 4345df7
Showing 1 changed file with 4 additions and 1 deletion.
diff --git a/vissl/utils/logger.py b/vissl/utils/logger.py
@@ -54,7 +54,10 @@ def setup_logging(name, output_dir=None, rank=0):
 # with the same file name can safely write to the same file.
 @functools.lru_cache(maxsize=None)
 def _cached_log_stream(filename):
-    io = PathManager.open(filename, "a")
+    # we tune the buffering value so that the logs are updated
+    # frequently.
+    log_buffer_kb = 10 * 1024  # 10KB
+    io = PathManager.open(filename, mode="a", buffering=log_buffer_kb)
     atexit.register(io.close)
     return io