Add more data pipeline doc clarifications (minerllabs#565)

adzcai · Aug 9, 2021 · 1250929 · 1250929
1 parent 7842e87
commit 1250929
Show file tree

Hide file tree

Showing 2 changed files with 15 additions and 2 deletions.
diff --git a/minerl/data/pipeline/merge.py b/minerl/data/pipeline/merge.py
@@ -1,8 +1,8 @@
 # Copyright (c) 2020 All Rights Reserved
 # Author: William H. Guss, Brandon Houghton
 """
-Merges similar trajectories (by stream name) into compressed zip archives.
-Uses 7z, which has the best compression ratios on the market, for zipping.
+Merges raw binary packet data from each "player stream" into compressed zip archives
+(MCPR files). Uses 7z, which has the best compression ratios on the market, for zipping.
 
 Requirements:
     * 7z: You can install `7z` on Ubuntu with `sudo apt install p7zip`.

diff --git a/minerl/data/pipeline/render.py b/minerl/data/pipeline/render.py
@@ -8,6 +8,19 @@
 # 2) Running the action_rendering scripts
 # 3) Running the video_rendering scripts
 
+Inputs: MCPR files from merge.py script.
+Outputs:
+    * Unedited MP4 files for entire player stream, including lobby video and
+        several episodes in the same MP4 file (generate.py will slice out individual
+        episodes later).
+    * univ.json and metadata.json
+        * univ.json contains a plethora of information including equip swaps, item pickups,
+            "touched block" events, etc which is later processed into Numpy actions and
+            observations. Again, this is unsliced at this stage.
+        * metadata.json: ReplayMod metadata, doesn't seem important.
+        * markers.json: Contains stop- and start-recording timesteps used to slice
+            player streams into individual episodes.
+
 
 By default, this generates low-res videos of dimensions 64x64.
 To generate videos of a different resolution, set or `export` the