Support offline compilation

AlejandroByrne · Dec 8, 2023 · 9e12238 · 9e12238
1 parent d131502
commit 9e12238
Show file tree

Hide file tree

Showing 4 changed files with 67 additions and 8 deletions.
diff --git a/README.md b/README.md
@@ -164,9 +164,8 @@ commands in the root of the tree builds and installs all schedulers under
 ```
 $ cd $SCX
 $ meson setup build --prefix ~
-$ cd build
-$ meson compile
-$ meson install
+$ meson compile -C build
+$ meson install -C build
 ```
 
 Note that `meson compile` step is not strictly necessary as `install`
@@ -199,14 +198,13 @@ want to build the simple example scheduler
 ```
 $ cd $SCX
 $ meson setup build -Dbuildtype=release
-$ cd build
-$ meson compile -v scx_simple scx_rusty
+$ meson compile -C build scx_simple scx_rusty
 ```
 
 You can also specify `-v` if you want to see the commands being used:
 
 ```
-$ meson compile -v scx_pair
+$ meson compile -C build -v scx_pair
 ```
 
 For C userspace schedulers such as the ones under `scheds/kernel-examples`,
@@ -232,6 +230,8 @@ options can be used in such cases.
 - `libbpf_a`: Static `libbpf.a` to use
 - `libbpf_h`: `libbpf` header directories, only meaningful with `libbpf_a` option
 - `cargo`: `cargo` to use when building rust sub-projects
+- 'cargo_home': 'CARGO_HOME env to use when invoking cargo'
+- `offline`: 'Compilation step should not access the internet'
 
 For example, let's say you want to use `bpftool` and `libbpf` shipped in the
 kernel tree located at `$KERNEL`. We need to build `bpftool` in the kernel
@@ -247,8 +247,7 @@ $ meson setup build -Dbuildtype=release -Dprefix=~/bin \
     -Dbpftool=$BPFTOOL/bpftool \
     -Dlibbpf_a=$BPFTOOL/libbpf/libbpf.a \
     -Dlibbpf_h=$BPFTOOL/libbpf/include
-$ cd build
-$ meson install
+$ meson install -C build
 ```
 
 Note that we use `libbpf` which was produced as a part of `bpftool` build
@@ -257,6 +256,33 @@ process rather than buliding `libbpf` directly. This is necessary because
 relative locations.
 
 
+### Offline Compilation
+
+Rust builds automatically download dependencies from crates.io; however,
+some build environments might not allow internet access requiring all
+dependencies to be available offline. The `fetch` target and `offline`
+option are provided for such cases.
+
+The following downloads all Rust dependencies into `$HOME/cargo-deps`.
+
+```
+$ cd $SCX
+$ meson setup build -Dcargo_home=$HOME/cargo-deps
+$ meson compile -C build fetch
+```
+
+The following builds the schedulers without accessing the internet. The
+`build` directory doesn't have to be the same one. The only requirement is
+that the `cargo_home` option points to a directory which contains the
+content generated from the previous step.
+
+```
+$ cd $SCX
+$ meson setup build -Dcargo_home=$HOME/cargo-deps -Doffline=true -Dbuildtype=release
+$ meson compile -C build
+```
+
+
 ### Working with Rust Sub-projects
 
 Each Rust sub-project is its own self-contained cargo project. When buildng

diff --git a/meson-scripts/cargo_fetch b/meson-scripts/cargo_fetch
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+shopt -s globstar
+set -e
+
+cargo="$1"
+
+#
+# Run `cargo fetch` for each Cargo.toml found in the source directory. This
+# can lead to fetching unnecessary packages if the source directory contains
+# past cargo build artifacts. While not ideal, this is a lot simpler and
+# shouldn't cause problems in practice.
+#
+for manifest in ${MESON_SOURCE_ROOT}/**/Cargo.toml; do
+    echo "Fetching for $manifest"
+    "$cargo" fetch --manifest-path="$manifest"
+done
diff --git a/meson.build b/meson.build
@@ -20,6 +20,8 @@ bpftool_build_skel = find_program(join_paths(meson.current_source_dir(),
                                              'meson-scripts/bpftool_build_skel'))
 get_sys_incls = find_program(join_paths(meson.current_source_dir(),
                                         'meson-scripts/get_sys_incls'))
+cargo_fetch = find_program(join_paths(meson.current_source_dir(),
+                                      'meson-scripts/cargo_fetch'))
 
 bpf_clang_ver = run_command(get_clang_ver, bpf_clang, check: true).stdout().strip()
 bpf_clang_maj = bpf_clang_ver.split('.')[0].to_int()
@@ -103,6 +105,10 @@ if get_option('buildtype') == 'release'
   cargo_build_args += '--release'
 endif
 
+if get_option('offline')
+  cargo_build_args += '--offline'
+endif
+
 cargo_env = environment()
 cargo_env.set('BPF_CLANG', bpf_clang.full_path())
 
@@ -130,7 +136,13 @@ if get_option('libbpf_a') != ''
   #                     'dependencies.libbpf-sys.features=["novendor", "static"]']
 endif
 
+if get_option('cargo_home') != ''
+  cargo_env.set('CARGO_HOME', get_option('cargo_home'))
+endif
+
 meson.add_install_script('meson-scripts/install_rust_user_scheds')
 
+run_target('fetch', command: [cargo_fetch, cargo], env: cargo_env)
+
 subdir('rust')
 subdir('scheds')
diff --git a/meson.options b/meson.options
@@ -8,3 +8,7 @@ option('libbpf_h', type: 'array',
        description: 'libbpf header directories, only meaningful with libbpf_a option')
 option('cargo', type: 'string', value: 'cargo',
        description: 'cargo to use when building rust sub-projects')
+option('cargo_home', type: 'string',
+       description: 'CARGO_HOME env to use when invoking cargo')
+option('offline', type: 'boolean', value: 'false',
+       description: 'Compilation step should not access the internet')