http://rebar3.org/docs/extending/custom_compiler_modules/ has the most documentation

compile_all makes use of that API but isn't something the person writing a plugin has to concern themselves with.

Each compiler we use within rebar3 uses it:

• erl compiler (most complex one)
• MIB compiler (a good example of average complexity)
• xrl compiler (very simple, has no dependency handling)
• yrl compiler (very simple, has no dependency handling)

I also even wrote a compiler for my blog which uses it and can show a different structure, but mostly the post I wrote on it has a description that might be worth migrating to our docsite:

The Rebar3 compiler infrastructure works by breaking up the flow of compilation in a generic and specific subset.

The specific subset will:

1. Define which file types and paths must be considered by the compiler.
2. Define which files are dependencies of other files.
3. Be given a graph of all files and their artifacts with their last modified times (and metadata), and specify which of them need rebuilding.
4. Compile individual files and provide metadata to track the artifacts.

The generic subset will:

1. Scan files and update their timestamps in a graph for the last modifications.
2. Use the dependency information to complete the dependency graph.
3. Propagate the timestamps of source files modifications transitively through the graph (assume you update header A, included by header B, applied by macro C, on file D; then B, C, and D are all marked as modified as recently as A in the DAG).
4. Pass this updated graph to the specific part to get a list of files to build (usually by comparing which source files are newer than their artifacts, but also if build options changed).
5. Schedule sequential or parallel compilation based on what the specific part specified.
6. Update the DAG with the artifacts and build metadata, and persist the data to disk.

In short, you build a compiler plugin that can name directories, file extensions, dependencies, and can compare timestamps and metadata. Then make sure this plugin can compile individual files, and the rest is handled for you.

I think with these high level explanations, the template for callbacks may make more sense, and so would the code for the compiler module.