Module re_renderer::file_resolver

source ·
Expand description

This module implements one half of our cross-platform #import system.

The other half is provided as an extension to the build system, see the build.rs file at the root of this crate.

While it is agnostic to the type of files being imported, in practice this is only used for shaders, thus this is what this documentation will linger on. In particular, integration with our hot-reloading capabilities can get tricky depending on the platform/target.

§Usage

#import <x/y/z/my_file.wgsl>

§Syntax

Import clauses follow the general form of #import <x/y/z/my_file.wgsl>. The path to be imported can be either absolute or relative to the path of the importer, or relative to any of the paths set in the search path (RERUN_SHADER_PATH).

The actual parsing rules themselves are very barebones:

  • An import clause can only span one line.
  • An import clause line must start with #import (exl. whitespaces).
  • Everything between the first < and the last > is interpreted as the import path, as-is. We do so because, between the 4 major platforms (Linux, macOS, Window, Web), basically any string is a valid path.

Everything is trim()ed at every step, you do not need to worry about whitespaces.

§Resolution

Resolution is done in three steps:

  1. First, we try to interpret the imported path as absolute. 1.1. If this is possible and leads to an existing file, we’re done. 1.2. Otherwise, we go to 2.

  2. Second, we try to interpret the imported path as relative to the importer’s. 2.1. If this leads to an existing file, we’re done. 2.2. Otherwise, we go to 3.

  3. Finally, we try to interpret the imported path as relative to all the directories present in the search path, in their prefined priority order, similar to e.g. how the standard $PATH environment variable behaves. 3.1. If this leads to an existing file, we’re done. 3.2. Otherwise, resolution failed: throw an error.

§Interpolation

Interpolation is done in the simplest way possible: the entire line containing the import clause is overwritten with the contents of the imported file. This is of course a recursive process.

§A word about #pragma semantics

Imports can behave in two different ways: #pragma once and #pragma many.

#pragma once means that each unique #import clause is only be resolved once even if it used several times, e.g. assuming that a.txt contains the string "xyz" then:

#import <a.txt>
#import <a.txt>

becomes

xyz

#pragma many on the other hand will resolve the clause as many times as it is used:

#import <a.txt>
#import <a.txt>

becomes

xyz
xyz

At the moment, our import system only provides support for #pragma once semantics.

§Hot-reloading: platform specifics

This import system transparently integrates with the renderer’s hot-reloading capabilities. What that actually means in practice depends on the platform/target.

A general over-simplification of what we’re aiming for can be expressed as:

Be lazy in debug, be eager in release.

When targeting native debug builds, we want everything to be as lazy as possible, everything to happen just-in-time, e.g.:

  • We always talk directly with the filesystem and check for missing files at the last moment.
  • We do resolution & interpolation just-in-time, e.g. just before calling create_shader_module.
  • Etc.

On the web, we don’t even have an actual filesystem to access at runtime, so not only we’d like to be as eager can be, we don’t have much of a choice to begin with. That said, we don’t want to be too eager either: while we do have to make sure that every single shader that we’re gonna use (whether directly or indirectly via an import) ends up in the final artifact one way or another, we still want to delay interpolation as much as we can, otherwise we’d be bloating the binary artifact with N copies of the exact same shader code.

Still, we’d like to limit the number of differences between targets/platforms. And indeed, the current implementation uses a virtual filesystem approach to effectively remove any difference between how the different platforms behave at run-time.

§Debug builds (excl. web)

Native debug builds are straightforward:

  • We handle resolution & interpolation just-in-time (i.e. when fetching file contents).
  • We always talk directly to the filesystem.

No surprises there.

§Release builds (incl. web)

Things are very different for release artifacts, as 1) we disable hot-reloading there and 2) we never interact with the OS filesystem at run-time. Still, in practice, we handle release builds just the same as debug ones.

What happens there is we have a virtual, hermetic, in-memory filesystem that gets pre-loaded with all the shaders defined within the Cargo workspace. This happens in part through a build script that you can find at the root of this crate.

From there, everything behaves exactly the same as usual. In fact, there is only one code path for all platforms at run-time.

There are many issues to deal with along the way though: paths comparisons across environments and build-time/run-time, hermeticism, etc… We won’t cover those here: please refer to the code if you’re curious.

§For developers

§Canonicalization vs. Normalization

Comparing paths can get tricky, especially when juggling target environments and run-time vs. compile-time constraints. For this reason you’ll see plenty mentions of canonicalization and normalization all over the code: better make sure there’s no confusion here.

Canonicalization (i.e. std::fs::canonicalize) relies on syscalls to both normalize a path (including following symlinks!) and make sure the file it references actually exist.

It’s the strictest form of path normalization you can get (and therefore ideal), but requires 1) to have access to an actual filesystem at run-time and 2) that the file being referenced already exists.

Normalization (not available in std) on the other hand is purely lexicographical: it normalizes paths as best as it can without ever touching the filesystem.

See also “Getting Dot-Dot Right”.

§Hermeticism

When shipping release artifacts (whether web or otherwise), we want to avoid leaking state from the original build environments into the final binary (think: paths, timestamps, etc). We need to the build to be hermetic.

Rust’s file!() macro already takes care of that to some extent, and we need to match that behavior on our side (e.g. by not leaking local paths), otherwise we won’t be able to compare paths at runtime.

Think of it as chrooting into our Cargo workspace :)

In our case, there’s an extra invariant on top on that: we must never embed shaders from outside the workspace into our release artifacts!

§Things we don’t support

  • Async: everything in this module is done using standard synchronous APIs.
  • Compression, minification, etc: everything we embed is embedded as-is.
  • Importing via network requests: only the (virtual) filesystem is supported for now.
  • Implicit file suffixes: e.g. #import <myshader> for myshader.wglsl.
  • Embedding raw Naga modules: not yet, though we have everything in place for it.

Structs§

  • FileResolver
    The FileResolver handles both resolving import clauses and doing the actual string interpolation.
  • ImportClause
    A pre-parsed import clause, as in #import <something>.
  • InterpolatedFile
  • SearchPath
    Specifies where to look for imports when both absolute and relative resolution fail.

Functions§

  • new_recommended
    Returns the recommended FileResolver for the current platform/target.

Type Aliases§