Dataloader

rerun.experimental.dataloader

PyTorch Datasets for training on data from the Rerun catalog.

BlockShuffle dataclass

Bases: ShuffleStrategy

Shuffle fetch-sized blocks of consecutive samples instead of individual samples.

Each block is one fetch's worth of consecutive, segment-local samples. The block order is shuffled, but samples keep their natural order within a block, so every fetch still reads one contiguous span: stored data is read about once per epoch instead of once per fetch, and decoders reuse their cache across a block. Set buffer_size to decorrelate batches at emission time.

PARAMETER	DESCRIPTION
buffer_size	Size of the post-decode buffer that randomizes emission order without changing the fetch order. None (the default) emits in fetch order. This is the only strategy that takes one — see emission_buffer. Warning The buffer holds up to buffer_size decoded samples per DataLoader worker, as your decoders produce them — full-resolution frames for video fields. Budget buffer_size * bytes_per_sample * num_workers; a few thousand video samples per worker is tens of gigabytes. TYPE: int | None DEFAULT: None
min_fill	Samples buffered before emission starts. Defaults to buffer_size // 2, which is also how long the first sample is delayed; lower it to shorten that warm-up at the cost of less mixing over the first few batches. TYPE: int | None DEFAULT: None

ColumnDecoder

Bases: ABC

Base class for column decoders.

Subclasses convert raw Arrow data into tensors. Stateless decoders (images, scalars) only need to implement decode. Context-aware decoders (compressed video) should also override context_range so the prefetcher fetches surrounding data.

fill_latest_at property

fill_latest_at: bool

Whether this column's prefetch read latest-at-fills empty grid slots.

True for stateless columns (images, scalars): each grid slot wants the most recent value snapped from the real rows. Compressed video keeps it True too (consecutive duplicates from a dense grid are dropped at decode time), but a decoder reading frame-indexed data where the grid lands 1:1 on real samples can override to False for exact, fill-free packet reads. The read is partitioned by this flag so it stays a global query argument per group rather than a per-column one.

context_range

def context_range(
    index_value: int | datetime64 | timedelta64,
) -> (
    tuple[
        int | datetime64 | timedelta64,
        int | datetime64 | timedelta64,
    ]
    | None
)

Extra index-value range needed to decode index_value.

Returns (start, end) inclusive, or None when only the exact index value is required (the default).

decode abstractmethod

def decode(
    raw: ChunkedArray,
    index_value: int | datetime64 | timedelta64,
    segment_id: str,
) -> Tensor | None

Decode raw Arrow data into a tensor, or return None to signal data missing.

prior_keyframe_path

def prior_keyframe_path(field_path: str) -> str | None

Sibling column whose non-null rows mark a re-entrant keyframe, or None.

Override on decoders that need the prefetch window anchored at the prior keyframe (compressed video). Default returns None.

DataSource dataclass

An immutable reference to a dataset with an optional segment filter.

PARAMETER	DESCRIPTION
dataset	The remote dataset to read from. TYPE: DatasetEntry
segments	Optional list of segment IDs to restrict to. TYPE: list[str] | None DEFAULT: None

filter_segments

def filter_segments(segment_ids: list[str]) -> DataSource

Return a new DataSource narrowed to segment_ids.

Field dataclass

Declarative spec for one field of a training sample.

Note

This API is provisional and will be improved, expect the surface to change.

PARAMETER	DESCRIPTION
path	entity_path:Archetype:component triple identifying the source column (e.g. "/camera:EncodedImage:blob"). TYPE: str
decode	A ColumnDecoder that turns the Arrow column into a tensor. TYPE: ColumnDecoder
select	Optional jq-like Selector applied client-side to the Arrow column before decode. Used for nested struct/list access. The server-side projection is unaffected. Field( path="/agent:ListOfStructs:animals", select=Selector(".[0].dog"), decode=NumericDecoder(), ) TYPE: Selector | None DEFAULT: None
window	Optional (start_offset, end_offset) range, inclusive on both ends and added to the current index value. The field then yields the slice of values across that window instead of a single sample. Offsets are in the index timeline's native unit: integer steps for integer-indexed timelines, nanoseconds for timestamp timelines (use multiples of the FixedRateSampling period to align with the sampling grid). For example, (1, 50) on an integer timeline fetches the next 50 values after the current sample. TYPE: tuple[int, int] | None DEFAULT: None
max_staleness	Optional maximum age, in the index timeline's native unit (same as window), of the data backing a sample. When set, a required sample is dropped if the nearest value at or before a queried point is older than this. None (the default) applies no staleness limit. Enforced only during manifest construction, not by the streaming dataloader. TYPE: int | None DEFAULT: None

to_recipe

def to_recipe() -> dict[str, Any]

A JSON-serializable snapshot of this field's spec, for a manifest's provenance header.

Kept on Field so it stays in sync as the spec evolves. decode / select are captured via repr — a human-readable record, not a round-trippable form.

FixedRateSampling dataclass

Sample timestamp or duration timelines at a fixed nominal rate.

Indices are drawn on an algebraic grid seg.index_start + k * ns_per_sample. The server's fill_latest_at absorbs any drift from real-row positions.

ImageDecoder

Bases: ColumnDecoder

Decode a single encoded-image blob (JPEG/PNG) to a [C, H, W] uint8 tensor.

fill_latest_at property

fill_latest_at: bool

Whether this column's prefetch read latest-at-fills empty grid slots.

True for stateless columns (images, scalars): each grid slot wants the most recent value snapped from the real rows. Compressed video keeps it True too (consecutive duplicates from a dense grid are dropped at decode time), but a decoder reading frame-indexed data where the grid lands 1:1 on real samples can override to False for exact, fill-free packet reads. The read is partitioned by this flag so it stays a global query argument per group rather than a per-column one.

context_range

def context_range(
    index_value: int | datetime64 | timedelta64,
) -> (
    tuple[
        int | datetime64 | timedelta64,
        int | datetime64 | timedelta64,
    ]
    | None
)

Extra index-value range needed to decode index_value.

Returns (start, end) inclusive, or None when only the exact index value is required (the default).

prior_keyframe_path

def prior_keyframe_path(field_path: str) -> str | None

Sibling column whose non-null rows mark a re-entrant keyframe, or None.

Override on decoders that need the prefetch window anchored at the prior keyframe (compressed video). Default returns None.

Manifest

A description of the exact sampling order of data for one epoch of a training run. This serves the purpose of making training runs reproducible and resumable.

A manifest is created by generating one from a source with Manifest.generate, or by loading an existing one with Manifest.from_parquet. Persist it with Manifest.write_parquet; a manifest is always backed by a parquet file on the read path so a DataLoader worker never holds the whole table in RAM.

Note

This API is provisional and will be improved, expect the surface to change.

metadata property

metadata: ManifestMeta

Decoded metadata header.

num_rows property

num_rows: int

Total number of samples in the manifest.

__getstate__

def __getstate__() -> dict[str, Any]

Ship only the path (not the loaded rows) to DataLoader workers when parquet-backed.

from_parquet classmethod

def from_parquet(path: str | PathLike[str]) -> Manifest

Load a manifest from a parquet file.

Rows are read lazily, per (rank, worker) shard, so a DataLoader worker never loads the whole manifest into RAM.

generate classmethod

def generate(
    source: DataSource,
    index: str,
    fields: dict[str, Field],
    *,
    timeline_sampling: FixedRateSampling | None = None,
    fetch_size: int = 128,
    num_ranks: int = 1,
    num_workers_per_rank: int = 1,
    required_fields: list[str] | None = None,
    scan_max_workers: int | None = None,
) -> Manifest

Generate an unshuffled manifest for one epoch by scanning the source.

Scans the source, drops invalid samples, and unrolls one epoch in natural order (segment by segment, along the timeline) into fetch_group / emit_rank columns. To shuffle, call shuffle on the result — it re-orders cheaply without re-scanning. Persist with write_parquet.

See RerunIterableDataset for source / index / fields / timeline_sampling. fetch_size is the co-fetch / co-decode block size and num_ranks / num_workers_per_rank freeze the (rank, worker) assignment. required_fields are the field keys that must resolve to real data for a sample to be kept. scan_max_workers caps the concurrent scan queries against the server (defaults to 8); raise it to speed up scanning a large dataset.

shuffle

def shuffle(
    strategy: ShuffleStrategy | None = None,
    *,
    seed: int = 0,
) -> Manifest

Shuffle a scanned manifest into a new epoch order, without re-scanning the source.

Keeps the manifest's validated sample set and per-field decode ranges (the expensive scan result) and only recomputes the fetch_group / emit_rank schedule under a new strategy and seed. fetch_size and the (num_ranks, num_workers_per_rank) topology are inherited from this manifest. Returns a new manifest; this one is unchanged.

Scan once with generate, then call this per epoch (bumping seed) to get fresh orders for free.

PARAMETER	DESCRIPTION
strategy	The ShuffleStrategy to apply, e.g. BlockShuffle (the default), SampleShuffle, or NoShuffle. It fixes both the fetch order and the emission buffer, so passing the same strategy object here and to RerunIterableDataset guarantees a replay matches the live run. TYPE: ShuffleStrategy | None DEFAULT: None
seed	Seed for the block shuffle and the emission buffer. TYPE: int DEFAULT: 0

to_arrow

def to_arrow() -> Table

The manifest as an Arrow table (loading it from the backing parquet file if needed).

validate_topology

def validate_topology(
    num_ranks: int, num_workers_per_rank: int
) -> None

Raise if the run's topology differs from the one the manifest was frozen for.

worker_assignments

def worker_assignments(rank: int, worker: int) -> Table

Rows assigned to one (rank, worker), in fetch order.

For a parquet-backed manifest this reads only this shard from the file (predicate pushdown), not the whole manifest.

worker_plan

def worker_plan(
    rank: int, worker: int
) -> tuple[list[Table], ndarray]

Return this (rank, worker)'s fetch_groups (co-fetch / co-decode units, in fetch order) and emission pull order.

Reading a shard can hit disk, so the fetch groups and the emission order are both derived from a single read.

write_parquet

def write_parquet(
    path: str | PathLike[str],
    *,
    row_group_size: int = 1 << 16,
) -> None

Write the manifest to a zstd-compressed parquet file; the header rides in the schema metadata.

NoShuffle dataclass

Bases: ShuffleStrategy

Natural order (segment by segment, along the timeline): maximal fetch locality, no randomness.

emission_buffer

def emission_buffer() -> ShuffleBuffer | None

The post-decode buffer samples are emitted through, or None to emit in fetch order.

Only BlockShuffle defines one, because it is the only strategy whose fetch order stays deliberately correlated: emission is where its batches get decorrelated. SampleShuffle already fetches a uniform permutation, so a buffer would add nothing, and NoShuffle is a deterministic baseline that a buffer would only contaminate.

Owning the buffer here is what keeps a live run and a manifest replay in step: both read it off the same strategy object, so they cannot be configured with different buffers by accident.

NumericDecoder

Bases: ColumnDecoder

Decode Arrow numeric / list-of-numeric columns to a tensor.

fill_latest_at property

fill_latest_at: bool

Whether this column's prefetch read latest-at-fills empty grid slots.

True for stateless columns (images, scalars): each grid slot wants the most recent value snapped from the real rows. Compressed video keeps it True too (consecutive duplicates from a dense grid are dropped at decode time), but a decoder reading frame-indexed data where the grid lands 1:1 on real samples can override to False for exact, fill-free packet reads. The read is partitioned by this flag so it stays a global query argument per group rather than a per-column one.

context_range

def context_range(
    index_value: int | datetime64 | timedelta64,
) -> (
    tuple[
        int | datetime64 | timedelta64,
        int | datetime64 | timedelta64,
    ]
    | None
)

Extra index-value range needed to decode index_value.

Returns (start, end) inclusive, or None when only the exact index value is required (the default).

prior_keyframe_path

def prior_keyframe_path(field_path: str) -> str | None

Sibling column whose non-null rows mark a re-entrant keyframe, or None.

Override on decoders that need the prefetch window anchored at the prior keyframe (compressed video). Default returns None.

RerunIterableDataset

Bases: IterableDataset[dict[str, Tensor | None]]

Iterable dataset backed by a catalog server.

Fetches fetch_size samples per server query and yields individual samples, so per-query overhead is amortized across many samples while the DataLoader controls the training batch size independently.

The index list is partitioned across DDP ranks and DataLoader workers internally. With shuffling enabled (default), the sample order is permuted once per epoch before partitioning; call set_epoch to re-seed between epochs.

PARAMETER	DESCRIPTION
source	The dataset to read from (with optional segment filter). TYPE: DataSource
index	Timeline to iterate (e.g. "frame_nr"). TYPE: str
fields	Sample fields, keyed by output name. TYPE: dict[str, Field]
timeline_sampling	Required when index is a timestamp timeline; ignored for integer indices. Pass FixedRateSampling to sample on a fixed grid (e.g. 30 Hz). TYPE: FixedRateSampling | None DEFAULT: None
fetch_size	Number of samples to fetch per server query. Larger values amortize network overhead but use more memory. Defaults to 128. TYPE: int DEFAULT: 128
shuffle_strategy	The ShuffleStrategy that determines the order samples are fetched in, and — for BlockShuffle — the optional post-decode buffer samples are emitted through. Defaults to SampleShuffle; pass NoShuffle for natural order. TYPE: ShuffleStrategy | None DEFAULT: None
decode_threads	Fields to decode concurrently within each DataLoader worker. TYPE: int | None DEFAULT: None

sample_index property

sample_index: SampleIndex

The underlying SampleIndex.

__iter__

def __iter__() -> Iterator[dict[str, Tensor | None]]

Yield this worker's samples: replayed from a manifest, or fetched live from the catalog.

__len__

def __len__() -> int

Total number of samples across all segments.

from_manifest classmethod

def from_manifest(
    manifest: Manifest,
    source: DataSource,
    fields: dict[str, Field],
    *,
    decode_threads: int | None = None,
) -> RerunIterableDataset

Build a dataset that replays a frozen Manifest's sampling order.

The order, shards, and decode ranges all come from manifest; a manifest records only the field specs, not the objects, so the live connection and decoders are supplied here.

PARAMETER	DESCRIPTION
manifest	The frozen manifest to replay. Provides the sampling order, shards, and decode ranges. TYPE: Manifest
source	The live catalog connection. TYPE: DataSource
fields	The decoders, keyed by field name (a manifest records only their spec, not the objects). TYPE: dict[str, Field]
decode_threads	Fields to decode concurrently within each DataLoader worker. TYPE: int | None DEFAULT: None

RETURNS	DESCRIPTION
RerunIterableDataset	A dataset that yields samples in the manifest's recorded order.

set_epoch

def set_epoch(epoch: int) -> None

Set the epoch for shuffling (like DistributedSampler.set_epoch).

RerunMapDataset

Bases: Dataset[dict[str, Tensor | None]]

Map-style dataset backed by a catalog server.

Supports random access by global index, so it works with PyTorch's sampler ecosystem (DistributedSampler, WeightedRandomSampler, SubsetRandomSampler, ...). Shuffling and cross-worker partitioning are driven by the DataLoader's sampler.

For streaming iteration with internal shuffling, use RerunIterableDataset instead.

PARAMETER	DESCRIPTION
source	The dataset to read from (with optional segment filter). TYPE: DataSource
index	Timeline column to use as the sample index (e.g. "frame_nr"). TYPE: str
fields	Sample fields, keyed by output name. TYPE: dict[str, Field]
timeline_sampling	Required when index is a timestamp timeline; ignored for integer indices. Pass FixedRateSampling to sample on a fixed grid (e.g. 30 Hz). TYPE: FixedRateSampling | None DEFAULT: None
decode_threads	Fields to decode concurrently within each DataLoader worker. TYPE: int | None DEFAULT: None

Examples:

dataset = RerunMapDataset(
    source,
    "frame_nr",
    {"image": Field("/camera:Image:blob", decode=ImageDecoder())},
)
sampler = DistributedSampler(dataset)
loader = DataLoader(dataset, batch_size=8, sampler=sampler, num_workers=4)
for batch in loader:
    ...

sample_index property

sample_index: SampleIndex

The underlying SampleIndex.

__getitem__

def __getitem__(idx: int) -> dict[str, Tensor | None]

Fetch a single sample by global index (one server query).

__getitems__

def __getitems__(
    indices: list[int],
) -> list[dict[str, Tensor | None]]

Fetch multiple samples by global index in a single server query.

PyTorch's DataLoader calls this automatically when present, so each batch round-trips once.

__len__

def __len__() -> int

Total number of samples across all segments.

SampleIndex

Pre-computed description of the complete sample space.

Maps every segment's positional indices to concrete index values, accounting for the timeline strategy (integer or fixed-rate grid). Small enough to hold in memory for any realistic dataset.

PARAMETER	DESCRIPTION
segments	Per-segment metadata (window-adjusted index range + sample count). TYPE: list[SegmentMetadata]
ns_per_sample	For FixedRateSampling: nanoseconds between grid points. None for integer indices. TYPE: int | None DEFAULT: None
ns_dtype	Numpy dtype string used when materializing index values: "datetime64[ns]" for timestamp timelines, "timedelta64[ns]" for duration timelines, or None for plain integer indices. TYPE: str | None DEFAULT: None

is_duration property

is_duration: bool

Whether the index is a duration timeline.

is_timestamp property

is_timestamp: bool

Whether the index is a timestamp timeline.

ns_dtype property

ns_dtype: str | None

Numpy dtype for materialized index values, or None for integer indices.

ns_per_sample property

ns_per_sample: int | None

Nanoseconds between grid points for fixed-rate sampling, or None.

segment_offsets property

segment_offsets: ndarray

Cumulative offsets: segments[i] covers global indices [segment_offsets[i], segment_offsets[i + 1]).

segments property

segments: list[SegmentMetadata]

Per-segment metadata list.

total_samples property

total_samples: int

Total number of samples across all segments.

build staticmethod

def build(
    source: DataSource,
    index: str,
    fields: dict[str, Field],
    *,
    timeline_sampling: FixedRateSampling | None = None,
) -> SampleIndex

Build a SampleIndex from lightweight metadata queries.

PARAMETER	DESCRIPTION
source	Data source to build from. TYPE: DataSource
index	Name of the index timeline column. TYPE: str
fields	Field definitions, used for window-trim calculation. TYPE: dict[str, Field]
timeline_sampling	Required for timestamp and duration indices; ignored for integer indices. Pass FixedRateSampling for a regular grid. TYPE: FixedRateSampling | None DEFAULT: None

global_to_local

def global_to_local(
    idx: int,
) -> tuple[SegmentMetadata, int | datetime64 | timedelta64]

Map a global index [0, total_samples) to (segment, concrete_idx_value).

The returned index value is a plain int for integer timelines, a datetime64[ns] for timestamp timelines, and a timedelta64[ns] for duration timelines.

indices_in_range

def indices_in_range(lo: int, hi: int) -> Iterable[int]

Enumerate valid index values in [lo, hi].

For fixed-rate timelines the returned values walk down from hi in ns_per_sample steps (so they remain on the grid as long as hi is). For integer timelines, every value in [lo, hi] is returned. Values are plain int (ns-since-epoch for timestamp indices, ns count for duration indices); the caller casts the aggregated set to the right numpy dtype.

resolve_local_index

def resolve_local_index(
    seg: SegmentMetadata, pos: int
) -> int | datetime64 | timedelta64

Convert a positional index within seg to a concrete index value.

pos is in [0, seg.num_samples). Returns datetime64[ns] for timestamp timelines, timedelta64[ns] for duration timelines, and a plain int for integer indices.

SampleShuffle dataclass

Bases: ShuffleStrategy

Uniform per-sample shuffle: maximal decorrelation, minimal fetch locality.

Every fetch scatters across all segments; prefer BlockShuffle when fetch throughput is the bottleneck.

emission_buffer

def emission_buffer() -> ShuffleBuffer | None

The post-decode buffer samples are emitted through, or None to emit in fetch order.

Only BlockShuffle defines one, because it is the only strategy whose fetch order stays deliberately correlated: emission is where its batches get decorrelated. SampleShuffle already fetches a uniform permutation, so a buffer would add nothing, and NoShuffle is a deterministic baseline that a buffer would only contaminate.

Owning the buffer here is what keeps a live run and a manifest replay in step: both read it off the same strategy object, so they cannot be configured with different buffers by accident.

SegmentMetadata dataclass

Per-segment metadata for sampling.

ShuffleStrategy

Bases: ABC

Determines the order in which an epoch's samples are fetched.

See the training guide for the trade-offs.

emission_buffer

def emission_buffer() -> ShuffleBuffer | None

The post-decode buffer samples are emitted through, or None to emit in fetch order.

Only BlockShuffle defines one, because it is the only strategy whose fetch order stays deliberately correlated: emission is where its batches get decorrelated. SampleShuffle already fetches a uniform permutation, so a buffer would add nothing, and NoShuffle is a deterministic baseline that a buffer would only contaminate.

Owning the buffer here is what keeps a live run and a manifest replay in step: both read it off the same strategy object, so they cannot be configured with different buffers by accident.

epoch_order abstractmethod

def epoch_order(
    sample_index: SampleIndex, *, fetch_size: int, seed: int
) -> tuple[ndarray, ndarray]

Return (indices, block_bounds) for one epoch: every global sample index once, in emission order.

block_bounds are cumulative end positions of blocks within indices; each block must be a contiguous, segment-local span of the global index space.

VideoFrameDecoder

Bases: ColumnDecoder

Compressed video random access via context-aware fetching.

Anchors the decode window at the prior keyframe by consulting the sibling is_keyframe component on the VideoStream archetype, derived from Field.path (e.g. /cam:VideoStream:sample pairs with /cam:VideoStream:is_keyframe). The marker is populated by the user or by LazyChunkStream.collect(optimize=…), and lives in dedicated chunks separate from the video sample, so the lookup is cheap.

When the column is missing from the schema, or has no row at or before the target, the decoder falls back to a fixed-size window: the previous keyframe_interval samples (counted directly for integer indices, converted to keyframe_interval / fps_estimate seconds for timestamp indices). keyframe_interval must be at least the actual GOP length, and for timestamp indices fps_estimate must be close to the true frame rate.

Samples may be raw H.264 AVC1/AVCC (length-prefixed NAL units) or Annex B; the format is detected automatically per sample.

A call whose window extends the previous call's (same GOP) reuses an open codec context and decodes only the new packets.

Returns None when the resolved window contains no decodable keyframe: the target precedes the entity's first frame in a multi-modal segment, the fallback keyframe_interval under-estimates the true GOP length, or the anchored row was user-logged is_keyframe=true on a sample that isn't actually a codec keyframe (run optimize with fix_keyframe=True to re-derive markers from the encoded samples). Consumers must filter these out in their collate function before stacking.

fill_latest_at property

fill_latest_at: bool

Whether this column's prefetch read latest-at-fills empty grid slots.

True for stateless columns (images, scalars): each grid slot wants the most recent value snapped from the real rows. Compressed video keeps it True too (consecutive duplicates from a dense grid are dropped at decode time), but a decoder reading frame-indexed data where the grid lands 1:1 on real samples can override to False for exact, fill-free packet reads. The read is partitioned by this flag so it stays a global query argument per group rather than a per-column one.

__getstate__

def __getstate__() -> dict[str, Any]

Drop the sessions: open codec contexts cannot be pickled.

__init__

def __init__(
    *,
    keyframe_interval: int = 30,
    fps_estimate: float = 30.0,
    codec: str = "h264",
    max_decoder_sessions: int = 8,
    thread_count: int = 1,
) -> None

Construct a decoder for a compressed video column.

PARAMETER	DESCRIPTION
keyframe_interval	Fallback GOP length (in frames) used to estimate how far back the prior keyframe sits when the stream has no explicit markers. TYPE: int DEFAULT: 30
fps_estimate	Fallback frame rate used to turn keyframe_interval into a time window. TYPE: float DEFAULT: 30.0
codec	Video codec of the encoded samples (e.g. "h264"). TYPE: str DEFAULT: 'h264'
max_decoder_sessions	Upper bound on the number of live codec contexts kept in the LRU cache. TYPE: int DEFAULT: 8
thread_count	ffmpeg decode thread count. Usually 1 for low resolution, larger for large resolutions; 1 is preferred over auto, so we do not propose auto. TYPE: int DEFAULT: 1

context_range

def context_range(
    index_value: int | datetime64 | timedelta64,
) -> (
    tuple[
        int | datetime64 | timedelta64,
        int | datetime64 | timedelta64,
    ]
    | None
)

Need frames from estimated keyframe position to target.

decode

def decode(
    raw: ChunkedArray,
    index_value: int | datetime64 | timedelta64,
    segment_id: str,
) -> Tensor | None

Decode the target frame from the context samples in raw, or None if no keyframe is available.

tracing_scope

def tracing_scope(name: str) -> Iterator[None]

Open an OpenTelemetry span for the duration of a with block and propagate trace context into Rerun's Rust SDK.

Context-manager counterpart to with_tracing — use it to scope arbitrary blocks of code without extracting them into a function. Any Rust-side #[instrument] spans triggered from within will be parented under this span in Jaeger.

No-op unless TELEMETRY_ENABLED=true and an OTLP endpoint is configured (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT).

Examples:

for epoch in range(num_epochs):
    with tracing_scope(f"epoch {epoch}"):
        train_one_epoch(...)

with_tracing

def with_tracing(name: str) -> Callable[[F], F]

Wrap a function in an OpenTelemetry span and propagate trace context into Rerun's Rust SDK.

When enabled, creates a span named name, injects the W3C traceparent header into Rerun's shared ContextVar, and runs the wrapped function. Any Rust-side #[instrument] spans triggered from within (e.g. catalog queries) will be parented under this span in Jaeger.

For ad-hoc blocks that don't belong in a dedicated function, use tracing_scope instead.

No-op unless TELEMETRY_ENABLED=true and an OTLP endpoint is configured (OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT).