Chunk

rerun.chunk

Chunk processing API.

Lens module-attribute

Lens: TypeAlias = DeriveLens | MutateLens

Union of all lens types.

Chunk

A single chunk of data from a recording.

entity_path property

entity_path: str

The entity path this chunk belongs to.

id property

id: str

The unique ID of this chunk.

is_empty property

is_empty: bool

Whether the chunk has zero rows.

is_static property

is_static: bool

Whether the chunk contains only static data (no timelines).

num_columns property

num_columns: int

The number of columns in this chunk.

num_rows property

num_rows: int

The number of rows in this chunk.

timeline_names property

timeline_names: list[str]

The names of all timelines in this chunk.

apply_lenses

def apply_lenses(lenses: Sequence[Lens] | Lens) -> list[Chunk]

Apply one or more lenses to this chunk, returning transformed chunks.

Each lens matches by input component. Columns not consumed by any matching lens are forwarded unchanged as a separate chunk.

If no lens matches the chunk (including when an empty list of lenses is passed), the original chunk is returned unchanged.

PARAMETER	DESCRIPTION
lenses	One or more Lens objects. TYPE: Sequence[Lens] | Lens

RETURNS	DESCRIPTION
A list of [`Chunk`][] objects.

apply_selector

def apply_selector(
    source: ComponentDescriptor | str,
    selector: Selector | str,
) -> Chunk

Apply a selector to a single component, returning a new chunk with the component transformed.

All other columns (timelines, other components) are preserved unchanged. The source component's existing descriptor is preserved.

For better performance, prefer MutateLens with apply_lenses which processes multiple transformations in a single pass.

PARAMETER	DESCRIPTION
source	A ComponentDescriptor or component identifier string for the input column to transform. TYPE: ComponentDescriptor | str
selector	A Selector or selector query string to apply to the component. TYPE: Selector | str

RETURNS	DESCRIPTION
A new [`Chunk`][rerun.chunk.Chunk] with the component transformed.

RAISES	DESCRIPTION
ValueError	If the source component is not found in the chunk or the selector fails to evaluate.

format

def format(
    *,
    width: int = 240,
    redact: bool = False,
    trim_metadata_keys: bool = True,
) -> str

Format this chunk as a human-readable table string.

PARAMETER	DESCRIPTION
width	Fixed width for the table. Default: 240. TYPE: int DEFAULT: 240
redact	If True, redact non-deterministic values (RowIds, ChunkIds, etc.) for stable snapshot testing. Default: False. TYPE: bool DEFAULT: False
trim_metadata_keys	If True, trim the rerun: / sorbet: prefix from metadata keys. Default: True. TYPE: bool DEFAULT: True

from_columns classmethod

def from_columns(
    entity_path: str,
    indexes: Iterable[TimeColumnLike],
    columns: Iterable[ComponentColumn],
) -> Chunk

Create a Chunk from columns, mirroring the rerun.send_columns API.

A fresh chunk ID and sequential row IDs are auto-generated.

PARAMETER	DESCRIPTION
entity_path	The entity path for this chunk (e.g., "/camera/image"). TYPE: str
indexes	The time columns for this chunk. Each TimeColumnLike provides a timeline name and a PyArrow array of timestamps. You typically use TimeColumn here. Pass an empty iterable for static data. TYPE: Iterable[TimeColumnLike]
columns	The component columns for this chunk. Each ComponentColumn provides a component descriptor and a PyArrow array of component data. TYPE: Iterable[ComponentColumn]

RAISES	DESCRIPTION
ValueError	If timeline and component column lengths don't match.

Example

chunk = Chunk.from_columns(
    "/robots/arm",
    indexes=[rr.TimeColumn("frame", sequence=[0, 1, 2])],
    columns=rr.Points3D.columns(positions=[[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
)

from_dataframe classmethod

def from_dataframe(
    dataframe: DataframeLike,
    *,
    index: str | list[str] | None | _AutoIndex = AUTO_INDEX,
    entity_path: str | None = None,
) -> Iterator[Chunk]

Lazily turn an Arrow-backed dataframe into chunks.

Accepts a Table, a RecordBatch, a RecordBatchReader, or any object implementing the Arrow C stream interface (__arrow_c_stream__) — most notably a datafusion.DataFrame (an optional dependency).

Yields each chunk of Chunk.from_record_batch applied to every record batch in turn. See that method for the index and entity_path semantics.

RAISES	DESCRIPTION
TypeError	If dataframe is not a pyarrow Table, a pyarrow RecordBatch, a pyarrow RecordBatchReader, or an Arrow-C-stream object (such as a datafusion.DataFrame).
ValueError	See Chunk.from_record_batch.

from_property classmethod

def from_property(
    name: str,
    values: AsComponents
    | Iterable[DescribedComponentBatch],
) -> Chunk

Create a Chunk from a property, mirroring the rerun.send_property API.

PARAMETER	DESCRIPTION
name	The name of the property. TYPE: str
values	The values for this property, either as AsComponents or an iterable of DescribedComponentBatch. TYPE: AsComponents | Iterable[DescribedComponentBatch]

RAISES	DESCRIPTION
TypeError	If values is neither AsComponents nor an iterable of DescribedComponentBatch.
ValueError	If the component data is invalid.

Example

chunk = Chunk.from_property(
    "my_property",
    values=rr.Points3D.positions([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
)

from_record_batch classmethod

def from_record_batch(
    record_batch: RecordBatch,
    *,
    index: str | list[str] | None | _AutoIndex = AUTO_INDEX,
    entity_path: str | None = None,
) -> list[Chunk]

Interpret an Arrow RecordBatch as Rerun chunk data.

Each column of the batch is classified as a row-id column, index (timeline) column, or a component column. Component columns are then grouped per entity path, and one chunk per entity path is emitted.

The rerun:* arrow metadata, if it exists, drives the kind of each input column, as well as the entity/archetype/component type for component columns.

If present, the row id column and chunk id metadata indicate that the batch represents a fully identified chunk, e.g. as produced by Chunk.to_record_batch. Both the row ids and chunk id are preserved under the following conditions: - both are present in the input batch - index is omitted - entity_path is omitted

If any of these conditions are not met, it means that either the batch is not fully identified, or that the chunk data is reinterpreted (e.g. entity path rewriting). In that case, fresh row ids and chunk id are generated and used instead of the input ones.

PARAMETER	DESCRIPTION
record_batch	The Arrow record batch to interpret. Component columns may be either lists (one component batch per row) or plain arrays (wrapped as single-element lists automatically). TYPE: RecordBatch
index	Determines which columns are index (timeline) columns. Each promoted column's time type is taken from its Arrow datatype: int64 → sequence, timestamp(ns) → timestamp, duration(ns) → duration. Omitted (the default): derive the index columns from the batch's Rerun metadata. The batch is treated as temporal if it carries index metadata. A batch with no index metadata is ambiguous and raises an error — unless it is an already-identified chunk (it carries a row-id column and a chunk id), which round-trips as-is and may therefore be static. Pass index=None to force a static interpretation. A column name, or list of column names: treat exactly these columns as timelines. The remaining (non-row-id) columns become components. None: produce static chunks (no timeline). Any index metadata or promoted index column is then a contradiction and is rejected. Note Static chunks with multiple rows are legitimate in some cases, but only the last row is visible from typical latest-at queries. An info-level message is emitted when this happens — except for an already-identified chunk that is preserved as-is (see above), which is passed through without this check. TYPE: str | list[str] | None | _AutoIndex DEFAULT: AUTO_INDEX
entity_path	Default entity path for component columns that do not otherwise specify one. Resolution order per component column is: its rerun:entity_path metadata, then the batch-level rerun:entity_path metadata, then the column-name convention (see Notes), then this argument, then the root entity (/). TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
One chunk per distinct entity path described by the batch, in first-seen column order.

RAISES

DESCRIPTION

ValueError

In any of the following cases: index was omitted and the batch carries no index metadata (an ambiguous raw batch). Pass index=<column> for temporal data or index=None for static data. index=None was given but the batch also carries index metadata or names an index column (contradiction). index names a column that is not present in the batch. The batch contains no component columns (there is nothing to log). A column promoted to an index contains null values. Time columns must be dense; static data is expressed with index=None, not with null times. An index column has an Arrow datatype that is not a supported time type. The batch is a fully-identified chunk (it carries both a row-id column and a chunk id) but resolves to more than one entity path. An identified chunk is preserved as a single chunk; drop the chunk-id metadata and/or the row-id column to reinterpret it into one chunk per entity (with freshly-minted ids).

Notes

Column-name convention. When a component column carries no rerun:entity_path / rerun:component metadata, its entity path is read from the column name: if the name starts with / and contains a :, the first part of the column name is interpreted as the entity path and the rest as the component identifier. Example: /point:Points3D:positions and /metadata:foo.

Limitations/Future work

A batch that mixes static and temporal rows — aka where some index values are null — are rejected. Handling this case requires row-splitting and generating a mix of temporal and static chunks.

Recording-property columns (named property:…, mapping to the /__properties entity) are not recognized by the column-name convention and are not mapped back to that entity.

to_record_batch

def to_record_batch() -> RecordBatch

Convert this chunk to an Arrow RecordBatch.

with_entity_path

def with_entity_path(entity_path: str) -> Chunk

Return a copy of this chunk with a new entity path.

A fresh chunk ID is generated to avoid aliasing the original chunk in downstream caches and indices. Row IDs, timelines, and components are preserved as-is.

PARAMETER	DESCRIPTION
entity_path	The new entity path for the returned chunk (e.g. "/left/camera/image"). TYPE: str

ChunkStore

A fully-materialized, in-memory chunk store.

Build one from chunks via ChunkStore.from_chunks, or fully materialize an IndexedReader via reader.stream().collect(). For lazy, on-demand chunk loading, see LazyStore.

Use stream() to process chunks through the lazy pipeline, or write_rrd() to persist to disk.

__len__

def __len__() -> int

Return the number of chunks in this store.

from_chunks staticmethod

def from_chunks(chunks: Sequence[Chunk]) -> ChunkStore

Build a ChunkStore from a sequence of chunks.

reader

def reader(
    index: str | None,
    *,
    contents: ContentFilter | str | list[str] | None = None,
    include_semantically_empty_columns: bool = False,
    include_tombstone_columns: bool = False,
    fill_latest_at: bool = False,
    using_index_values: IndexValuesLike | None = None,
    ctx: SessionContext | None = None,
) -> DataFrame

Build a DataFusion DataFrame over this store.

The returned DataFrame is data-equivalent to the result of round-tripping the same chunks through write_rrd → rr.server.Server → dataset.reader(), modulo the rerun_segment_id column (absent here because a single ChunkStore has no segment concept).

PARAMETER	DESCRIPTION
index	The index (timeline) column to use, or None for the static-only view. TYPE: str | None
contents	Entity-path filter. A ContentFilter built with the fluent API, a single entity-path expression, a list of expressions, or None for everything. An empty list returns no rows. TYPE: ContentFilter | str | list[str] | None DEFAULT: None
include_semantically_empty_columns	Whether to include columns that are semantically empty. TYPE: bool DEFAULT: False
include_tombstone_columns	Whether to include tombstone columns. TYPE: bool DEFAULT: False
fill_latest_at	Whether to fill null values with the latest valid data. TYPE: bool DEFAULT: False
using_index_values	Index values at which to resample data. When specified, this argument changes the way rows are returned. Instead of returning the rows that exist in the data, one row is returned per index_value you provide. If the segment has no row at that index value, nulls are returned — or the latest prior value if fill_latest_at=True` (which is typically what you want for resampling). Don't use this argument for plain index slicing — use a DataFusion filter on the index column instead. For example: from datafusion import col, lit # All rows in a time window. store.reader(index="real_time").filter( (col("real_time") >= lit(t0)) & (col("real_time") <= lit(t1)) ) TYPE: IndexValuesLike | None DEFAULT: None
ctx	DataFusion SessionContext to register the table into. When None, uses datafusion.SessionContext.global_ctx() — the process-wide default. Pass an explicit ctx for isolation or a custom SessionConfig. TYPE: SessionContext | None DEFAULT: None

schema

def schema() -> Schema

The schema describing all columns in this store.

stream

def stream() -> LazyChunkStream

Return a lazy stream over all chunks in this store.

summary

def summary() -> str

Compact, deterministic summary of every chunk in the store.

Each line describes one chunk:

{entity_path}  rows={n}  static={True|False}  timelines=[…]  cols=[…]

Useful for snapshot testing.

write_rrd

def write_rrd(
    path: str | Path,
    *,
    application_id: str,
    recording_id: str,
) -> None

Write all chunks to an RRD file.

The caller must provide application_id and recording_id explicitly.

DeriveLens

A derive lens that creates new component/time columns from an input component.

Derive lenses extract fields from a component and produce new columns, optionally at a different entity and/or with new time columns.

Pass scatter=True to enable 1:N row mapping (exploding lists).

Example usage::

lens = (
    DeriveLens("Imu:accel")
    .to_component(rr.Scalars.descriptor_scalars(), Selector(".x"))
)

To write to an explicit target entity::

lens = (
    DeriveLens("Imu:accel", output_entity="/out/x")
    .to_component(rr.Scalars.descriptor_scalars(), Selector(".x"))
)

__init__

def __init__(
    input_component: str,
    *,
    output_entity: str | None = None,
    scatter: bool = False,
) -> None

Create a new derive lens.

PARAMETER	DESCRIPTION
input_component	The component identifier to match (e.g. "Imu:accel"). TYPE: str
output_entity	Optional target entity path. When set, output is written to this entity instead of the input entity. TYPE: str | None DEFAULT: None
scatter	When True, use 1:N row mapping (explode lists). TYPE: bool DEFAULT: False

to_component

def to_component(
    component: ComponentDescriptor | str,
    selector: Selector | str,
    *,
    cast_to: DataType | Literal["auto"] | None = None,
) -> DeriveLens

Add a component output column.

PARAMETER	DESCRIPTION
component	A ComponentDescriptor or a component identifier string for the output column (e.g. "Scalars:scalars"). TYPE: ComponentDescriptor | str
selector	A Selector or selector query string to apply to the input column. TYPE: Selector | str
cast_to	How to cast the produced column to match the target component. By default (None) the column is emitted as-is. Pass "auto" to cast it to the component's canonical Arrow datatype, or an explicit pyarrow DataType to cast it to that type. Casting errors if the conversion is unsupported. TYPE: DataType | Literal['auto'] | None DEFAULT: None

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the component added.

to_packed_component

def to_packed_component(
    component: ComponentDescriptor | str,
    *fields: str,
    cast_to: DataType | Literal["auto"] | None = "auto",
) -> DeriveLens

Add a component output column by packing the provided fields in a fixed-size list.

PARAMETER	DESCRIPTION
component	A ComponentDescriptor or a component identifier string for the output column (e.g. "Points3D:positions"). TYPE: ComponentDescriptor | str
*fields	Names of the struct fields to pack, in order. They must all resolve to the same datatype. At least one field is required. TYPE: str DEFAULT: ()
cast_to	How to cast the packed column to match the target component. Defaults to "auto", which casts to the component's canonical Arrow datatype (e.g. the f64 columns a parquet file typically holds → the f32 a Transform3D:translation expects). Pass an explicit pyarrow DataType to cast to that type, or None to emit the packed list as-is. TYPE: DataType | Literal['auto'] | None DEFAULT: 'auto'

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the packed component added.

to_quaternion

def to_quaternion(x: str, y: str, z: str, w: str) -> DeriveLens

Add a Transform3D:quaternion component from the provided paths.

PARAMETER	DESCRIPTION
x	Paths of the struct fields holding the quaternion components, in xyzw order. TYPE: str
y	Paths of the struct fields holding the quaternion components, in xyzw order. TYPE: str
z	Paths of the struct fields holding the quaternion components, in xyzw order. TYPE: str
w	Paths of the struct fields holding the quaternion components, in xyzw order. TYPE: str

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the quaternion added.

to_rotation_axis_angle

def to_rotation_axis_angle(
    axis_x: str, axis_y: str, axis_z: str, angle: str
) -> DeriveLens

Add a Transform3D:rotation_axis_angle component from the provided paths.

PARAMETER	DESCRIPTION
axis_x	Paths of the struct fields holding the rotation axis components. TYPE: str
axis_y	Paths of the struct fields holding the rotation axis components. TYPE: str
axis_z	Paths of the struct fields holding the rotation axis components. TYPE: str
angle	Path of the struct field holding the rotation angle, in radians. TYPE: str

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the rotation added.

to_scalars

def to_scalars(*fields: str) -> DeriveLens

Add a Scalars:scalars component from the provided path(s).

Each path becomes one scalar instance per row, so a single path yields one series and multiple paths yield one series each at the same entity.

PARAMETER	DESCRIPTION
*fields	Paths of the struct fields to read as scalars, in order. At least one is required. TYPE: str DEFAULT: ()

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the scalars added.

to_scale

def to_scale(x: str, y: str, z: str) -> DeriveLens

Add a Transform3D:scale component from the provided paths.

PARAMETER	DESCRIPTION
x	Paths of the struct fields holding the per-axis scale factors. TYPE: str
y	Paths of the struct fields holding the per-axis scale factors. TYPE: str
z	Paths of the struct fields holding the per-axis scale factors. TYPE: str

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the scale added.

to_timeline

def to_timeline(
    timeline_name: str,
    timeline_type: Literal[
        "sequence", "duration_ns", "timestamp_ns"
    ],
    selector: Selector | str,
) -> DeriveLens

Add a time extraction column.

PARAMETER	DESCRIPTION
timeline_name	Name of the timeline to create. TYPE: str
timeline_type	Type of the timeline: "sequence", "duration_ns", or "timestamp_ns". TYPE: Literal['sequence', 'duration_ns', 'timestamp_ns']
selector	A Selector or selector query string to extract time values (must produce Int64 arrays). TYPE: Selector | str

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the time column added.

to_translation

def to_translation(x: str, y: str, z: str) -> DeriveLens

Add a Transform3D:translation component from the provided paths.

PARAMETER	DESCRIPTION
x	Paths of the struct fields holding the translation components. TYPE: str
y	Paths of the struct fields holding the translation components. TYPE: str
z	Paths of the struct fields holding the translation components. TYPE: str

RETURNS	DESCRIPTION
A new [`DeriveLens`][rerun.chunk.DeriveLens] with the translation added.

IndexColumn dataclass

A dataset/column to use as a timeline index, and how to interpret it.

Construct one with timestamp, duration, or sequence — the timeline kind is the constructor you pick, so there is nothing to mistype:

IndexColumn.timestamp("/time", input_unit="s")
IndexColumn.duration("/elapsed", input_unit="us")
IndexColumn.sequence("/frame_id")

input_unit class-attribute instance-attribute

input_unit: TimeUnit | None = None

What the raw integer/float values represent (not a desired output unit); values are scaled to nanoseconds internally. None for sequence.

kind instance-attribute

kind: Literal['timestamp', 'duration', 'sequence']

The timeline kind: time since epoch, elapsed time, or an ordinal integer index.

path instance-attribute

path: str

Path of the 1-D dataset (HDF5) or name of the column (Parquet) to use as the index.

duration classmethod

def duration(
    path: str, *, input_unit: TimeUnit = "ns"
) -> IndexColumn

An elapsed-time timeline. input_unit describes the raw values (default "ns").

sequence classmethod

def sequence(path: str) -> IndexColumn

An ordinal integer-index timeline. No unit applies.

timestamp classmethod

def timestamp(
    path: str, *, input_unit: TimeUnit = "ns"
) -> IndexColumn

A time-since-epoch timeline. input_unit describes the raw values (default "ns").

IndexedReader

Bases: StreamingReader, Protocol

Protocol for readers backed by an index/manifest.

Extends StreamingReader: every IndexedReader also supports stream() -> LazyChunkStream for pure-streaming processing.

Indexed readers expose a LazyStore view over the source via store() — the manifest is read up-front; chunks load on demand. To fully materialize into a ChunkStore, call stream().collect().

store

def store() -> LazyStore

Return a LazyStore view of this source.

stream

def stream() -> LazyChunkStream

Return a lazy stream over all chunks from this source.

LazyChunkStream

A lazy, composable pipeline over chunks.

Builder methods (filter, drop, split, map, flat_map, lenses, merge) consume the input stream(s) and return new stream(s). A consumed stream cannot be used again; attempting to do so raises a ValueError. This prevents accidental reuse that would result in duplicate use of the same stream in a pipeline.

Terminal methods (to_chunks, __iter__, collect, write_rrd) do not consume the stream — they run the pipeline and leave the stream usable. Each call creates a fresh execution.

__iter__

def __iter__() -> Iterator[Chunk]

Iterate over chunks one at a time (triggers execution).

collect

def collect(
    *, optimize: OptimizationProfile | None = None
) -> ChunkStore

Run the pipeline and materialize all chunks into a ChunkStore.

By default, only the single-pass compaction that happens naturally during chunk insertion is applied. Pass optimize=OptimizationProfile.LIVE or optimize=OptimizationProfile.OBJECT_STORE to run additional optimization (extra convergence passes, video GoP rebatching) tuned for the chosen target.

PARAMETER	DESCRIPTION
optimize	If None (default), no extra optimization is performed beyond the single pass that happens on insert. Otherwise, apply the given profile after insertion. TYPE: OptimizationProfile | None DEFAULT: None

Examples:

Run with the object-store-tuned profile:

store = reader.stream().collect(optimize=OptimizationProfile.OBJECT_STORE)

drop

def drop(
    *,
    content: ContentFilter
    | str
    | Sequence[str]
    | None = None,
    has_timeline: str | None = None,
    is_static: bool | None = None,
    components: ComponentDescriptor
    | str
    | Sequence[ComponentDescriptor | str]
    | None = None,
) -> LazyChunkStream

Drop the matching portion of each chunk; keep the rest. Consumes this stream.

Complement of filter(): what filter() would keep is discarded, what it would discard is kept.

PARAMETER	DESCRIPTION
content	Entity path filter. Accepts a single expression, a list of expressions, or a ContentFilter object. TYPE: ContentFilter | str | Sequence[str] | None DEFAULT: None
has_timeline	Only drop chunks that have a column for this timeline. TYPE: str | None DEFAULT: None
is_static	If True, drop only static chunks. If False, drop only temporal chunks. TYPE: bool | None DEFAULT: None
components	Drop the listed component columns. Accepts ComponentDescriptor objects or str component identifiers (e.g. "Points3D:positions"). A single value or a list are both accepted. TYPE: ComponentDescriptor | str | Sequence[ComponentDescriptor | str] | None DEFAULT: None

filter

def filter(
    *,
    content: ContentFilter
    | str
    | Sequence[str]
    | None = None,
    has_timeline: str | None = None,
    is_static: bool | None = None,
    components: ComponentDescriptor
    | str
    | Sequence[ComponentDescriptor | str]
    | None = None,
) -> LazyChunkStream

Keep the matching portion of each chunk; drop the rest. Consumes this stream.

All criteria are combined with AND. For chunk-level predicates (content, has_timeline, is_static) the chunk either passes or is dropped entirely. For components, the chunk is split by component columns: only matching component columns are kept (timelines and entity path are preserved). When a list is given, any column matching any of the listed components is kept (OR semantics). Chunks that contain none of the listed components are dropped entirely.

If a chunk fails any predicate, it is dropped entirely -- no component splitting occurs.

PARAMETER	DESCRIPTION
content	Entity path filter. Accepts a single expression, a list of expressions, or a ContentFilter object. TYPE: ContentFilter | str | Sequence[str] | None DEFAULT: None
has_timeline	Only keep chunks that have a column for this timeline. TYPE: str | None DEFAULT: None
is_static	If True, keep only static chunks. If False, keep only temporal chunks. TYPE: bool | None DEFAULT: None
components	Keep only the listed component columns. Accepts ComponentDescriptor objects or str component identifiers (e.g. "Points3D:positions"). A single value or a list are both accepted. TYPE: ComponentDescriptor | str | Sequence[ComponentDescriptor | str] | None DEFAULT: None

flat_map

def flat_map(
    fn: Callable[[Chunk], Iterable[Chunk]],
) -> LazyChunkStream

Apply a Python function to each chunk, producing zero or more output chunks. Consumes this stream.

Runs in Python (GIL-bound, sequential).

from_iter staticmethod

def from_iter(chunks: Iterable[Chunk]) -> LazyChunkStream

Wrap a Python iterable of Chunks into a LazyChunkStream.

Enables user-defined sources and the generator escape hatch.

lenses

def lenses(
    lenses: Sequence[Lens] | Lens,
    *,
    output_mode: Literal[
        "drop_unmatched", "forward_unmatched", "forward_all"
    ] = "drop_unmatched",
    content: ContentFilter
    | str
    | Sequence[str]
    | None = None,
) -> LazyChunkStream

Apply lenses to transform chunk data. Consumes this stream.

Each lens matches chunks by entity path and input component, then transforms the data according to its output specifications.

PARAMETER	DESCRIPTION
lenses	One or more Lens objects. TYPE: Sequence[Lens] | Lens
output_mode	How to handle unmatched chunks: "forward_all": forward both transformed and original data "forward_unmatched": forward transformed if matched, otherwise original "drop_unmatched": only forward transformed data (default) TYPE: Literal['drop_unmatched', 'forward_unmatched', 'forward_all'] DEFAULT: 'drop_unmatched'
content	Optional entity path filter. When set, lenses are applied only to chunks whose entity path matches; non-matching chunks pass through unchanged regardless of output_mode. TYPE: ContentFilter | str | Sequence[str] | None DEFAULT: None

map

def map(fn: Callable[[Chunk], Chunk]) -> LazyChunkStream

Apply a Python function to each chunk, producing exactly one output chunk. Consumes this stream.

Runs in Python (GIL-bound, sequential). For transforms that may produce zero or many chunks, use flat_map instead.

merge staticmethod

def merge(*streams: LazyChunkStream) -> LazyChunkStream

Merge chunks from multiple streams into one. Consumes all input streams.

All inputs execute concurrently. Chunks are yielded as they become available. Within each input, chunk order is preserved. Across inputs, ordering is non-deterministic.

split

def split(
    *,
    content: ContentFilter
    | str
    | Sequence[str]
    | None = None,
    has_timeline: str | None = None,
    is_static: bool | None = None,
    components: ComponentDescriptor
    | str
    | Sequence[ComponentDescriptor | str]
    | None = None,
) -> tuple[LazyChunkStream, LazyChunkStream]

Split into (matching, non_matching). Consumes this stream.

Equivalent to (stream.filter(…), stream.drop(…)), but the upstream executes only once. merge(matching, non_matching) reconstructs the original stream in a semantically lossless way (component-wise chunk splitting is not undone).

Both branches share the same upstream -- it executes once. Both branches MUST be consumed for the pipeline to complete (dropping an unconsumed branch is fine and unblocks the other).

PARAMETER	DESCRIPTION
content	Entity path filter. Accepts a single expression, a list of expressions, or a ContentFilter object. TYPE: ContentFilter | str | Sequence[str] | None DEFAULT: None
has_timeline	Only match chunks that have a column for this timeline. TYPE: str | None DEFAULT: None
is_static	If True, match only static chunks. If False, match only temporal chunks. TYPE: bool | None DEFAULT: None
components	Match the listed component columns. Accepts ComponentDescriptor objects or str component identifiers (e.g. "Points3D:positions"). A single value or a list are both accepted. TYPE: ComponentDescriptor | str | Sequence[ComponentDescriptor | str] | None DEFAULT: None

to_chunks

def to_chunks() -> list[Chunk]

Run the pipeline and return all chunks as a list.

write_rrd

def write_rrd(
    path: str | Path,
    *,
    application_id: str,
    recording_id: str,
) -> None

Run the pipeline and write all chunks to an RRD file.

The caller must provide application_id and recording_id explicitly.

LazyStore

Index-based, on-demand chunk store.

The manifest is held in memory (so schema(), summary(), and __len__ work without loading any chunks), but chunk data is loaded only when requested.

Example: lazy = RrdReader("recording.rrd").store()

Use stream() to process chunks through the lazy pipeline, or write_rrd() to persist to disk. To fully materialize into a ChunkStore, call lazy.stream().collect().

__len__

def __len__() -> int

Return the number of chunks described by the manifest.

schema

def schema() -> Schema

The schema describing all columns in this store, derived from the manifest.

stream

def stream() -> LazyChunkStream

Return a lazy stream over all chunks in this store.

summary

def summary() -> str

Compact, deterministic summary of every chunk in the store.

Built from the manifest; no chunk data is loaded. Each line describes one chunk:

{entity_path}  rows={n}  static={True|False}  timelines=[…]  cols=[…]

Useful for snapshot testing.

write_rrd

def write_rrd(
    path: str | Path,
    *,
    application_id: str,
    recording_id: str,
) -> None

Write all chunks to an RRD file.

The caller must provide application_id and recording_id explicitly.

McapChannelInfo dataclass

Information about one MCAP channel.

McapChunkInfo dataclass

Aggregate information about indexed MCAP chunks.

McapCompressionInfo dataclass

Aggregate information for one MCAP chunk compression codec.

savings_ratio property

savings_ratio: float | None

Return the fraction of uncompressed bytes removed by compression.

McapInfo dataclass

Header and summary information about a complete MCAP file.

This joins channels with their schemas, aggregates chunk compression, and provides derived time and frequency values. It contains no message or schema payloads and is unaffected by the reader's decoder, topic, and time filters.

McapReader

Read chunks from an MCAP file.

path property

path: Path

The file path of the MCAP file.

__init__

def __init__(
    path: str | Path,
    *,
    timeline_type: Literal[
        "timestamp", "duration"
    ] = "timestamp",
    timestamp_offset_ns: int | None = None,
    decoders: Sequence[str] | None = None,
    include_topic_regex: Sequence[str] | None = None,
    exclude_topic_regex: Sequence[str] | None = None,
    start_time_ns: int | None = None,
    end_time_ns: int | None = None,
    recover: bool = False,
) -> None

Construct a new MCAP reader.

PARAMETER	DESCRIPTION
path	Path to the .mcap file to read. TYPE: str | Path
timeline_type	Whether to interpret the MCAP log_time column as wall-clock timestamps ("timestamp") or as nanosecond durations ("duration"). TYPE: Literal['timestamp', 'duration'] DEFAULT: 'timestamp'
timestamp_offset_ns	Optional offset in nanoseconds to add to all TimestampNs time columns. TYPE: int | None DEFAULT: None
decoders	Optional list of MCAP decoder identifiers to enable. If omitted, all available decoders are enabled. Use McapReader.available_decoders to enumerate them. TYPE: Sequence[str] | None DEFAULT: None
include_topic_regex	Optional list of regex patterns. If provided, only topics matching at least one pattern are decoded. Patterns use RE2 syntax and are not implicitly anchored. TYPE: Sequence[str] | None DEFAULT: None
exclude_topic_regex	Optional list of regex patterns. Topics matching any pattern are skipped. Applied after includes. Same syntax as include_topic_regex. TYPE: Sequence[str] | None DEFAULT: None
start_time_ns	Optional inclusive lower bound on the raw MCAP log_time (nanoseconds). Messages before this time are skipped. None leaves the range open at the start. TYPE: int | None DEFAULT: None
end_time_ns	Optional exclusive upper bound on the raw MCAP log_time (nanoseconds). Messages at or after this time are skipped. None leaves the range open at the end. TYPE: int | None DEFAULT: None
recover	Whether to recover a missing or invalid MCAP summary in memory. Our reader normally requires the summary + chunk index that live at the end of the file, so an interrupted recording (valid start, truncated tail, no footer/summary) fails to read. When recover is set, the summary is reconstructed from a front-to-back scan instead. A chunk is recovered in full even if the recording stopped part-way through the index records that follow it, but a chunk or record whose own body is incomplete is dropped with a warning, and any channel declared only inside it is lost. The recovered statistics only count the channels and messages that could be recovered. Healthy files are unaffected. TYPE: bool DEFAULT: False

available_decoders staticmethod

def available_decoders() -> list[str]

Return the list of all supported MCAP decoder identifiers.

info

def info() -> McapInfo

Return structured information about the complete MCAP file.

The underlying file information is assembled and cached in Rust on first use. Decoder selection, topic filters, time filters, timeline type, and timestamp offset do not affect it. With recover=True, a reconstructed summary describes only the recoverable portion of a damaged or incomplete file.

stream

def stream(
    *,
    start_time_ns: int | None = None,
    end_time_ns: int | None = None,
) -> LazyChunkStream

Return a lazy stream over the chunks in the MCAP file.

start_time_ns and end_time_ns override the values passed to the constructor, for this scan only. If either start_time_ns or end_time_ns are provided both are reset.

time_bounds

def time_bounds() -> tuple[int, int]

Return the (min, max) MCAP log_time bounds (nanoseconds, inclusive).

McapSchemaInfo dataclass

Schema metadata referenced by an MCAP channel, excluding the schema payload.

MutateLens

A mutate lens that modifies the input component in-place.

Mutate lenses apply a selector transformation to the input component, replacing it in the chunk. By default, new row IDs are generated. Pass keep_row_ids=True to preserve original row IDs.

Example usage::

lens = MutateLens("Imu:accel", Selector(".x"))

__init__

def __init__(
    input_component: str,
    selector: Selector | str,
    *,
    keep_row_ids: bool = False,
) -> None

Create a new mutate lens.

PARAMETER	DESCRIPTION
input_component	The component identifier to modify in-place. TYPE: str
selector	A Selector or selector query string to apply. TYPE: Selector | str
keep_row_ids	When True, preserve the original row IDs. TYPE: bool DEFAULT: False

OptimizationProfile dataclass

Named optimization profile passed to LazyChunkStream.collect(optimize=...).

Two presets:

• OptimizationProfile.LIVE: small chunks tuned for the live Viewer workflow.
• OptimizationProfile.OBJECT_STORE: large chunks tuned for object-store-backed query and streaming (e.g. a catalog server).

The presets are fully concrete: every field has a value. Custom profiles built by calling OptimizationProfile(...) directly may pass None on the threshold fields to fall back to the SDK's internal default (OptimizationProfile.LIVE's thresholds).

LIVE class-attribute

LIVE: OptimizationProfile

Optimized for the live Viewer workflow: small chunks for low-latency rendering and fine-grained time-panel precision.

OBJECT_STORE class-attribute

OBJECT_STORE: OptimizationProfile

Optimized for object-store-backed storage (e.g. a catalog server): larger chunks tuned for query throughput and streaming over the network.

extra_passes class-attribute instance-attribute

extra_passes: int = 50

Number of extra convergence passes run after the initial insert.

fix_keyframe class-attribute instance-attribute

fix_keyframe: bool = False

If True, any user-supplied VideoStream:is_keyframe data is dropped and re-derived from the encoded samples during video rebatching.

gop_batching class-attribute instance-attribute

gop_batching: bool = True

If True (default), video stream chunks are rebatched to align with GoP (keyframe) boundaries after normal compaction.

GoP rebatching never splits a GoP across chunks, so streams with long keyframe intervals can produce chunks much larger than max_bytes.

max_bytes class-attribute instance-attribute

max_bytes: int | None = None

Chunk size threshold in bytes. None means use LIVE's default.

max_rows class-attribute instance-attribute

max_rows: int | None = None

Maximum rows per sorted chunk. None means use LIVE's default.

max_rows_if_unsorted class-attribute instance-attribute

max_rows_if_unsorted: int | None = None

Maximum rows per unsorted chunk. None means use LIVE's default.

split_size_ratio class-attribute instance-attribute

split_size_ratio: float | None = None

If set, split chunks so no two archetype groups sharing a chunk differ in byte size by more than this factor. Values should be >= 1; at 1.0, every archetype is forced into its own chunk.

This keeps large columns (images, videos, blobs) out of the same chunk as small columns (scalars, transforms, text), so the viewer can fetch just the small columns without dragging along the large payload.

Components belonging to the same archetype are always kept together, because an archetype's components are only meaningful as a set: an EncodedImage:blob cannot be decoded without its EncodedImage:media_type. Splitting them would only force a reader to fetch both chunks anyway.

The exception is a component that always gets a chunk of its own, such as VideoStream:is_keyframe. Those are separated out first, whether or not this is set.

A good starting value is 10.0. If None (default), no splitting is performed.

RrdReader

Read chunks from an RRD file.

Use recordings() or blueprints() to discover what stores exist in the file, then stream() or store() to access a specific one. When no store is specified, the first recording store is used.

path property

path: Path

The file path of the RRD file.

blueprints

def blueprints() -> list[StoreEntry]

List the blueprint entries in this RRD file.

recordings

def recordings() -> list[StoreEntry]

List the recording entries in this RRD file.

store

def store(*, store: StoreEntry | None = None) -> LazyStore

Open a specific store as a LazyStore.

Reads the manifest immediately; chunk data is loaded on demand. Legacy RRDs without a footer/manifest are not supported here — use RrdReader(...).stream().collect() for those.

PARAMETER	DESCRIPTION
store	Which store to load. If None, uses the first recording store. TYPE: StoreEntry | None DEFAULT: None

RAISES	DESCRIPTION
ValueError	If the specified store is not in this RRD file, or None was passed and the file contains no recording stores.

stream

def stream(
    *, store: StoreEntry | None = None
) -> LazyChunkStream

Return a lazy stream over chunks from a store.

PARAMETER	DESCRIPTION
store	Which store to stream. If None, uses the first recording store. TYPE: StoreEntry | None DEFAULT: None

RAISES	DESCRIPTION
ValueError	If the specified store is not in this RRD file, or None was passed and the file contains no recording stores.

Selector

A jq-like query selector for Arrow arrays.

Selectors provide a path-based query language (inspired by jq) that operates on Arrow arrays in a columnar fashion.

Syntax overview:

• .field — access a named field in a struct
• [] — iterate over every element of a list
• [N] — index into a list by position
• ? — error suppression / optional operator
• ! — assert non-null
• | — pipe the output of one expression to another

Example usage::

selector = Selector(".location")
result = selector.execute(my_struct_array)

Selectors can also be piped into Python functions::

selector = Selector(".values").pipe(lambda arr: pa.compute.multiply(arr, 2))
result = selector.execute(my_struct_array)

__init__

def __init__(query: str) -> None

Parse a selector from a query string.

PARAMETER	DESCRIPTION
query	The selector query string (e.g. ".field", ".foo | .bar"). TYPE: str

execute

def execute(source: Array) -> Array | None

Execute this selector against a pyarrow array.

PARAMETER	DESCRIPTION
source	The input Arrow array to query. TYPE: Array

RETURNS	DESCRIPTION
The result array, or None if the selector's error was suppressed.

execute_per_row

def execute_per_row(source: Array) -> Array | None

Execute this selector against each row of a pyarrow list array.

The output is guaranteed to have the same number of rows as the input.

PARAMETER	DESCRIPTION
source	The input Arrow list array to query. TYPE: Array

RETURNS	DESCRIPTION
The result list array, or None if the selector's error was suppressed.

pipe

def pipe(
    func: Callable[[Array], Array | None] | Selector,
) -> Selector

Pipe the output of this selector through a transformation function or another selector.

Returns a new selector; the original is not modified.

PARAMETER	DESCRIPTION
func	A callable that accepts a pyarrow.Array and returns a pyarrow.Array or None, or another Selector to chain. TYPE: Callable[[Array], Array | None] | Selector

RETURNS	DESCRIPTION
A new [`Selector`][rerun.chunk.Selector] with the transformation applied.

StoreEntry

Describes a store found in an RRD file.

application_id property

application_id: str

The application ID of the store.

kind property

kind: Literal['recording', 'blueprint']

Store kind: "recording" or "blueprint".

recording_id property

recording_id: str

The recording ID of the store.

StreamingReader

Bases: Protocol

Protocol for readers that produce a sequential stream of chunks.

All readers provide stream() -> LazyChunkStream. Readers for indexable formats will additionally satisfy IndexedReader, which adds store() -> LazyStore and load() -> ChunkStore.

stream

def stream() -> LazyChunkStream

Return a lazy stream over all chunks from this source.