Catalog
rerun.catalog
IndexValuesLike
module-attribute
IndexValuesLike: TypeAlias = (
NDArray[int_] | NDArray[datetime64] | Int64Array
)
A type alias for index values.
This can be any numpy-compatible array of integers, or a pyarrow.Int64Array
AlreadyExistsError
Bases: Exception
Raised when trying to create a resource that already exists.
CatalogClient
Client for a remote Rerun catalog server.
Note: the datafusion package is required to use this client. Initialization will fail with an error if the package
is not installed.
ctx
property
ctx: SessionContext
Returns a DataFusion session context for querying the catalog.
__init__
benchmark
Measure round-trip time and download bandwidth to the server.
The RTT is estimated as the minimum elapsed time across num_pings 1-byte requests
(using the minimum rejects latency spikes from scheduling jitter or transient network
congestion). Bandwidth is measured by downloading num_bytes of pseudo-random
(incompressible) bytes, subtracting the RTT from the elapsed time, and dividing by the
payload size.
| PARAMETER | DESCRIPTION |
|---|---|
num_bytes
|
Total payload size to download from the server when measuring bandwidth.
TYPE:
|
num_pings
|
How many 1-byte requests to send when estimating RTT.
TYPE:
|
Examples:
client = rr.catalog.CatalogClient("…")
print(client.benchmark()) # BenchmarkResult(rtt=12.0 ms, bandwidth=112.0 MiB/s)
create_dataset
def create_dataset(
name: str, *, exist_ok: bool = False
) -> DatasetEntry
Creates a new dataset with the given name.
Entry names may only contain ASCII alphanumeric characters, underscores, hyphens, dots, colons and spaces, and must be at most 180 characters long.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
The name of the dataset to create.
TYPE:
|
exist_ok
|
If True, return the existing dataset if it already exists; otherwise, raise an error.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
AlreadyExistsError
|
If a dataset with the given name already exists and |
create_table
def create_table(
name: str, schema: Schema, url: str | None = None
) -> TableEntry
Create and register a new table.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
The name of the table entry to create. It must be unique within all entries in the catalog. An exception will be raised if an entry with the same name already exists. Entry names may only contain ASCII alphanumeric characters, underscores, hyphens, dots, colons and spaces, and must be at most 180 characters long.
TYPE:
|
schema
|
The schema of the table to create.
TYPE:
|
url
|
The URL of the directory for where to store the Lance table. If provided, the table will be stored in a globally unique subdirectory. If not provided, the server will use an automatically generated URL based on its configured writable storage. On Rerun Hub, custom table URLs are currently not supported: the request will be rejected unless this parameter is None.
TYPE:
|
dataset_names
Returns a list of all dataset names in the catalog.
| PARAMETER | DESCRIPTION |
|---|---|
include_hidden
|
If True, include blueprint datasets.
TYPE:
|
datasets
def datasets(
*, include_hidden: bool = False
) -> list[DatasetEntry]
Returns a list of all dataset entries in the catalog.
| PARAMETER | DESCRIPTION |
|---|---|
include_hidden
|
If True, include blueprint datasets.
TYPE:
|
do_global_maintenance
def do_global_maintenance() -> None
Perform maintenance tasks on the whole system.
entries
def entries(
*, include_hidden: bool = False
) -> list[DatasetEntry | TableEntry]
Returns a list of all entries in the catalog.
| PARAMETER | DESCRIPTION |
|---|---|
include_hidden
|
If True, include hidden entries (blueprint datasets and system tables like
TYPE:
|
entry_names
Returns a list of all entry names in the catalog.
| PARAMETER | DESCRIPTION |
|---|---|
include_hidden
|
If True, include hidden entries (blueprint datasets and system tables like
TYPE:
|
get_dataset
def get_dataset(*, id: EntryId | str) -> DatasetEntry
def get_dataset(name: str) -> DatasetEntry
def get_dataset(
name: str | None = None,
*,
id: EntryId | str | None = None,
) -> DatasetEntry
Returns a dataset by its ID or name.
Exactly one of id or name must be provided.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
The name of the dataset.
TYPE:
|
id
|
The unique identifier of the dataset. Can be an |
get_table
def get_table(*, id: EntryId | str) -> TableEntry
def get_table(name: str) -> TableEntry
def get_table(
name: str | None = None,
*,
id: EntryId | str | None = None,
) -> TableEntry
register_table
def register_table(name: str, url: str) -> TableEntry
Registers a foreign Lance table (identified by its URL) as a new table entry with the given name.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
The name of the table entry to create. It must be unique within all entries in the catalog. An exception will be raised if an entry with the same name already exists. Entry names may only contain ASCII alphanumeric characters, underscores, hyphens, dots, colons and spaces, and must be at most 180 characters long.
TYPE:
|
url
|
The URL of the Lance table to register.
TYPE:
|
table_names
Returns a list of all table names in the catalog.
| PARAMETER | DESCRIPTION |
|---|---|
include_hidden
|
If True, include system tables (e.g.,
TYPE:
|
tables
def tables(*, include_hidden: bool = False) -> list[TableEntry]
Returns a list of all table entries in the catalog.
| PARAMETER | DESCRIPTION |
|---|---|
include_hidden
|
If True, include system tables (e.g.,
TYPE:
|
version_info
def version_info() -> VersionInfo
Returns version and deployment information from the server.
Returns a VersionInfo object with version, cloud_provider, and cloud_region fields.
Cloud fields are None if the server is not deployed on a cloud provider.
ComponentColumnDescriptor
The descriptor of a component column.
Component columns contain the data for a specific component of an entity.
Column descriptors are used to describe the columns in a
Schema. They are read-only. To select a component
column, use ComponentColumnSelector.
component_type
property
component_type: str | None
The component type, if any.
This property is read-only.
ComponentColumnSelector
A selector for a component column.
Component columns contain the data for a specific component of an entity.
ContentFilter
Immutable builder for entity path filter expressions passed to filter_contents().
Start from everything() or nothing(), then chain include(), exclude(),
and include_properties() calls to build up the filter. Each method returns a
new ContentFilter, leaving the original unchanged.
Examples:
# Include everything except raw robot data, but keep one specific path:
view = dataset.filter_contents(
ContentFilter.everything()
.exclude("/robot/raw/**")
.include("/robot/raw/i_need_this")
)
# Include nothing, then allow specific subtrees:
view = dataset.filter_contents(
ContentFilter.nothing()
.include("/points", subtree=True)
.include("/world/camera")
)
# Path segments with special characters are auto-escaped via include_path/exclude_path:
f = ContentFilter.everything().exclude_path(["robot", "raw data", "sensor!"], subtree=True)
# equivalent to: exclude("/robot/raw\ data/sensor\!/**")
everything
classmethod
def everything() -> ContentFilter
Start with all entity paths included (auto-excludes __properties, see include_properties()).
Equivalent to filter_contents("/**").
exclude
def exclude(
path: str, *, subtree: bool = False
) -> ContentFilter
Exclude entity paths matching path.
| PARAMETER | DESCRIPTION |
|---|---|
path
|
A pre-formed entity path expression string (e.g.
TYPE:
|
subtree
|
If
TYPE:
|
include
def include(
path: str, *, subtree: bool = False
) -> ContentFilter
Include entity paths matching path.
| PARAMETER | DESCRIPTION |
|---|---|
path
|
A pre-formed entity path expression string (e.g.
TYPE:
|
subtree
|
If
TYPE:
|
include_properties
def include_properties() -> ContentFilter
Include the __properties/** subtree.
By default __properties is auto-excluded. Calling this method suppresses
that auto-exclusion for the entire subtree. See the filter_contents() docs
for details on __properties handling.
nothing
classmethod
def nothing() -> ContentFilter
Start with all entity paths excluded.
Equivalent to filter_contents([]).
DatasetEntry
Bases: Entry[DatasetEntryInternal]
A dataset entry in the catalog.
__eq__
Compare this entry to another object.
Supports comparison with str and EntryId to enable the following patterns:
"entry_name" in client.entries()
entry_id in client.entries()
arrow_schema
def arrow_schema() -> Schema
Return the Arrow schema of the data contained in the dataset.
asset_dataset
def asset_dataset() -> DatasetEntry | None
The associated asset dataset, if any.
The associated asset dataset is owned by this dataset for lifecycle purposes. Deleting this dataset also deletes the associated asset dataset and its storage.
blueprint_dataset
def blueprint_dataset() -> DatasetEntry | None
The associated blueprint dataset, if any.
The associated blueprint dataset is owned by this dataset for lifecycle purposes. Deleting this dataset also deletes the associated blueprint dataset and its storage.
blueprints
Lists all blueprints currently registered with this dataset.
default_segment_table_blueprint
def default_segment_table_blueprint() -> str | None
Return the name of the currently set segment table blueprint.
delete
def delete() -> None
Delete this entry from the catalog.
do_maintenance
def do_maintenance(
optimize_indexes: bool = False,
retrain_indexes: bool = False,
compact_fragments: bool = False,
cleanup_before: datetime | None = None,
unsafe_allow_recent_cleanup: bool = False,
) -> None
Perform maintenance tasks on the datasets.
filter_contents
def filter_contents(
exprs: ContentFilter | str | Sequence[str],
) -> DatasetView
Return a new DatasetView filtered to the given entity paths.
Entity path expressions support wildcards:
- "/points/**" matches all entities under /points
- "-/text/**" excludes all entities under /text
| PARAMETER | DESCRIPTION |
|---|---|
exprs
|
A
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
DatasetView
|
A new view filtered to the matching entity paths. |
Examples:
# Filter to a single entity path
view = dataset.filter_contents("/points/**")
view = dataset.filter_contents(
ContentFilter.nothing()
.include("/points/**")
)
# Filter to specific entity paths
view = dataset.filter_contents(["/points/**"])
# Exclude certain paths
view = dataset.filter_contents(["/points/**", "-/text/**"])
view = dataset.filter_contents(
ContentFilter.nothing()
.include("/points", subtree=True)
.exclude("/text/**")
)
# Chain with segment filters
view = dataset.filter_segments(["recording_0"]).filter_contents("/points/**")
filter_segments
def filter_segments(
segment_ids: str | Sequence[str] | DataFrame,
) -> DatasetView
Return a new DatasetView filtered to the given segment IDs.
| PARAMETER | DESCRIPTION |
|---|---|
segment_ids
|
A segment ID string, a list of segment ID strings, or a DataFusion DataFrame with a column named 'rerun_segment_id'. When passing a DataFrame, if there are additional columns, they will be ignored. |
| RETURNS | DESCRIPTION |
|---|---|
DatasetView
|
A new view filtered to the given segments. |
Examples:
# Filter to a single segment
view = dataset.filter_segments("recording_0")
# Filter to specific segments
view = dataset.filter_segments(["recording_0", "recording_1"])
# Filter using a DataFrame
good_segments = segment_table.filter(col("success"))
view = dataset.filter_segments(good_segments)
# Read data from the filtered view
df = view.reader(index="timeline")
get_index_ranges
def get_index_ranges() -> DataFrame
Returns the range bounds of all indexes per segment.
manifest
deprecated
def manifest(
include_diagnostic_data: bool = False,
) -> DataFrame
Deprecated
DatasetEntry.manifest() is deprecated and will be removed in a future release. It was intended for internal and debugging use only.
Return the dataset manifest as a DataFusion DataFrame.
| PARAMETER | DESCRIPTION |
|---|---|
include_diagnostic_data
|
Include diagnostic data in the manifest. That may include rows that correspond to layers which failed registration, were deleted, or are in pending states. Note Diagnostic data is subject to change in any release and should not be relied on for production.
TYPE:
|
reader
def reader(
index: str | None,
*,
include_semantically_empty_columns: bool = False,
include_tombstone_columns: bool = False,
fill_latest_at: bool = False,
using_index_values: dict[str, IndexValuesLike]
| DataFrame
| IndexValuesLike
| None = None,
) -> DataFrame
Create a reader over this dataset.
Returns a DataFusion DataFrame.
Server side filters
The returned DataFrame supports server side filtering for both rerun_segment_id
and the index (timeline) column, which can greatly improve performance. For
example, the following filters will effectively be handled by the Rerun server.
dataset.reader(index="real_time").filter(col("rerun_segment_id") == "aabbccddee")
dataset.reader(index="real_time").filter(col("real_time") == "1234567890")
dataset.reader(index="real_time").filter(
(col("rerun_segment_id") == "aabbccddee") & (col("real_time") == "1234567890")
)
| PARAMETER | DESCRIPTION |
|---|---|
index
|
The index (timeline) to use for the view.
Pass
TYPE:
|
include_semantically_empty_columns
|
Whether to include columns that are semantically empty.
TYPE:
|
include_tombstone_columns
|
Whether to include tombstone columns.
TYPE:
|
fill_latest_at
|
Whether to fill null values with the latest valid data.
TYPE:
|
using_index_values
|
Index values at which to resample data. When specified, this argument changes the way rows are returned. Instead
of returning the rows that exist in the data, one row is returned per
Don't use this argument for plain index slicing — use a DataFusion filter on the index column instead. For example: This argument accepts the following shapes:
- plain array: values are applied only to segments whose index
range covers them (segments outside the range are excluded).
- dict: keys are segment IDs, values are per-segment index values
to sample at.
- DataFrame: must have Note The plain array form requires a scan of the segment table to map values to the segments whose index range covers them. On datasets with many segments this can be expensive. Prefer the dict or DataFrame form when the per-segment values are already known on the client side. Note Unknown segment IDs are silently ignored — they contribute no rows to the result. Validate client-side if you need to catch unknown segment IDs.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
A DataFusion DataFrame. |
register
def register(
recording_uri: list[str],
*,
layer_name: str | Sequence[str] = "base",
on_duplicate: OnDuplicateSegmentLayer = ERROR,
) -> RegistrationHandle
Register RRD URIs to the dataset and return a handle to track progress.
This method initiates the registration of recordings to the dataset, and returns a handle that can be used to wait for completion or iterate over results.
Prefer batching many URIs into a single register call rather than calling
register repeatedly in a loop, which is much slower.
| PARAMETER | DESCRIPTION |
|---|---|
recording_uri
|
The URIs of the RRDs to register, as a sequence of strings. |
layer_name
|
The layer(s) to which the recordings will be registered to.
Can be a single layer name (applied to all recordings) or a sequence of layer names
(must match the length of |
on_duplicate
|
How to handle the cases where the segment id and layer name already exist in the dataset?
Defaults to
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
RegistrationHandle
|
A handle to track and wait on the registration tasks. |
register_asset
Register an existing .rrd visible to the server as an asset.
Asset datasets hold a small set of static blobs shared across a dataset's segments, so they are kept deliberately small. The server enforces a few limits on the .rrd you register:
- it must contain only static data, temporal chunks are rejected,
- each asset segment must stay under a per-segment size limit,
- the asset dataset may only hold a limited number of segments.
| PARAMETER | DESCRIPTION |
|---|---|
uri
|
The URI of the .rrd file to register. It must be visible to the server.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str
|
The segment id of the registered asset. |
register_blueprint
def register_blueprint(
uri: str,
set_default: bool = True,
*,
segment_table: bool = False,
) -> None
Register an existing .rbl visible to the server.
By default, also set this blueprint as default.
Set segment_table=True (and set_default=True) to register it as this dataset's
default for the segment table blueprint.
The associated blueprint dataset is owned by this dataset for lifecycle purposes. Deleting this dataset also deletes the associated blueprint dataset and its storage.
register_prefix
def register_prefix(
recordings_prefix: str,
layer_name: str | None = None,
on_duplicate: OnDuplicateSegmentLayer = ERROR,
) -> RegistrationHandle
Register all RRDs under a given prefix to the dataset and return a handle to track progress.
A prefix is a directory-like path in an object store (e.g. an S3 bucket or ABS container). All RRDs that are recursively found under the given prefix will be registered to the dataset.
This method initiates the registration of the recordings to the dataset, and returns a handle that can be used to wait for completion or iterate over results.
| PARAMETER | DESCRIPTION |
|---|---|
recordings_prefix
|
The prefix under which to register all RRDs.
TYPE:
|
layer_name
|
The layer to which the recordings will be registered to.
If
TYPE:
|
on_duplicate
|
How to handle the cases where the segment id and layer name already exist in the dataset?
Defaults to
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
A handle to track and wait on the registration tasks.
|
|
segment_store
Open a remote segment as a LazyStore.
The manifest is fetched immediately; chunk data is loaded on demand
via LazyStore.stream. To fully
materialize into a ChunkStore, call
lazy.stream().collect().
segment_table
def segment_table(
join_meta: TableEntry | DataFrame | None = None,
join_key: str = "rerun_segment_id",
) -> DataFrame
Return the segment table as a DataFusion DataFrame.
The segment table contains metadata about each segment in the dataset, including segment IDs, layer names, storage URLs, and size information.
| PARAMETER | DESCRIPTION |
|---|---|
join_meta
|
Optional metadata table or DataFrame to join with the segment table.
If a
TYPE:
|
join_key
|
The column name to use for joining, defaults to "rerun_segment_id".
Both the segment table and
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
The segment metadata table, optionally joined with |
segment_url
def segment_url(
segment_id: str,
timeline: str | None = None,
start: datetime | timedelta | int | None = None,
end: datetime | timedelta | int | None = None,
) -> str
Return the URL for the given segment.
| PARAMETER | DESCRIPTION |
|---|---|
segment_id
|
The ID of the segment to get the URL for.
TYPE:
|
timeline
|
The name of the timeline to display.
TYPE:
|
start
|
The start selected time for the segment. Integer for ticks, datetime/nanoseconds for timestamps, or timedelta for durations. |
end
|
The end selected time for the segment.
Integer for ticks, datetime/nanoseconds for timestamps, or timedelta for durations.
If omitted, no time range selection is emitted (only the |
Examples:
With ticks
>>> start_tick, end_time = 0, 10
>>> dataset.segment_url("some_id", "log_tick", start_tick, end_time)
With timestamps
>>> start_time, end_time = datetime.now() - timedelta(seconds=4), datetime.now()
>>> dataset.segment_url("some_id", "real_time", start_time, end_time)
| RETURNS | DESCRIPTION |
|---|---|
str
|
The URL for the given segment. |
set_default_blueprint
def set_default_blueprint(blueprint_name: str | None) -> None
Set an already-registered blueprint as default for this dataset.
set_default_segment_table_blueprint
def set_default_segment_table_blueprint(
blueprint_name: str | None,
) -> None
Set an already-registered blueprint as the default segment table blueprint for this dataset.
set_name
def set_name(name: str) -> None
Change the name of this entry.
Note: entry names must be unique within the catalog. If the new name is not unique, an error will be raised.
Entry names may only contain ASCII alphanumeric characters, underscores, hyphens, dots, colons and spaces, and must be at most 180 characters long.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
New name for the entry
TYPE:
|
unregister
def unregister(
*,
segments_to_drop: str | Sequence[str],
layers_to_drop: str | Sequence[str],
force: bool = False,
) -> UnregistrationHandle
Unregisters segments and layers from the dataset.
Excluding IO errors, this will always succeed as long the target dataset exists. Corollary: unregistering data that doesn't exist is a no-op.
This method acts as a product filter:
* empty segments_to_drop + empty layers_to_drop: invalid argument error
* empty segments_to_drop + non-empty layers_to_drop: remove specified layers for all segments
* non-empty segments_to_drop + empty layers_to_drop: remove all layers for specified segments
* non-empty segments_to_drop + non-empty layers_to_drop: delete all specified layers for all specified segments
| PARAMETER | DESCRIPTION |
|---|---|
segments_to_drop
|
The segment IDs to drop. All of them if empty.
The final filter will be the outer product of this and |
layers_to_drop
|
The layer names to drop. All of them if empty.
The final filter will be the outer product of this and |
force
|
If true, deletion will go through regardless of the segments/layers' current statuses.
This is only useful in the very specific, catatrophic scenario where the contents of the
task queue were lost and some tasks are now stuck in
TYPE:
|
unregister_asset
def unregister_asset(segment_id: str) -> None
Unregister a previously registered asset.
Since assets are shared across all of a dataset's segments, there is no way to scope an asset to a subset of them, so removing one means unregistering it here.
Unregistering an asset that doesn't exist is a no-op.
| PARAMETER | DESCRIPTION |
|---|---|
segment_id
|
The segment id of the asset to unregister, as returned by
TYPE:
|
DatasetView
A filtered view over a dataset in the catalog.
A DatasetView provides lazy filtering over a dataset's segments and entity paths.
Filters are composed lazily and only applied when data is actually read.
Create a DatasetView by calling filter_segments() or filter_contents() on a
DatasetEntry.
Examples:
# Filter to specific segments
view = dataset.filter_segments(["recording_0", "recording_1"])
# Filter to specific entity paths
view = dataset.filter_contents(["/points/**"])
# Chain filters
view = dataset.filter_segments(["recording_0"]).filter_contents(["/points/**"])
# Read data
df = view.reader(index="timeline")
__init__
def __init__(internal: DatasetViewInternal) -> None
Create a new DatasetView wrapper.
| PARAMETER | DESCRIPTION |
|---|---|
internal
|
The internal Rust-side DatasetView object.
TYPE:
|
arrow_schema
def arrow_schema() -> Schema
Return the filtered Arrow schema for this view.
| RETURNS | DESCRIPTION |
|---|---|
Schema
|
The filtered Arrow schema. |
filter_contents
def filter_contents(
exprs: ContentFilter | str | Sequence[str],
) -> DatasetView
Return a new DatasetView filtered to the given entity paths.
Entity path expressions support wildcards:
- "/points/**" matches all entities under /points
- "-/text/**" excludes all entities under /text
| PARAMETER | DESCRIPTION |
|---|---|
exprs
|
A
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
DatasetView
|
A new view filtered to the matching entity paths. |
Examples:
# Filter to a single entity path
view = dataset.filter_contents("/points/**")
# Filter to specific entity paths
view = dataset.filter_contents(["/points/**"])
# Exclude certain paths
view = dataset.filter_contents(["/points/**", "-/text/**"])
# Using ContentFilter builder
view = dataset.filter_contents(
ContentFilter.everything()
.exclude("/robot/raw/**")
.include("/robot/raw/i_need_this")
)
# Chain with segment filters
view = dataset.filter_segments(["recording_0"]).filter_contents("/points/**")
filter_segments
def filter_segments(
segment_ids: str | Sequence[str] | DataFrame,
) -> DatasetView
Return a new DatasetView filtered to the given segment IDs.
| PARAMETER | DESCRIPTION |
|---|---|
segment_ids
|
A segment ID string, a list of segment ID strings, or a DataFusion DataFrame with a column named 'rerun_segment_id'. When passing a DataFrame, if there are additional columns, they will be ignored. |
| RETURNS | DESCRIPTION |
|---|---|
DatasetView
|
A new view filtered to the given segments. |
Examples:
# Filter to a single segment
view = dataset.filter_segments("recording_0")
# Filter to specific segments
view = dataset.filter_segments(["recording_0", "recording_1"])
# Filter using a DataFrame
good_segments = segment_table.filter(col("success"))
view = dataset.filter_segments(good_segments)
# Read data from the filtered view
df = view.reader(index="timeline")
get_index_ranges
def get_index_ranges() -> DataFrame
Returns the range bounds of all indexes per segment.
reader
def reader(
index: str | None,
*,
include_semantically_empty_columns: bool = False,
include_tombstone_columns: bool = False,
using_index_values: dict[str, IndexValuesLike]
| DataFrame
| IndexValuesLike
| None = None,
fill_latest_at: bool = False,
) -> DataFrame
Create a reader over this DatasetView.
Returns a DataFusion DataFrame.
Server side filters
The returned DataFrame supports server side filtering for both rerun_segment_id
and the index (timeline) column, which can greatly improve performance. For
example, the following filters will effectively be handled by the Rerun server.
dataset.reader(index="real_time").filter(col("rerun_segment_id") == "aabbccddee")
dataset.reader(index="real_time").filter(col("real_time") == "1234567890")
dataset.reader(index="real_time").filter(
(col("rerun_segment_id") == "aabbccddee") & (col("real_time") == "1234567890")
)
| PARAMETER | DESCRIPTION |
|---|---|
index
|
The index (timeline) to use for the view.
Pass
TYPE:
|
include_semantically_empty_columns
|
Whether to include columns that are semantically empty.
TYPE:
|
include_tombstone_columns
|
Whether to include tombstone columns.
TYPE:
|
using_index_values
|
Index values at which to resample data. When specified, this argument changes the way rows are returned. Instead
of returning the rows that exist in the data, one row is returned per
Don't use this argument for plain index slicing — use a DataFusion filter on the index column instead. For example: This argument accepts the following shapes:
- plain array: values are applied only to segments whose index
range covers them (segments outside the range are excluded).
- dict: keys are segment IDs, values are per-segment index values
to sample at.
- DataFrame: must have Note The plain array form requires a scan of the segment table to map values to the segments whose index range covers them. On datasets with many segments this can be expensive. Prefer the dict or DataFrame form when the per-segment values are already known on the client side. Note Unknown segment IDs are silently ignored — they contribute no rows to the result. Validate client-side if you need to catch unknown segment IDs.
TYPE:
|
fill_latest_at
|
Whether to fill null values with the latest valid data.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
A DataFusion DataFrame.
|
|
schema
def schema() -> Schema
Return the filtered schema for this view.
The schema reflects any content filters applied to the view.
| RETURNS | DESCRIPTION |
|---|---|
Schema
|
The filtered schema. |
segment_ids
segment_table
def segment_table(
join_meta: TableEntry | DataFrame | None = None,
join_key: str = "rerun_segment_id",
) -> DataFrame
Return the segment table as a DataFusion DataFrame.
The segment table contains metadata about each segment in the dataset, including segment IDs, layer names, storage URLs, and size information.
Only segments matching this view's filters are included.
| PARAMETER | DESCRIPTION |
|---|---|
join_meta
|
Optional metadata table or DataFrame to join with the segment table.
If a
TYPE:
|
join_key
|
The column name to use for joining, defaults to "rerun_segment_id".
Both the segment table and
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
DataFrame
|
The segment metadata table, optionally joined with |
Entry
Bases: ABC, Generic[InternalEntryT]
An entry in the catalog.
__eq__
Compare this entry to another object.
Supports comparison with str and EntryId to enable the following patterns:
"entry_name" in client.entries()
entry_id in client.entries()
delete
def delete() -> None
Delete this entry from the catalog.
set_name
def set_name(name: str) -> None
Change the name of this entry.
Note: entry names must be unique within the catalog. If the new name is not unique, an error will be raised.
Entry names may only contain ASCII alphanumeric characters, underscores, hyphens, dots, colons and spaces, and must be at most 180 characters long.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
New name for the entry
TYPE:
|
EntryId
EntryKind
IndexColumnDescriptor
The descriptor of an index column.
Index columns contain the index values for when the data was updated. They generally correspond to Rerun timelines.
Column descriptors are used to describe the columns in a
Schema. They are read-only. To select an index
column, use IndexColumnSelector.
IndexColumnSelector
A selector for an index column.
Index columns contain the index values for when the data was updated. They generally correspond to Rerun timelines.
NotFoundError
Bases: Exception
Raised when the requested resource is not found.
OnDuplicateSegmentLayer
RegistrationHandle
Handle to track and wait on segment registration tasks.
cancel
def cancel() -> None
Cancel dataset registration. If the registration is already done, this is a noop.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the cancellation fails. |
iter_results
def iter_results(
timeout_secs: int | None = None,
) -> Iterator[SegmentRegistrationResult]
Stream completed registrations as they finish.
Uses the server's streaming API to yield results as tasks complete. Each result is yielded exactly once when its task completes (success or error).
| PARAMETER | DESCRIPTION |
|---|---|
timeout_secs
|
Timeout in seconds. None for blocking. Note that using None doesn't guarantee that a TimeoutError will never be eventually raised for long-running tasks. Setting a timeout and polling is recommended for monitoring very large registration batches.
TYPE:
|
| YIELDS | DESCRIPTION |
|---|---|
SegmentRegistrationResult
|
The result of each completed registration. |
| RAISES | DESCRIPTION |
|---|---|
TimeoutError
|
If the timeout is reached before all tasks complete. |
wait
def wait(timeout_secs: int | None = None) -> RegistrationResult
Block until all registrations complete.
| PARAMETER | DESCRIPTION |
|---|---|
timeout_secs
|
Timeout in seconds. None for blocking. Note that using None doesn't guarantee that a TimeoutError will never be eventually raised for long-running tasks. Setting a timeout and polling is recommended for monitoring very large registration batches.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
RegistrationResult
|
The result containing the list of segment IDs in registration order. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If any registration fails. |
TimeoutError
|
If the timeout is reached before all tasks complete. |
RegistrationResult
dataclass
Schema
The schema representing a set of available columns for a dataset.
A schema contains both index columns (timelines) and component columns (entity/component data).
__init__
def __init__(inner: SchemaInternal) -> None
Create a new Schema wrapper.
| PARAMETER | DESCRIPTION |
|---|---|
inner
|
The internal schema object from the bindings.
TYPE:
|
__iter__
def __iter__() -> Iterator[
IndexColumnDescriptor | ComponentColumnDescriptor
]
Iterate over all column descriptors in the schema (index columns first, then component columns).
archetypes
Return a list of all the archetypes in the schema.
| PARAMETER | DESCRIPTION |
|---|---|
include_properties
|
If
TYPE:
|
column_for
def column_for(
entity_path: str, component: str
) -> ComponentColumnDescriptor | None
Look up the column descriptor for a specific entity path and component.
| PARAMETER | DESCRIPTION |
|---|---|
entity_path
|
The entity path to look up.
TYPE:
|
component
|
The component to look up. Example:
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ComponentColumnDescriptor | None
|
The column descriptor, if it exists. |
column_for_selector
def column_for_selector(
selector: str
| ComponentColumnSelector
| ComponentColumnDescriptor,
) -> ComponentColumnDescriptor
Look up the column descriptor for a specific selector.
| PARAMETER | DESCRIPTION |
|---|---|
selector
|
The selector to look up. String arguments are expected to follow the format:
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ComponentColumnDescriptor
|
The column descriptor. |
| RAISES | DESCRIPTION |
|---|---|
LookupError
|
If the column is not found. |
ValueError
|
If the string selector format is invalid or the input type is unsupported. |
Note: if the input is already a `ComponentColumnDescriptor`, it is
|
|
returned directly without checking for existence.
|
|
column_names
Return a list of all column names in the schema.
| RETURNS | DESCRIPTION |
|---|---|
The names of all columns (index columns first, then component columns).
|
|
column_names_for
def column_names_for(
*,
entity_path: str | None = None,
archetype: str | type[Archetype] | None = None,
component_type: str | None = None,
include_properties: bool = False,
) -> list[str]
Return a list of column names matching the given criteria.
| PARAMETER | DESCRIPTION |
|---|---|
entity_path
|
If set, only return columns with this entity path.
TYPE:
|
archetype
|
If set, only return columns with this archetype. Accepts a fully-qualified
archetype name (e.g., |
component_type
|
If set, only return columns with this component type.
TYPE:
|
include_properties
|
If
TYPE:
|
columns_for
def columns_for(
*,
entity_path: str | None = None,
archetype: str | type[Archetype] | None = None,
component_type: str | None = None,
include_properties: bool = False,
) -> list[ComponentColumnDescriptor]
Return a filtered list of component columns matching the given criteria.
| PARAMETER | DESCRIPTION |
|---|---|
entity_path
|
If set, only return columns with this entity path.
TYPE:
|
archetype
|
If set, only return columns with this archetype. Accepts a fully-qualified
archetype name (e.g., |
component_type
|
If set, only return columns with this component type.
TYPE:
|
include_properties
|
If
TYPE:
|
component_columns
def component_columns() -> Sequence[ComponentColumnDescriptor]
Return a list of all the component columns in the schema.
Component columns contain the data for a specific component of an entity.
component_types
Return a list of all the component types in the schema.
| PARAMETER | DESCRIPTION |
|---|---|
include_properties
|
If
TYPE:
|
entity_paths
Return a sorted list of all unique entity paths in the schema. By default, the properties are not included.
| PARAMETER | DESCRIPTION |
|---|---|
include_properties
|
If
TYPE:
|
index_columns
def index_columns() -> Sequence[IndexColumnDescriptor]
Return a list of all the index columns in the schema.
Index columns contain the index values for when the data was updated. They generally correspond to Rerun timelines.
SegmentRegistrationResult
dataclass
Result of a completed segment registration.
error
instance-attribute
error: str | None
Error message if registration failed, or None if successful.
TableEntry
Bases: Entry[TableEntryInternal]
A table entry in the catalog.
Note: this object acts as a table provider for DataFusion.
__datafusion_table_provider__
Returns a DataFusion table provider capsule.
__eq__
Compare this entry to another object.
Supports comparison with str and EntryId to enable the following patterns:
"entry_name" in client.entries()
entry_id in client.entries()
append
def append(
batches: _BatchesType | None = None, **named_params: Any
) -> None
Append to the Table.
| PARAMETER | DESCRIPTION |
|---|---|
batches
|
Arrow data to append to the table. Can be a RecordBatchReader, a single RecordBatch, a list of
RecordBatches, or a list of lists of RecordBatches (as returned by
TYPE:
|
**named_params
|
Each named parameter corresponds to a column in the table.
TYPE:
|
blueprint_dataset
def blueprint_dataset() -> DatasetEntry
The associated blueprint dataset.
Tables get a blueprint dataset automatically when they are created. For tables created by older servers, this creates the missing blueprint dataset before returning.
The associated blueprint dataset is owned by this table for lifecycle purposes. Deleting this table also deletes the associated blueprint dataset and its storage.
Note
⚠️ This API is experimental and may change or be removed in future versions! ⚠️ TODO(#12746): Stabilize table blueprint APIs.
blueprints
Lists all blueprints currently registered with this table.
Note
⚠️ This API is experimental and may change or be removed in future versions! ⚠️ TODO(#12746): Stabilize table blueprint APIs.
default_blueprint
def default_blueprint() -> str | None
Return the name currently set blueprint.
Note
⚠️ This API is experimental and may change or be removed in future versions! ⚠️ TODO(#12746): Stabilize table blueprint APIs.
delete
def delete() -> None
Delete this entry from the catalog.
overwrite
def overwrite(
batches: _BatchesType | None = None, **named_params: Any
) -> None
Overwrite the Table with new data.
| PARAMETER | DESCRIPTION |
|---|---|
batches
|
Arrow data to overwrite the table with. Can be a RecordBatchReader, a single RecordBatch, a list of
RecordBatches, or a list of lists of RecordBatches (as returned by
TYPE:
|
**named_params
|
Each named parameter corresponds to a column in the table.
TYPE:
|
reader
def reader() -> DataFrame
Registers the table with the DataFusion context and return a DataFrame.
register_blueprint
Register an existing .rbl visible to the server as this table's blueprint.
By default, also set this blueprint as default.
The associated blueprint dataset is owned by this table for lifecycle purposes. Deleting this table also deletes the associated blueprint dataset and its storage.
Note
⚠️ This API is experimental and may change or be removed in future versions! ⚠️ TODO(#12746): Stabilize table blueprint APIs.
set_default_blueprint
def set_default_blueprint(blueprint_name: str | None) -> None
Set an already-registered blueprint as default for this table.
Note
⚠️ This API is experimental and may change or be removed in future versions! ⚠️ TODO(#12746): Stabilize table blueprint APIs.
set_name
def set_name(name: str) -> None
Change the name of this entry.
Note: entry names must be unique within the catalog. If the new name is not unique, an error will be raised.
Entry names may only contain ASCII alphanumeric characters, underscores, hyphens, dots, colons and spaces, and must be at most 180 characters long.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
New name for the entry
TYPE:
|
to_arrow_reader
def to_arrow_reader() -> RecordBatchReader
Convert this table to a pyarrow.RecordBatchReader.
update
deprecated
def update(*, name: str | None = None) -> None
Deprecated
Entry.update() is deprecated. Use Entry.set_name() instead.
Update this entry's properties.
| PARAMETER | DESCRIPTION |
|---|---|
name
|
New name for the entry
TYPE:
|
upsert
def upsert(
batches: _BatchesType | None = None, **named_params: Any
) -> None
Upsert data into the Table.
To use upsert, the table must contain a column with the metadata:
{"rerun:is_table_index" = "true"}
Any row with a matching index value will have the new data inserted. Any row without a matching index value will be appended as a new row.
| PARAMETER | DESCRIPTION |
|---|---|
batches
|
Arrow data to upsert into the table. Can be a RecordBatchReader, a single RecordBatch, a list of
RecordBatches, or a list of lists of RecordBatches (as returned by
TYPE:
|
**named_params
|
Each named parameter corresponds to a column in the table
TYPE:
|
UnregistrationHandle
Handle to track and wait on segment unregistration tasks.
cancel
def cancel() -> None
Cancel unrregistration. If the unregistration is already done, this is a noop.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the cancellation fails. |
wait
def wait(timeout_secs: int | None = None) -> None
Block until the unregistriation completes.
| PARAMETER | DESCRIPTION |
|---|---|
timeout_secs
|
Timeout in seconds. None for blocking. Note that using None doesn't guarantee that a TimeoutError will never be eventually raised for long-running tasks.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the uregistration fails. |
TimeoutError
|
If the timeout is reached before all tasks complete. |
VersionInfo
dataclass
Version and deployment information from a Rerun server.