Struct re_chunk::TransportChunk

pub struct TransportChunk {
    pub schema: Schema,
    pub data: Chunk<Box<dyn ArrowArray>>,
}

A Chunk that is ready for transport. Obtained by calling Chunk::to_transport.

Implemented as an Arrow dataframe: a schema and a batch.

Use the Display implementation to dump the chunk as a nicely formatted table.

This has a stable ABI! The entire point of this type is to allow users to send raw arrow data into Rerun. This means we have to be very careful when checking the validity of the data: slipping corrupt data into the store could silently break all the index search logic (e.g. think of a chunk claiming to be sorted while it is in fact not).

Fields

schema: Schema

The schema of the dataframe, and all chunk-level and field-level metadata.

Take a look at the TransportChunk::CHUNK_METADATA_* and TransportChunk::FIELD_METADATA_* constants for more information about available metadata.

data: Chunk<Box<dyn ArrowArray>>

All the control, time and component data.

Implementations

impl TransportChunk

pub const CHUNK_METADATA_KEY_ID: &'static str = "rerun.id"

The key used to identify a Rerun ChunkId in chunk-level [ArrowSchema] metadata.

pub const CHUNK_METADATA_KEY_ENTITY_PATH: &'static str = "rerun.entity_path"

The key used to identify a Rerun [EntityPath] in chunk-level [ArrowSchema] metadata.

pub const CHUNK_METADATA_KEY_HEAP_SIZE_BYTES: &'static str = "rerun.heap_size_bytes"

The key used to identify the size in bytes of the data, once loaded in memory, in chunk-level [ArrowSchema] metadata.

pub const CHUNK_METADATA_MARKER_IS_SORTED_BY_ROW_ID: &'static str = "rerun.is_sorted"

The marker used to identify whether a chunk is sorted in chunk-level [ArrowSchema] metadata.

The associated value is irrelevant – if this marker is present, then it is true.

Chunks are ascendingly sorted by their RowId column.

pub const FIELD_METADATA_KEY_KIND: &'static str = "rerun.kind"

The key used to identify the kind of a Rerun column in field-level [ArrowSchema] metadata.

That is: control columns (e.g. row_id), time columns or component columns.

pub const FIELD_METADATA_VALUE_KIND_TIME: &'static str = "time"

The value used to identify a Rerun time column in field-level [ArrowSchema] metadata.

pub const FIELD_METADATA_VALUE_KIND_CONTROL: &'static str = "control"

The value used to identify a Rerun control column in field-level [ArrowSchema] metadata.

pub const FIELD_METADATA_VALUE_KIND_DATA: &'static str = "data"

The value used to identify a Rerun data column in field-level [ArrowSchema] metadata.

pub const FIELD_METADATA_MARKER_IS_SORTED_BY_TIME: &'static str = Self::CHUNK_METADATA_MARKER_IS_SORTED_BY_ROW_ID

The marker used to identify whether a column is sorted in field-level [ArrowSchema] metadata.

The associated value is irrelevant – if this marker is present, then it is true.

Chunks are ascendingly sorted by their RowId column but, depending on whether the data was logged out of order or not for a given time column, that column might follow the global RowId while still being unsorted relative to its own time order.

pub fn chunk_metadata_id(id: ChunkId) -> ArrowMetadata

Returns the appropriate chunk-level [ArrowSchema] metadata for a Rerun ChunkId.

pub fn chunk_metadata_heap_size_bytes(heap_size_bytes: u64) -> ArrowMetadata

Returns the appropriate chunk-level [ArrowSchema] metadata for the in-memory size in bytes.

pub fn chunk_metadata_entity_path(entity_path: &EntityPath) -> ArrowMetadata

Returns the appropriate chunk-level [ArrowSchema] metadata for a Rerun [EntityPath].

pub fn chunk_metadata_is_sorted() -> ArrowMetadata

Returns the appropriate chunk-level [ArrowSchema] metadata for an IS_SORTED marker.

pub fn field_metadata_time_column() -> ArrowMetadata

Returns the appropriate field-level [ArrowSchema] metadata for a Rerun time column.

pub fn field_metadata_control_column() -> ArrowMetadata

Returns the appropriate field-level [ArrowSchema] metadata for a Rerun control column.

pub fn field_metadata_data_column() -> ArrowMetadata

Returns the appropriate field-level [ArrowSchema] metadata for a Rerun data column.

pub fn field_metadata_is_sorted() -> ArrowMetadata

Returns the appropriate field-level [ArrowSchema] metadata for an IS_SORTED marker.

impl TransportChunk

pub fn id(&self) -> ChunkResult<ChunkId>

pub fn entity_path(&self) -> ChunkResult<EntityPath>

pub fn heap_size_bytes(&self) -> Option<u64>

pub fn is_sorted(&self) -> bool

Looks in the chunk metadata for the IS_SORTED marker.

It is possible that a chunk is sorted but didn’t set that marker. This is fine, although wasteful.

pub fn columns<'a>( &'a self, kind: &'a str ) -> impl Iterator<Item = (&ArrowField, &'a Box<dyn ArrowArray>)> + 'a

Iterates all columns of the specified kind.

See:

• Self::FIELD_METADATA_VALUE_KIND_TIME
• Self::FIELD_METADATA_VALUE_KIND_CONTROL
• Self::FIELD_METADATA_VALUE_KIND_DATA

pub fn all_columns( &self ) -> impl Iterator<Item = (&ArrowField, &Box<dyn ArrowArray>)> + '_

pub fn controls( &self ) -> impl Iterator<Item = (&ArrowField, &Box<dyn ArrowArray>)>

Iterates all control columns present in this chunk.

pub fn components( &self ) -> impl Iterator<Item = (&ArrowField, &Box<dyn ArrowArray>)>

Iterates all data columns present in this chunk.

pub fn timelines( &self ) -> impl Iterator<Item = (&ArrowField, &Box<dyn ArrowArray>)>

Iterates all timeline columns present in this chunk.

pub fn num_columns(&self) -> usize

How many columns in total? Includes control, time, and component columns.

pub fn num_controls(&self) -> usize

pub fn num_timelines(&self) -> usize

pub fn num_components(&self) -> usize

pub fn num_rows(&self) -> usize

impl TransportChunk

pub fn try_to_arrow_record_batch(&self) -> Result<RecordBatch, ArrowError>

Create an arrow-rs [RecordBatch] containing the data from this TransportChunk.

This is a “fairly” cheap operation, as it does not copy the underlying arrow data, but does incur overhead of generating an alternative representation of the arrow- related rust structures that refer to those data buffers.

pub fn from_arrow_record_batch(batch: &RecordBatch) -> Self

Create a TransportChunk from an arrow-rs [RecordBatch].

This is a “fairly” cheap operation, as it does not copy the underlying arrow data, but does incur overhead of generating an alternative representation of the arrow- related rust structures that refer to those data buffers.

Trait Implementations

impl Clone for TransportChunk

fn clone(&self) -> TransportChunk

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TransportChunk

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TransportChunk

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.