Struct rerun::dataframe::external::re_chunk::TransportChunk

pub struct TransportChunk {
    batch: RecordBatch,
}

A Chunk that is ready for transport. Obtained by calling Chunk::to_transport.

It contains a schema with a matching number of columns, all of the same length.

This is just a wrapper around an ArrowRecordBatch, with some helper functions on top. It can be converted to and from ArrowRecordBatch without overhead.

Use the Display implementation to dump the chunk as a nicely formatted table.

This has a stable ABI! The entire point of this type is to allow users to send raw arrow data into Rerun. This means we have to be very careful when checking the validity of the data: slipping corrupt data into the store could silently break all the index search logic (e.g. think of a chunk claiming to be sorted while it is in fact not).

Fields

batch: RecordBatch

Implementations

impl TransportChunk

pub const CHUNK_METADATA_KEY_ID: &'static str = "rerun.id"

The key used to identify a Rerun ChunkId in chunk-level ArrowSchema metadata.

pub const CHUNK_METADATA_KEY_ENTITY_PATH: &'static str = "rerun.entity_path"

The key used to identify a Rerun EntityPath in chunk-level ArrowSchema metadata.

pub const CHUNK_METADATA_KEY_HEAP_SIZE_BYTES: &'static str = "rerun.heap_size_bytes"

The key used to identify the size in bytes of the data, once loaded in memory, in chunk-level ArrowSchema metadata.

pub const CHUNK_METADATA_MARKER_IS_SORTED_BY_ROW_ID: &'static str = "rerun.is_sorted"

The marker used to identify whether a chunk is sorted in chunk-level ArrowSchema metadata.

The associated value is irrelevant – if this marker is present, then it is true.

Chunks are ascendingly sorted by their RowId column.

pub const FIELD_METADATA_KEY_KIND: &'static str = "rerun.kind"

The key used to identify the kind of a Rerun column in field-level ArrowSchema metadata.

That is: control columns (e.g. row_id), time columns or component columns.

pub const FIELD_METADATA_VALUE_KIND_TIME: &'static str = "time"

The value used to identify a Rerun time column in field-level ArrowSchema metadata.

pub const FIELD_METADATA_VALUE_KIND_CONTROL: &'static str = "control"

The value used to identify a Rerun control column in field-level ArrowSchema metadata.

pub const FIELD_METADATA_VALUE_KIND_DATA: &'static str = "data"

The value used to identify a Rerun data column in field-level ArrowSchema metadata.

pub const FIELD_METADATA_KEY_ARCHETYPE_NAME: &'static str = "rerun.archetype_name"

The key used to identify the crate::ArchetypeName in field-level ArrowSchema metadata.

pub const FIELD_METADATA_KEY_ARCHETYPE_FIELD_NAME: &'static str = "rerun.archetype_field_name"

The key used to identify the crate::ArchetypeFieldName in field-level ArrowSchema metadata.

pub const FIELD_METADATA_MARKER_IS_SORTED_BY_TIME: &'static str = Self::CHUNK_METADATA_MARKER_IS_SORTED_BY_ROW_ID

The marker used to identify whether a column is sorted in field-level ArrowSchema metadata.

The associated value is irrelevant – if this marker is present, then it is true.

Chunks are ascendingly sorted by their RowId column but, depending on whether the data was logged out of order or not for a given time column, that column might follow the global RowId while still being unsorted relative to its own time order.

pub fn chunk_metadata_id(id: ChunkId) -> HashMap<String, String>

Returns the appropriate chunk-level ArrowSchema metadata for a Rerun ChunkId.

pub fn chunk_metadata_heap_size_bytes( heap_size_bytes: u64, ) -> HashMap<String, String>

Returns the appropriate chunk-level ArrowSchema metadata for the in-memory size in bytes.

pub fn chunk_metadata_entity_path( entity_path: &EntityPath, ) -> HashMap<String, String>

Returns the appropriate chunk-level ArrowSchema metadata for a Rerun EntityPath.

pub fn chunk_metadata_is_sorted() -> HashMap<String, String>

Returns the appropriate chunk-level ArrowSchema metadata for an IS_SORTED marker.

pub fn field_metadata_time_column() -> HashMap<String, String>

Returns the appropriate field-level ArrowSchema metadata for a Rerun time column.

pub fn field_metadata_control_column() -> HashMap<String, String>

Returns the appropriate field-level ArrowSchema metadata for a Rerun control column.

pub fn field_metadata_data_column() -> HashMap<String, String>

Returns the appropriate field-level ArrowSchema metadata for a Rerun data column.

pub fn field_metadata_is_sorted() -> HashMap<String, String>

Returns the appropriate field-level ArrowSchema metadata for an IS_SORTED marker.

pub fn field_metadata_component_descriptor( component_desc: &ComponentDescriptor, ) -> HashMap<String, String>

pub fn component_descriptor_from_field(field: &Field) -> ComponentDescriptor

impl TransportChunk

pub fn id(&self) -> Result<ChunkId, ChunkError>

pub fn entity_path(&self) -> Result<EntityPath, ChunkError>

pub fn heap_size_bytes(&self) -> Option<u64>

The size in bytes of the data, once loaded in memory, in chunk-level.

This is stored in the metadata. Returns None if that key is not set.

pub fn fields(&self) -> &Fields

pub fn metadata(&self) -> &HashMap<String, String>

pub fn is_sorted(&self) -> bool

Looks in the chunk metadata for the IS_SORTED marker.

It is possible that a chunk is sorted but didn’t set that marker. This is fine, although wasteful.

pub fn controls(&self) -> impl Iterator<Item = (&Field, &Arc<dyn Array>)>

Iterates all control columns present in this chunk.

pub fn components(&self) -> impl Iterator<Item = (&Field, &Arc<dyn Array>)>

Iterates all data columns present in this chunk.

pub fn timelines(&self) -> impl Iterator<Item = (&Field, &Arc<dyn Array>)>

Iterates all timeline columns present in this chunk.

pub fn num_controls(&self) -> usize

pub fn num_timelines(&self) -> usize

pub fn num_components(&self) -> usize

Methods from Deref<Target = RecordBatch>

pub fn schema(&self) -> Arc<Schema>

Returns the Schema of the record batch.

pub fn schema_ref(&self) -> &Arc<Schema>

Returns a reference to the Schema of the record batch.

pub fn project(&self, indices: &[usize]) -> Result<RecordBatch, ArrowError>

Projects the schema onto the specified columns

pub fn num_columns(&self) -> usize

Returns the number of columns in the record batch.

Example

let id_array = Int32Array::from(vec![1, 2, 3, 4, 5]);
let schema = Schema::new(vec![
    Field::new("id", DataType::Int32, false)
]);

let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(id_array)]).unwrap();

assert_eq!(batch.num_columns(), 1);

pub fn num_rows(&self) -> usize

Returns the number of rows in each column.

Example

let id_array = Int32Array::from(vec![1, 2, 3, 4, 5]);
let schema = Schema::new(vec![
    Field::new("id", DataType::Int32, false)
]);

let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(id_array)]).unwrap();

assert_eq!(batch.num_rows(), 5);

pub fn column(&self, index: usize) -> &Arc<dyn Array>

Get a reference to a column’s array by index.

Panics

Panics if index is outside of 0..num_columns.

pub fn column_by_name(&self, name: &str) -> Option<&Arc<dyn Array>>

Get a reference to a column’s array by name.

pub fn columns(&self) -> &[Arc<dyn Array>]

Get a reference to all columns in the record batch.

pub fn slice(&self, offset: usize, length: usize) -> RecordBatch

Return a new RecordBatch where each column is sliced according to offset and length

Panics

Panics if offset with length is greater than column length.

pub fn get_array_memory_size(&self) -> usize

Returns the total number of bytes of memory occupied physically by this batch.

Note that this does not always correspond to the exact memory usage of a RecordBatch (might overestimate), since multiple columns can share the same buffers or slices thereof, the memory used by the shared buffers might be counted multiple times.

Trait Implementations

impl AsRef<RecordBatch> for TransportChunk

fn as_ref(&self) -> &RecordBatch

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for TransportChunk

fn clone(&self) -> TransportChunk

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TransportChunk

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Deref for TransportChunk

type Target = RecordBatch

The resulting type after dereferencing.

fn deref(&self) -> &RecordBatch

Dereferences the value.

impl Display for TransportChunk

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl From<RecordBatch> for TransportChunk

fn from(batch: RecordBatch) -> TransportChunk

Converts to this type from the input type.

impl From<TransportChunk> for RecordBatch

fn from(chunk: TransportChunk) -> RecordBatch

Converts to this type from the input type.