Struct re_chunk::helpers::UnitChunkShared

pub struct UnitChunkShared(ChunkShared);

A ChunkShared that is guaranteed to always contain a single row’s worth of data.

Tuple Fields

0: ChunkShared

Implementations

impl UnitChunkShared

pub fn into_chunk(self) -> ChunkShared

impl UnitChunkShared

pub fn index(&self, timeline: &Timeline) -> Option<(TimeInt, RowId)>

Returns the index ((TimeInt, RowId) pair) of the single row within, on the given timeline.

Returns the single static index if the chunk is static.

pub fn row_id(&self) -> Option<RowId>

Returns the RowId of the single row within, on the given timeline.

Returns the single static RowId if the chunk is static.

pub fn num_instances(&self, component_name: &ComponentName) -> u64

Returns the number of instances of the single row within for a given component.

impl UnitChunkShared

pub fn component_batch_raw( &self, component_name: &ComponentName ) -> Option<Box<dyn ArrowArray>>

Returns the raw data for the specified component.

pub fn component_batch<C: Component>(&self) -> Option<ChunkResult<Vec<C>>>

Returns the deserialized data for the specified component.

Returns an error if the data cannot be deserialized.

pub fn component_instance_raw( &self, component_name: &ComponentName, instance_index: usize ) -> Option<ChunkResult<Box<dyn ArrowArray>>>

Returns the raw data for the specified component at the given instance index.

Returns an error if the instance index is out of bounds.

pub fn component_instance<C: Component>( &self, instance_index: usize ) -> Option<ChunkResult<C>>

Returns the deserialized data for the specified component at the given instance index.

Returns an error if the data cannot be deserialized, or if the instance index is out of bounds.

pub fn component_mono_raw( &self, component_name: &ComponentName ) -> Option<ChunkResult<Box<dyn ArrowArray>>>

Returns the raw data for the specified component, assuming a mono-batch.

Returns an error if the underlying batch is not of unit length.

pub fn component_mono<C: Component>(&self) -> Option<ChunkResult<C>>

Returns the deserialized data for the specified component, assuming a mono-batch.

Returns an error if the data cannot be deserialized, or if the underlying batch is not of unit length.

Methods from Deref<Target = Chunk>

pub fn are_equal_ignoring_extension_types(&self, other: &Self) -> bool

Check for equality while ignoring possible Extension type information

This is necessary because arrow2 loses the Extension datatype when deserializing back from the arrow_schema::DataType representation.

In theory we could fix this, but as we’re moving away from arrow2 anyways it’s unlikely worth the effort.

pub fn clone_as(&self, id: ChunkId, first_row_id: RowId) -> Self

Clones the chunk and assign new IDs to the resulting chunk and its rows.

first_row_id will become the RowId of the first row in the duplicated chunk. Each row after that will be monotonically increasing.

pub fn time_range_per_component( &self ) -> BTreeMap<Timeline, BTreeMap<ComponentName, ResolvedTimeRange>>

Computes the time range covered by each individual component column on each timeline.

This is different from the time range covered by the Chunk as a whole because component columns are potentially sparse.

This is crucial for indexing and queries to work properly.

pub fn num_events_cumulative(&self) -> u64

The cumulative number of events in this chunk.

I.e. how many component batches (“cells”) were logged in total?

pub fn num_events_cumulative_per_unique_time( &self, timeline: &Timeline ) -> Vec<(TimeInt, u64)>

The cumulative number of events in this chunk for each unique timestamp.

I.e. how many component batches (“cells”) were logged in total at each timestamp?

Keep in mind that a timestamp can appear multiple times in a Chunk. This method will do a sum accumulation to account for these cases (i.e. every timestamp in the returned vector is guaranteed to be unique).

fn num_events_cumulative_per_unique_time_sorted( &self, time_column: &TimeColumn ) -> Vec<(TimeInt, u64)>

fn num_events_cumulative_per_unique_time_unsorted( &self, time_column: &TimeColumn ) -> Vec<(TimeInt, u64)>

pub fn num_events_for_component( &self, component_name: ComponentName ) -> Option<u64>

The number of events in this chunk for the specified component.

I.e. how many component batches (“cells”) were logged in total for this component?

pub fn row_id_range_per_component( &self ) -> BTreeMap<ComponentName, (RowId, RowId)>

Computes the RowId range covered by each individual component column on each timeline.

This is different from the RowId range covered by the Chunk as a whole because component columns are potentially sparse.

This is crucial for indexing and queries to work properly.

pub fn id(&self) -> ChunkId

pub fn entity_path(&self) -> &EntityPath

pub fn num_columns(&self) -> usize

How many columns in total? Includes control, time, and component columns.

pub fn num_controls(&self) -> usize

pub fn num_timelines(&self) -> usize

pub fn num_components(&self) -> usize

pub fn num_rows(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn row_ids_array(&self) -> &ArrowStructArray

pub fn row_ids_raw( &self ) -> (&ArrowPrimitiveArray<u64>, &ArrowPrimitiveArray<u64>)

Returns the RowIds in their raw-est form: a tuple of (times, counters) arrays.

pub fn row_ids(&self) -> impl Iterator<Item = RowId> + '_

All the RowId in this chunk.

This could be in any order if this chunk is unsorted.

pub fn component_row_ids( &self, component_name: &ComponentName ) -> impl Iterator<Item = RowId> + '_

Returns an iterator over the RowIds of a Chunk, for a given component.

This is different than Self::row_ids: it will only yield RowIds for rows at which there is data for the specified component_name.

pub fn row_id_range(&self) -> Option<(RowId, RowId)>

Returns the RowId-range covered by this Chunk.

None if the chunk is_empty.

This is O(1) if the chunk is sorted, O(n) otherwise.

pub fn is_static(&self) -> bool

pub fn timelines(&self) -> &BTreeMap<Timeline, TimeColumn>

pub fn component_names(&self) -> impl Iterator<Item = ComponentName> + '_

pub fn components(&self) -> &BTreeMap<ComponentName, ArrowListArray<i32>>

pub fn timepoint_max(&self) -> TimePoint

Computes the maximum value for each and every timeline present across this entire chunk, and returns the corresponding [TimePoint].

pub fn sanity_check(&self) -> ChunkResult<()>

Returns an error if the Chunk’s invariants are not upheld.

Costly checks are only run in debug builds.

pub fn component_batch_raw( &self, component_name: &ComponentName, row_index: usize ) -> Option<ChunkResult<Box<dyn ArrowArray>>>

Returns the raw data for the specified component.

Returns an error if the row index is out of bounds.

pub fn component_batch<C: Component>( &self, row_index: usize ) -> Option<ChunkResult<Vec<C>>>

Returns the deserialized data for the specified component.

Returns an error if the data cannot be deserialized, or if the row index is out of bounds.

pub fn component_instance_raw( &self, component_name: &ComponentName, row_index: usize, instance_index: usize ) -> Option<ChunkResult<Box<dyn ArrowArray>>>

Returns the raw data for the specified component at the given instance index.

Returns an error if either the row index or instance index are out of bounds.

pub fn component_instance<C: Component>( &self, row_index: usize, instance_index: usize ) -> Option<ChunkResult<C>>

Returns the component data of the specified instance.

Returns an error if the data cannot be deserialized, or if either the row index or instance index are out of bounds.

pub fn component_mono_raw( &self, component_name: &ComponentName, row_index: usize ) -> Option<ChunkResult<Box<dyn ArrowArray>>>

Returns the raw data for the specified component, assuming a mono-batch.

Returns an error if either the row index is out of bounds, or the underlying batch is not of unit length.

pub fn component_mono<C: Component>( &self, row_index: usize ) -> Option<ChunkResult<C>>

Returns the deserialized data for the specified component, assuming a mono-batch.

Returns an error if the data cannot be deserialized, or if either the row index is out of bounds, or the underlying batch is not of unit length.

pub fn to_unit(self: &ChunkShared) -> Option<UnitChunkShared>

Turns the chunk into a UnitChunkShared, if possible.

pub fn iter_indices( &self, timeline: &Timeline ) -> impl Iterator<Item = (TimeInt, RowId)> + '_

Returns an iterator over the indices ((TimeInt, RowId)) of a Chunk, for a given timeline.

If the chunk is static, timeline will be ignored.

See also:

• Self::iter_component_indices.
• Self::iter_indices_owned.

pub fn iter_component_indices( &self, timeline: &Timeline, component_name: &ComponentName ) -> impl Iterator<Item = (TimeInt, RowId)> + '_

Returns an iterator over the indices ((TimeInt, RowId)) of a Chunk, for a given timeline and component.

If the chunk is static, timeline will be ignored.

This is different than Self::iter_indices in that it will only yield indices for rows at which there is data for the specified component_name.

See also Self::iter_indices.

pub fn iter_timepoints(&self) -> impl Iterator<Item = TimePoint> + '_

Returns an iterator over the [TimePoint]s of a Chunk.

See also:

• Self::iter_component_timepoints.

pub fn iter_component_timepoints( &self, component_name: &ComponentName ) -> impl Iterator<Item = TimePoint> + '_

Returns an iterator over the [TimePoint]s of a Chunk, for a given component.

This is different than Self::iter_timepoints in that it will only yield timepoints for rows at which there is data for the specified component_name.

See also Self::iter_timepoints.

pub fn iter_component_offsets( &self, component_name: &ComponentName ) -> impl Iterator<Item = (usize, usize)> + '_

Returns an iterator over the offsets ((offset, len)) of a Chunk, for a given component.

I.e. each (offset, len) pair describes the position of a component batch in the underlying arrow array of values.

pub fn iter_component_arrays( &self, component_name: &ComponentName ) -> impl Iterator<Item = Box<dyn ArrowArray>> + '_

Returns an iterator over the raw arrays of a Chunk, for a given component.

See also:

• Self::iter_primitive
• Self::iter_primitive_array
• Self::iter_primitive_array_list
• Self::iter_string
• Self::iter_buffer.
• Self::iter_component.

pub fn iter_primitive<T: NativeType>( &self, component_name: &ComponentName ) -> impl Iterator<Item = &[T]> + '_

Returns an iterator over the raw primitive values of a Chunk, for a given component.

This is a very fast path: the entire column will be downcasted at once, and then every component batch will be a slice reference into that global slice. Use this when working with simple arrow datatypes and performance matters (e.g. scalars, points, etc).

See also:

• Self::iter_primitive_array
• Self::iter_primitive_array_list
• Self::iter_string
• Self::iter_buffer.
• Self::iter_component_arrays.
• Self::iter_component.

pub fn iter_primitive_array<const N: usize, T: NativeType>( &self, component_name: &ComponentName ) -> impl Iterator<Item = &[[T; N]]> + '_

where [T; N]: Pod,

Returns an iterator over the raw primitive arrays of a Chunk, for a given component.

This is a very fast path: the entire column will be downcasted at once, and then every component batch will be a slice reference into that global slice. Use this when working with simple arrow datatypes and performance matters (e.g. scalars, points, etc).

See also:

• Self::iter_primitive
• Self::iter_string
• Self::iter_buffer.
• Self::iter_component_arrays.
• Self::iter_component.

pub fn iter_primitive_array_list<const N: usize, T: NativeType>( &self, component_name: &ComponentName ) -> impl Iterator<Item = Vec<&[[T; N]]>> + '_

where [T; N]: Pod,

Returns an iterator over the raw list of primitive arrays of a Chunk, for a given component.

This is a very fast path: the entire column will be downcasted at once, and then every component batch will be a slice reference into that global slice. Use this when working with simple arrow datatypes and performance matters (e.g. strips, etc).

See also:

• Self::iter_primitive
• Self::iter_primitive_array
• Self::iter_string
• Self::iter_buffer.
• Self::iter_component_arrays.
• Self::iter_component.

pub fn iter_string( &self, component_name: &ComponentName ) -> impl Iterator<Item = Vec<ArrowString>> + '_

Returns an iterator over the raw strings of a Chunk, for a given component.

This is a very fast path: the entire column will be downcasted at once, and then every component batch will be a slice reference into that global slice. Use this when working with simple arrow datatypes and performance matters (e.g. labels, etc).

See also:

• Self::iter_primitive
• Self::iter_primitive_array
• Self::iter_primitive_array_list
• Self::iter_buffer.
• Self::iter_component_arrays.
• Self::iter_component.

pub fn iter_buffer<T: NativeType>( &self, component_name: &ComponentName ) -> impl Iterator<Item = Vec<ArrowBuffer<T>>> + '_

Returns an iterator over the raw buffers of a Chunk, for a given component.

This is a very fast path: the entire column will be downcasted at once, and then every component batch will be a slice reference into that global slice. Use this when working with simple arrow datatypes and performance matters (e.g. blobs, etc).

See also:

• Self::iter_primitive
• Self::iter_primitive_array
• Self::iter_primitive_array_list
• Self::iter_string.
• Self::iter_component_arrays.
• Self::iter_component.

pub fn iter_indices_owned( self: Arc<Self>, timeline: &Timeline ) -> impl Iterator<Item = (TimeInt, RowId)>

Returns an iterator over the indices ((TimeInt, RowId)) of a Chunk, for a given timeline.

If the chunk is static, timeline will be ignored.

The returned iterator outlives self, thus it can be passed around freely. The tradeoff is that self must be an Arc.

See also Self::iter_indices.

pub fn iter_component<C: Component>( &self ) -> ChunkComponentIter<C, impl Iterator<Item = (usize, usize)> + '_>

Returns an iterator over the deserialized batches of a Chunk, for a given component.

This is a dedicated fast path: the entire column will be downcasted and deserialized at once, and then every component batch will be a slice reference into that global slice. Use this when working with complex arrow datatypes and performance matters (e.g. ranging through enum types across many timestamps).

See also:

• Self::iter_primitive
• Self::iter_primitive_array
• Self::iter_primitive_array_list
• Self::iter_string
• Self::iter_buffer.
• Self::iter_component_arrays.

pub fn latest_at( &self, query: &LatestAtQuery, component_name: ComponentName ) -> Self

Runs a LatestAtQuery filter on a Chunk.

This behaves as a row-based filter: the result is a new Chunk that is vertically sliced to only contain the row relevant for the specified query.

The resulting Chunk is guaranteed to contain all the same columns has the queried chunk: there is no horizontal slicing going on.

An empty Chunk (i.e. 0 rows, but N columns) is returned if the query yields nothing.

Because the resulting chunk doesn’t discard any column information, you can find extra relevant information by inspecting the data, for examples timestamps on other timelines. See Self::timeline_sliced and Self::component_sliced if you do want to filter this extra data.

pub fn concatenated(&self, rhs: &Self) -> ChunkResult<Self>

Concatenates two Chunks into a new one.

The order of the arguments matter: self‘s contents will precede rhs’ contents in the returned Chunk.

This will return an error if the chunks are not concatenable.

pub fn overlaps_on_row_id(&self, rhs: &Self) -> bool

Returns true if self and rhs overlap on their RowId range.

pub fn overlaps_on_time(&self, rhs: &Self) -> bool

Returns true if self and rhs overlap on any of their time range(s).

This does not imply that they share the same exact set of timelines.

pub fn same_entity_paths(&self, rhs: &Self) -> bool

Returns true if both chunks share the same entity path.

pub fn same_timelines(&self, rhs: &Self) -> bool

Returns true if both chunks contains the same set of timelines.

pub fn same_datatypes(&self, rhs: &Self) -> bool

Returns true if both chunks share the same datatypes for the components that they have in common.

pub fn concatenable(&self, rhs: &Self) -> bool

Returns true if two chunks are concatenable.

To be concatenable, two chunks must:

• Share the same entity path.
• Share the same exact set of timelines.
• Use the same datatypes for the components they have in common.

pub fn range(&self, query: &RangeQuery, component_name: ComponentName) -> Self

Runs a RangeQuery filter on a Chunk.

This behaves as a row-based filter: the result is a new Chunk that is vertically sliced, sorted and filtered in order to only contain the row(s) relevant for the specified query.

The resulting Chunk is guaranteed to contain all the same columns has the queried chunk: there is no horizontal slicing going on.

An empty Chunk (i.e. 0 rows, but N columns) is returned if the query yields nothing.

Because the resulting chunk doesn’t discard any column information, you can find extra relevant information by inspecting the data, for examples timestamps on other timelines. See Self::timeline_sliced and Self::component_sliced if you do want to filter this extra data.

pub fn is_sorted(&self) -> bool

Is the chunk currently ascendingly sorted by crate::RowId?

This is O(1) (cached).

See also Self::is_sorted_uncached.

pub fn is_time_sorted(&self) -> bool

Is the chunk ascendingly sorted by time, for all of its timelines?

This is O(1) (cached).

pub fn is_timeline_sorted(&self, timeline: &Timeline) -> bool

Is the chunk ascendingly sorted by time, for a specific timeline?

This is O(1) (cached).

See also Self::is_timeline_sorted_uncached.

pub fn sorted_by_timeline_if_unsorted(&self, timeline: &Timeline) -> Self

Returns a new Chunk that is sorted by (<timeline>, RowId).

The underlying arrow data will be copied and shuffled in memory in order to make it contiguous.

This is a no-op if the underlying timeline is already sorted appropriately (happy path).

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn cell( &self, row_id: RowId, component_name: &ComponentName ) -> Option<Box<dyn ArrowArray>>

Returns the cell corresponding to the specified RowId for a given [ComponentName].

This is O(log(n)) if self.is_sorted(), and O(n) otherwise.

Reminder: duplicated RowIds results in undefined behavior.

pub fn row_sliced(&self, index: usize, len: usize) -> Self

Slices the Chunk vertically.

The result is a new Chunk with the same columns and (potentially) less rows.

This cannot fail nor panic: index and len will be capped so that they cannot run out of bounds. This can result in an empty Chunk being returned if the slice is completely OOB.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn timeline_sliced(&self, timeline: Timeline) -> Self

Slices the Chunk horizontally by keeping only the selected timeline.

The result is a new Chunk with the same rows and (at-most) one timeline column. All non-timeline columns will be kept as-is.

If timeline is not found within the Chunk, the end result will be the same as the current chunk but without any timeline column.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn component_sliced(&self, component_name: ComponentName) -> Self

Slices the Chunk horizontally by keeping only the selected component_name.

The result is a new Chunk with the same rows and (at-most) one component column. All non-component columns will be kept as-is.

If component_name is not found within the Chunk, the end result will be the same as the current chunk but without any component column.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn timelines_sliced(&self, timelines_to_keep: &IntSet<Timeline>) -> Self

Slices the Chunk horizontally by keeping only the selected timelines.

The result is a new Chunk with the same rows and (at-most) the selected timeline columns. All non-timeline columns will be kept as-is.

If none of the selected timelines exist in the Chunk, the end result will be the same as the current chunk but without any timeline column.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn components_sliced(&self, component_names: &IntSet<ComponentName>) -> Self

Slices the Chunk horizontally by keeping only the selected component_names.

The result is a new Chunk with the same rows and (at-most) the selected component columns. All non-component columns will be kept as-is.

If none of the component_names exist in the Chunk, the end result will be the same as the current chunk but without any component column.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn densified(&self, component_name_pov: ComponentName) -> Self

Densifies the Chunk vertically based on the component_name column.

Densifying here means dropping all rows where the associated value in the component_name column is null.

The result is a new Chunk where the component_name column is guaranteed to be dense.

If component_name doesn’t exist in this Chunk, or if it is already dense, this method is a no-op.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn emptied(&self) -> Self

Empties the Chunk vertically.

The result is a new Chunk with the same columns but zero rows.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn deduped_latest_on_index(&self, index: &Timeline) -> Self

Removes duplicate rows from sections of consecutive identical indices.

• If the Chunk is sorted on that index, the remaining values in the index column will be unique.
• If the Chunk has been densified on a specific column, the resulting chunk will effectively contain the latest value of that column for each given index value.

If this is a temporal chunk and timeline isn’t present in it, this method is a no-op.

This does not obey RowId-ordering semantics (or any other kind of semantics for that matter) – it merely respects how the chunk is currently laid out: no more, no less. Sort the chunk according to the semantics you’re looking for before calling this method.

pub fn filtered(&self, filter: &ArrowBooleanArray) -> Option<Self>

Applies a filter kernel to the Chunk as a whole.

Returns None if the length of the filter does not match the number of rows in the chunk.

In release builds, filters are allowed to have null entries (they will be interpreted as false). In debug builds, null entries will panic.

Note: a filter kernel copies the data in order to make the resulting arrays contiguous in memory.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn taken<O: Index>(&self, indices: &ArrowPrimitiveArray<O>) -> Self

Applies a take kernel to the Chunk as a whole.

In release builds, indices are allowed to have null entries (they will be taken as nulls). In debug builds, null entries will panic.

Note: a take kernel copies the data in order to make the resulting arrays contiguous in memory.

Takes care of up- and down-casting the data back and forth on behalf of the caller.

WARNING: the returned chunk has the same old crate::ChunkId! Change it with Self::with_id.

pub fn to_transport(&self) -> ChunkResult<TransportChunk>

Prepare the Chunk for transport.

It is probably a good idea to sort the chunk first.

pub fn to_arrow_msg(&self) -> ChunkResult<ArrowMsg>

Trait Implementations

impl Clone for UnitChunkShared

fn clone(&self) -> UnitChunkShared

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnitChunkShared

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Deref for UnitChunkShared

type Target = Chunk

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl SizeBytes for UnitChunkShared

fn heap_size_bytes(&self) -> u64

Returns the total size of self on the heap, in bytes.

fn total_size_bytes(&self) -> u64

Returns the total size of self in bytes, accounting for both stack and heap space.

fn stack_size_bytes(&self) -> u64

Returns the total size of self on the stack, in bytes.

fn is_pod() -> bool

Is Self just plain old data?