Struct rerun::dataframe::external::re_chunk_store::ChunkStore

pub struct ChunkStore {

    pub(crate) id: StoreId,
    pub(crate) info: Option<StoreInfo>,
    pub(crate) config: ChunkStoreConfig,
    pub(crate) type_registry: HashMap<ComponentName, DataType, BuildHasherDefault<NoHashHasher<ComponentName>>>,
    pub(crate) per_column_metadata: BTreeMap<EntityPath, BTreeMap<ComponentName, ColumnMetadataState>>,
    pub(crate) chunks_per_chunk_id: BTreeMap<ChunkId, Arc<Chunk>>,
    pub(crate) chunk_ids_per_min_row_id: BTreeMap<RowId, Vec<ChunkId>>,
    pub(crate) temporal_chunk_ids_per_entity_per_component: BTreeMap<EntityPath, BTreeMap<Timeline, BTreeMap<ComponentName, ChunkIdSetPerTime>>>,
    pub(crate) temporal_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<Timeline, ChunkIdSetPerTime>>,
    pub(crate) temporal_chunks_stats: ChunkStoreChunkStats,
    pub(crate) static_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<ComponentName, ChunkId>>,
    pub(crate) static_chunks_stats: ChunkStoreChunkStats,
    pub(crate) insert_id: u64,
    pub(crate) query_id: AtomicU64,
    pub(crate) gc_id: u64,
    pub(crate) event_id: AtomicU64,
}

A complete chunk store: covers all timelines, all entities, everything.

The chunk store always works at the chunk level, whether it is for write & read queries or garbage collection. It is completely oblivious to individual rows.

Use the Display implementation for a detailed view of the internals.

Fields

id: StoreId

info: Option<StoreInfo>

config: ChunkStoreConfig

type_registry: HashMap<ComponentName, DataType, BuildHasherDefault<NoHashHasher<ComponentName>>>

per_column_metadata: BTreeMap<EntityPath, BTreeMap<ComponentName, ColumnMetadataState>>

chunks_per_chunk_id: BTreeMap<ChunkId, Arc<Chunk>>

chunk_ids_per_min_row_id: BTreeMap<RowId, Vec<ChunkId>>

temporal_chunk_ids_per_entity_per_component: BTreeMap<EntityPath, BTreeMap<Timeline, BTreeMap<ComponentName, ChunkIdSetPerTime>>>

temporal_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<Timeline, ChunkIdSetPerTime>>

temporal_chunks_stats: ChunkStoreChunkStats

static_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<ComponentName, ChunkId>>

static_chunks_stats: ChunkStoreChunkStats

insert_id: u64

query_id: AtomicU64

gc_id: u64

event_id: AtomicU64

Implementations

impl ChunkStore

pub fn schema(&self) -> Vec<ColumnDescriptor>

Returns the full schema of the store.

This will include a column descriptor for every timeline and every component on every entity that has been written to the store so far.

The order of the columns is guaranteed to be in a specific order:

• first, the time columns in lexical order (frame_nr, log_time, …);
• second, the component columns in lexical order (Color, Radius, ...).

pub fn resolve_time_selector( &self, selector: &TimeColumnSelector ) -> TimeColumnDescriptor

Given a TimeColumnSelector, returns the corresponding TimeColumnDescriptor.

pub fn resolve_component_selector( &self, selector: &ComponentColumnSelector ) -> ComponentColumnDescriptor

Given a ComponentColumnSelector, returns the corresponding ComponentColumnDescriptor.

If the component is not found in the store, a default descriptor is returned with a null datatype.

pub fn resolve_selectors( &self, selectors: impl IntoIterator<Item = impl Into<ColumnSelector>> ) -> Vec<ColumnDescriptor>

Given a set of ColumnSelectors, returns the corresponding ColumnDescriptors.

pub fn schema_for_query(&self, query: &QueryExpression) -> Vec<ColumnDescriptor>

Returns the filtered schema for the given QueryExpression.

The order of the columns is guaranteed to be in a specific order:

• first, the time columns in lexical order (frame_nr, log_time, …);
• second, the component columns in lexical order (Color, Radius, ...).

impl ChunkStore

pub fn drop_time_range( &mut self, timeline: &Timeline, drop_range: ResolvedTimeRange ) -> Vec<ChunkStoreEvent>

Drop all events that are in the given range on the given timeline.

Note that matching events will be dropped from all timelines they appear on.

Static chunks are unaffected.

Used to implement undo (erase the last event from the blueprint db).

impl ChunkStore

pub fn gc( &mut self, options: &GarbageCollectionOptions ) -> (Vec<ChunkStoreEvent>, ChunkStoreStats)

Triggers a garbage collection according to the desired target.

Returns the list of Chunks that were purged from the store in the form of ChunkStoreEvents.

Semantics

Garbage collection works on a chunk-level basis and is driven by RowId order (specifically, the smallest RowId of each respective Chunk), i.e. the order defined by the clients’ wall-clocks, allowing it to drop data across the different timelines in a fair, deterministic manner. Similarly, out-of-order data is supported out of the box.

The garbage collector doesn’t deallocate data in and of itself: all it does is drop the store’s internal references to that data (the Chunks), which will be deallocated once their reference count reaches 0.

Limitations

The garbage collector has limited support for latest-at semantics. The configuration option: GarbageCollectionOptions::protect_latest will protect the N latest values of each component on each timeline. The only practical guarantee this gives is that a latest-at query with a value of max-int will be unchanged. However, latest-at queries from other arbitrary points in time may provide different results pre- and post- GC.

impl ChunkStore

pub fn all_timelines(&self) -> BTreeSet<Timeline>

Retrieve all Timelines in the store.

pub fn all_entities(&self) -> BTreeSet<EntityPath>

Retrieve all EntityPaths in the store.

pub fn all_components(&self) -> BTreeSet<ComponentName>

Retrieve all ComponentNames in the store.

pub fn all_components_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<BTreeSet<ComponentName>>

Retrieve all the ComponentNames that have been written to for a given EntityPath on the specified Timeline.

Static components are always included in the results.

Returns None if the entity doesn’t exist at all on this timeline.

pub fn all_components_for_entity( &self, entity_path: &EntityPath ) -> Option<BTreeSet<ComponentName>>

Retrieve all the ComponentNames that have been written to for a given EntityPath.

Static components are always included in the results.

Returns None if the entity has never had any data logged to it.

pub fn entity_has_component_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a static component or a temporal component on the specified timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_component( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a static component or a temporal component on any timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_static_component( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a specific static component.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_temporal_component( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a temporal component on any timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_temporal_component_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a temporal component on a specific timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_data_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath ) -> bool

Check whether an entity has any data on a specific timeline, or any static data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any static data or any temporal data on any timeline.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_static_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any static data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_temporal_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any temporal data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_temporal_data_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath ) -> bool

Check whether an entity has any temporal data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_min_time( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<TimeInt>

Find the earliest time at which something was logged for a given entity on the specified timeline.

Ignores static data.

pub fn entity_time_range( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<ResolvedTimeRange>

Returns the min and max times at which data was logged for an entity on a specific timeline.

This ignores static data.

pub fn time_range(&self, timeline: &Timeline) -> Option<ResolvedTimeRange>

Returns the min and max times at which data was logged on a specific timeline, considering all entities.

This ignores static data.

impl ChunkStore

pub fn latest_at_relevant_chunks( &self, query: &LatestAtQuery, entity_path: &EntityPath, component_name: ComponentName ) -> Vec<Arc<Chunk>>

Returns the most-relevant chunk(s) for the given LatestAtQuery and ComponentName.

The returned vector is guaranteed free of duplicates, by definition.

The ChunkStore always work at the Chunk level (as opposed to the row level): it is oblivious to the data therein. For that reason, and because Chunks are allowed to temporally overlap, it is possible that a query has more than one relevant chunk.

The caller should filter the returned chunks further (see Chunk::latest_at) in order to determine what exact row contains the final result.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn latest_at_relevant_chunks_for_all_components( &self, query: &LatestAtQuery, entity_path: &EntityPath ) -> Vec<Arc<Chunk>>

Returns the most-relevant temporal chunk(s) for the given LatestAtQuery.

The returned vector is guaranteed free of duplicates, by definition.

The ChunkStore always work at the Chunk level (as opposed to the row level): it is oblivious to the data therein. For that reason, and because Chunks are allowed to temporally overlap, it is possible that a query has more than one relevant chunk.

The caller should filter the returned chunks further (see Chunk::latest_at) in order to determine what exact row contains the final result.

This ignores static data.

impl ChunkStore

pub fn range_relevant_chunks( &self, query: &RangeQuery, entity_path: &EntityPath, component_name: ComponentName ) -> Vec<Arc<Chunk>>

Returns the most-relevant chunk(s) for the given RangeQuery and ComponentName.

The returned vector is guaranteed free of duplicates, by definition.

The criteria for returning a chunk is only that it may contain data that overlaps with the queried range.

The caller should filter the returned chunks further (see Chunk::range) in order to determine how exactly each row of data fit with the rest.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn range_relevant_chunks_for_all_components( &self, query: &RangeQuery, entity_path: &EntityPath ) -> Vec<Arc<Chunk>>

Returns the most-relevant temporal chunk(s) for the given RangeQuery.

The returned vector is guaranteed free of duplicates, by definition.

The criteria for returning a chunk is only that it may contain data that overlaps with the queried range.

The caller should filter the returned chunks further (see Chunk::range) in order to determine how exactly each row of data fit with the rest.

This ignores static data.

impl ChunkStore

pub fn stats(&self) -> ChunkStoreStats

impl ChunkStore

Entity stats

pub fn entity_stats_static( &self, entity_path: &EntityPath ) -> ChunkStoreChunkStats

Stats about all chunks with static data for an entity.

pub fn entity_stats_on_timeline( &self, entity_path: &EntityPath, timeline: &Timeline ) -> ChunkStoreChunkStats

Stats about all the chunks that has data for an entity on a specific timeline.

Does NOT include static data.

impl ChunkStore

Component path stats

pub fn num_static_events_for_component( &self, entity_path: &EntityPath, component_name: ComponentName ) -> u64

Returns the number of static events logged for an entity for a specific component.

This ignores temporal events.

pub fn num_temporal_events_for_component_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath, component_name: ComponentName ) -> u64

Returns the number of temporal events logged for an entity for a specific component on a given timeline.

This ignores static events.

impl ChunkStore

pub fn new(id: StoreId, config: ChunkStoreConfig) -> ChunkStore

Instantiate a new empty ChunkStore with the given ChunkStoreConfig.

See also:

• ChunkStore::new
• ChunkStore::from_rrd_filepath

pub fn new_handle(id: StoreId, config: ChunkStoreConfig) -> ChunkStoreHandle

Instantiate a new empty ChunkStore with the given ChunkStoreConfig.

Pre-wraps the result in a ChunkStoreHandle.

See also:

• ChunkStore::from_rrd_filepath

pub fn id(&self) -> StoreId

pub fn set_info(&mut self, info: StoreInfo)

pub fn info(&self) -> Option<&StoreInfo>

pub fn generation(&self) -> ChunkStoreGeneration

Return the current ChunkStoreGeneration. This can be used to determine whether the database has been modified since the last time it was queried.

pub fn config(&self) -> &ChunkStoreConfig

See ChunkStoreConfig for more information about configuration.

pub fn iter_chunks(&self) -> impl Iterator<Item = &Arc<Chunk>>

Iterate over all chunks in the store, in ascending ChunkId order.

pub fn chunk(&self, id: &ChunkId) -> Option<&Arc<Chunk>>

Get a chunk based on its ID.

pub fn num_chunks(&self) -> usize

Get the number of chunks.

pub fn lookup_datatype( &self, component_name: &ComponentName ) -> Option<&DataType>

Lookup the latest arrow ArrowDataType used by a specific re_types_core::Component.

pub fn lookup_column_metadata( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> Option<ColumnMetadata>

Lookup the ColumnMetadata for a specific EntityPath and re_types_core::Component.

impl ChunkStore

pub fn from_rrd_filepath( store_config: &ChunkStoreConfig, path_to_rrd: impl AsRef<Path>, version_policy: VersionPolicy ) -> Result<BTreeMap<StoreId, ChunkStore>, Error>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

The stores will be prefilled with the data at the specified path.

See also:

• ChunkStore::new

pub fn handle_from_rrd_filepath( store_config: &ChunkStoreConfig, path_to_rrd: impl AsRef<Path>, version_policy: VersionPolicy ) -> Result<BTreeMap<StoreId, ChunkStoreHandle>, Error>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

Wraps the results in ChunkStoreHandles.

The stores will be prefilled with the data at the specified path.

See also:

• ChunkStore::new_handle

impl ChunkStore

pub fn register_subscriber( subscriber: Box<dyn ChunkStoreSubscriber> ) -> ChunkStoreSubscriberHandle

Registers a ChunkStoreSubscriber so it gets automatically notified when data gets added and/or removed to/from a ChunkStore.

Refer to ChunkStoreEvent’s documentation for more information about these events.

Scope

Registered ChunkStoreSubscribers are global scope: they get notified of all events from all existing ChunkStores, including ChunkStores created after the subscriber was registered.

Use ChunkStoreEvent::store_id to identify the source of an event.

Late registration

Subscribers must be registered before a store gets created to guarantee that no events were missed.

ChunkStoreEvent::event_id can be used to identify missing events.

Ordering

The order in which registered subscribers are notified is undefined and will likely become concurrent in the future.

If you need a specific order across multiple subscribers, embed them into an orchestrating subscriber.

pub fn with_subscriber<V, T, F>( _: ChunkStoreSubscriberHandle, f: F ) -> Option<T>

where V: ChunkStoreSubscriber, F: FnMut(&V) -> T,

Passes a reference to the downcasted subscriber to the given FnMut callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_once<V, T, F>( _: ChunkStoreSubscriberHandle, f: F ) -> Option<T>

where V: ChunkStoreSubscriber, F: FnOnce(&V) -> T,

Passes a reference to the downcasted subscriber to the given FnOnce callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_mut<V, T, F>( _: ChunkStoreSubscriberHandle, f: F ) -> Option<T>

where V: ChunkStoreSubscriber, F: FnMut(&mut V) -> T,

Passes a mutable reference to the downcasted subscriber to the given callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

impl ChunkStore

pub fn insert_chunk( &mut self, chunk: &Arc<Chunk> ) -> Result<Vec<ChunkStoreEvent>, ChunkStoreError>

Inserts a Chunk in the store.

Iff the store was modified, all registered subscribers will be notified and the resulting ChunkStoreEvent will be returned, or None otherwise.

• Trying to insert an unsorted chunk (Chunk::is_sorted) will fail with an error.
• Inserting a duplicated ChunkId will result in a no-op.
• Inserting an empty Chunk will result in a no-op.

pub fn drop_entity_path( &mut self, entity_path: &EntityPath ) -> Vec<ChunkStoreEvent>

Unconditionally drops all the data for a given entity_path.

Returns the list of Chunks that were dropped from the store in the form of ChunkStoreEvents.

This is not recursive. The store is unaware of the entity hierarchy.

Trait Implementations

impl Clone for ChunkStore

fn clone(&self) -> ChunkStore

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Display for ChunkStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.