Struct re_chunk_store::store::ChunkStore

pub struct ChunkStore {

    pub(crate) id: StoreId,
    pub(crate) info: Option<StoreInfo>,
    pub(crate) config: ChunkStoreConfig,
    pub(crate) type_registry: IntMap<ComponentName, DataType>,
    pub(crate) per_column_metadata: BTreeMap<EntityPath, BTreeMap<ComponentName, ColumnMetadataState>>,
    pub(crate) chunks_per_chunk_id: BTreeMap<ChunkId, Arc<Chunk>>,
    pub(crate) chunk_ids_per_min_row_id: BTreeMap<RowId, Vec<ChunkId>>,
    pub(crate) temporal_chunk_ids_per_entity_per_component: BTreeMap<EntityPath, BTreeMap<Timeline, BTreeMap<ComponentName, ChunkIdSetPerTime>>>,
    pub(crate) temporal_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<Timeline, ChunkIdSetPerTime>>,
    pub(crate) temporal_chunks_stats: ChunkStoreChunkStats,
    pub(crate) static_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<ComponentName, ChunkId>>,
    pub(crate) static_chunks_stats: ChunkStoreChunkStats,
    pub(crate) insert_id: u64,
    pub(crate) query_id: AtomicU64,
    pub(crate) gc_id: u64,
    pub(crate) event_id: AtomicU64,
}

A complete chunk store: covers all timelines, all entities, everything.

The chunk store always works at the chunk level, whether it is for write & read queries or garbage collection. It is completely oblivious to individual rows.

Use the Display implementation for a detailed view of the internals.

Fields

id: StoreId

info: Option<StoreInfo>

config: ChunkStoreConfig

The configuration of the chunk store (e.g. compaction settings).

type_registry: IntMap<ComponentName, DataType>

Keeps track of the latest datatype information for all component types that have been written to the store so far.

See also Self::lookup_datatype.

per_column_metadata: BTreeMap<EntityPath, BTreeMap<ComponentName, ColumnMetadataState>>

chunks_per_chunk_id: BTreeMap<ChunkId, Arc<Chunk>>

chunk_ids_per_min_row_id: BTreeMap<RowId, Vec<ChunkId>>

All [ChunkId]s currently in the store, indexed by the smallest [RowId] in each of them.

This is effectively all chunks in global data order. Used for garbage collection.

This is a map of vecs instead of individual [ChunkId] in order to better support duplicated [RowId]s.

temporal_chunk_ids_per_entity_per_component: BTreeMap<EntityPath, BTreeMap<Timeline, BTreeMap<ComponentName, ChunkIdSetPerTime>>>

All temporal [ChunkId]s for all entities on all timelines, further indexed by [ComponentName].

See also:

• Self::temporal_chunk_ids_per_entity.
• Self::static_chunk_ids_per_entity.

temporal_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<Timeline, ChunkIdSetPerTime>>

All temporal [ChunkId]s for all entities on all timelines, without the [ComponentName] index.

See also:

• Self::temporal_chunk_ids_per_entity_per_component.
• Self::static_chunk_ids_per_entity.

temporal_chunks_stats: ChunkStoreChunkStats

Accumulated size statitistics for all temporal [Chunk]s currently present in the store.

This is too costly to be computed from scratch every frame, and is required by e.g. the GC.

static_chunk_ids_per_entity: BTreeMap<EntityPath, BTreeMap<ComponentName, ChunkId>>

Static data. Never garbage collected.

Static data unconditionally shadows temporal data at query time.

Existing temporal will not be removed. Events won’t be fired.

static_chunks_stats: ChunkStoreChunkStats

Accumulated size statitistics for all static [Chunk]s currently present in the store.

This is too costly to be computed from scratch every frame, and is required by e.g. the GC.

insert_id: u64

Monotonically increasing ID for insertions.

query_id: AtomicU64

Monotonically increasing ID for queries.

gc_id: u64

Monotonically increasing ID for GCs.

event_id: AtomicU64

Monotonically increasing ID for store events.

Implementations

impl ChunkStore

pub fn schema(&self) -> Vec<ColumnDescriptor>

Returns the full schema of the store.

This will include a column descriptor for every timeline and every component on every entity that has been written to the store so far.

The order of the columns is guaranteed to be in a specific order:

• first, the time columns in lexical order (frame_nr, log_time, …);
• second, the component columns in lexical order (Color, Radius, ...).

pub fn resolve_time_selector( &self, selector: &TimeColumnSelector ) -> TimeColumnDescriptor

Given a TimeColumnSelector, returns the corresponding TimeColumnDescriptor.

pub fn resolve_component_selector( &self, selector: &ComponentColumnSelector ) -> ComponentColumnDescriptor

Given a ComponentColumnSelector, returns the corresponding ComponentColumnDescriptor.

If the component is not found in the store, a default descriptor is returned with a null datatype.

pub fn resolve_selectors( &self, selectors: impl IntoIterator<Item = impl Into<ColumnSelector>> ) -> Vec<ColumnDescriptor>

Given a set of ColumnSelectors, returns the corresponding ColumnDescriptors.

pub fn schema_for_query(&self, query: &QueryExpression) -> Vec<ColumnDescriptor>

Returns the filtered schema for the given QueryExpression.

The order of the columns is guaranteed to be in a specific order:

• first, the time columns in lexical order (frame_nr, log_time, …);
• second, the component columns in lexical order (Color, Radius, ...).

impl ChunkStore

pub fn drop_time_range( &mut self, timeline: &Timeline, drop_range: ResolvedTimeRange ) -> Vec<ChunkStoreEvent>

Drop all events that are in the given range on the given timeline.

Note that matching events will be dropped from all timelines they appear on.

Static chunks are unaffected.

Used to implement undo (erase the last event from the blueprint db).

impl ChunkStore

pub fn gc( &mut self, options: &GarbageCollectionOptions ) -> (Vec<ChunkStoreEvent>, ChunkStoreStats)

Triggers a garbage collection according to the desired target.

Returns the list of Chunks that were purged from the store in the form of ChunkStoreEvents.

Semantics

Garbage collection works on a chunk-level basis and is driven by [RowId] order (specifically, the smallest RowId of each respective Chunk), i.e. the order defined by the clients’ wall-clocks, allowing it to drop data across the different timelines in a fair, deterministic manner. Similarly, out-of-order data is supported out of the box.

The garbage collector doesn’t deallocate data in and of itself: all it does is drop the store’s internal references to that data (the Chunks), which will be deallocated once their reference count reaches 0.

Limitations

The garbage collector has limited support for latest-at semantics. The configuration option: GarbageCollectionOptions::protect_latest will protect the N latest values of each component on each timeline. The only practical guarantee this gives is that a latest-at query with a value of max-int will be unchanged. However, latest-at queries from other arbitrary points in time may provide different results pre- and post- GC.

fn find_all_protected_chunk_ids(&self, target_count: usize) -> BTreeSet<ChunkId>

For each EntityPath, Timeline, Component find the N latest [ChunkId]s.

fn gc_drop_at_least_num_bytes( &mut self, options: &GarbageCollectionOptions, num_bytes_to_drop: f64, protected_chunk_ids: &BTreeSet<ChunkId> ) -> Vec<ChunkStoreDiff>

pub(crate) fn remove_chunk(&mut self, chunk_id: ChunkId) -> Vec<ChunkStoreDiff>

Surgically removes a temporal [ChunkId] from all indices.

This is orders of magnitude faster than trying to retain() on all our internal indices.

See also ChunkStore::remove_chunks.

pub(crate) fn remove_chunks( &mut self, chunk_ids_to_be_removed: IntMap<EntityPath, IntMap<Timeline, IntMap<ComponentName, HashMap<TimeInt, Vec<ChunkId>>>>>, time_budget: Option<(Instant, Duration)> ) -> Vec<ChunkStoreDiff>

Surgically removes a set of temporal [ChunkId]s from all indices.

This is orders of magnitude faster than trying to retain() on all our internal indices, when you already know where these chunks live.

See also ChunkStore::remove_chunk.

impl ChunkStore

pub fn all_timelines(&self) -> BTreeSet<Timeline>

Retrieve all [Timeline]s in the store.

pub fn all_entities(&self) -> BTreeSet<EntityPath>

Retrieve all [EntityPath]s in the store.

pub fn all_components(&self) -> BTreeSet<ComponentName>

Retrieve all [ComponentName]s in the store.

pub fn all_components_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<ComponentNameSet>

Retrieve all the [ComponentName]s that have been written to for a given [EntityPath] on the specified [Timeline].

Static components are always included in the results.

Returns None if the entity doesn’t exist at all on this timeline.

pub fn all_components_for_entity( &self, entity_path: &EntityPath ) -> Option<ComponentNameSet>

Retrieve all the [ComponentName]s that have been written to for a given [EntityPath].

Static components are always included in the results.

Returns None if the entity has never had any data logged to it.

pub fn entity_has_component_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a static component or a temporal component on the specified timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_component( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a static component or a temporal component on any timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_static_component( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a specific static component.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_temporal_component( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a temporal component on any timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_temporal_component_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath, component_name: &ComponentName ) -> bool

Check whether an entity has a temporal component on a specific timeline.

This does not check if the entity actually currently holds any data for that component.

pub fn entity_has_data_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath ) -> bool

Check whether an entity has any data on a specific timeline, or any static data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any static data or any temporal data on any timeline.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_static_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any static data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_temporal_data(&self, entity_path: &EntityPath) -> bool

Check whether an entity has any temporal data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_has_temporal_data_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath ) -> bool

Check whether an entity has any temporal data.

This is different from checking if the entity has any component, it also ensures that some data currently exists in the store for this entity.

pub fn entity_min_time( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<TimeInt>

Find the earliest time at which something was logged for a given entity on the specified timeline.

Ignores static data.

pub fn entity_time_range( &self, timeline: &Timeline, entity_path: &EntityPath ) -> Option<ResolvedTimeRange>

Returns the min and max times at which data was logged for an entity on a specific timeline.

This ignores static data.

pub fn time_range(&self, timeline: &Timeline) -> Option<ResolvedTimeRange>

Returns the min and max times at which data was logged on a specific timeline, considering all entities.

This ignores static data.

impl ChunkStore

pub fn latest_at_relevant_chunks( &self, query: &LatestAtQuery, entity_path: &EntityPath, component_name: ComponentName ) -> Vec<Arc<Chunk>>

Returns the most-relevant chunk(s) for the given [LatestAtQuery] and [ComponentName].

The returned vector is guaranteed free of duplicates, by definition.

The ChunkStore always work at the [Chunk] level (as opposed to the row level): it is oblivious to the data therein. For that reason, and because [Chunk]s are allowed to temporally overlap, it is possible that a query has more than one relevant chunk.

The caller should filter the returned chunks further (see [Chunk::latest_at]) in order to determine what exact row contains the final result.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn latest_at_relevant_chunks_for_all_components( &self, query: &LatestAtQuery, entity_path: &EntityPath ) -> Vec<Arc<Chunk>>

Returns the most-relevant temporal chunk(s) for the given [LatestAtQuery].

The returned vector is guaranteed free of duplicates, by definition.

The ChunkStore always work at the [Chunk] level (as opposed to the row level): it is oblivious to the data therein. For that reason, and because [Chunk]s are allowed to temporally overlap, it is possible that a query has more than one relevant chunk.

The caller should filter the returned chunks further (see [Chunk::latest_at]) in order to determine what exact row contains the final result.

This ignores static data.

fn latest_at( &self, query: &LatestAtQuery, temporal_chunk_ids_per_time: &ChunkIdSetPerTime ) -> Option<Vec<Arc<Chunk>>>

impl ChunkStore

pub fn range_relevant_chunks( &self, query: &RangeQuery, entity_path: &EntityPath, component_name: ComponentName ) -> Vec<Arc<Chunk>>

Returns the most-relevant chunk(s) for the given [RangeQuery] and [ComponentName].

The returned vector is guaranteed free of duplicates, by definition.

The criteria for returning a chunk is only that it may contain data that overlaps with the queried range.

The caller should filter the returned chunks further (see [Chunk::range]) in order to determine how exactly each row of data fit with the rest.

If the entity has static component data associated with it, it will unconditionally override any temporal component data.

pub fn range_relevant_chunks_for_all_components( &self, query: &RangeQuery, entity_path: &EntityPath ) -> Vec<Arc<Chunk>>

Returns the most-relevant temporal chunk(s) for the given [RangeQuery].

The returned vector is guaranteed free of duplicates, by definition.

The criteria for returning a chunk is only that it may contain data that overlaps with the queried range.

The caller should filter the returned chunks further (see [Chunk::range]) in order to determine how exactly each row of data fit with the rest.

This ignores static data.

fn range<'a>( &'a self, query: &RangeQuery, temporal_chunk_ids_per_times: impl Iterator<Item = &'a ChunkIdSetPerTime> ) -> Vec<Arc<Chunk>>

impl ChunkStore

pub fn stats(&self) -> ChunkStoreStats

impl ChunkStore

Entity stats

pub fn entity_stats_static( &self, entity_path: &EntityPath ) -> ChunkStoreChunkStats

Stats about all chunks with static data for an entity.

pub fn entity_stats_on_timeline( &self, entity_path: &EntityPath, timeline: &Timeline ) -> ChunkStoreChunkStats

Stats about all the chunks that has data for an entity on a specific timeline.

Does NOT include static data.

impl ChunkStore

Component path stats

pub fn num_static_events_for_component( &self, entity_path: &EntityPath, component_name: ComponentName ) -> u64

Returns the number of static events logged for an entity for a specific component.

This ignores temporal events.

pub fn num_temporal_events_for_component_on_timeline( &self, timeline: &Timeline, entity_path: &EntityPath, component_name: ComponentName ) -> u64

Returns the number of temporal events logged for an entity for a specific component on a given timeline.

This ignores static events.

impl ChunkStore

pub fn new(id: StoreId, config: ChunkStoreConfig) -> Self

Instantiate a new empty ChunkStore with the given ChunkStoreConfig.

See also:

• ChunkStore::new
• ChunkStore::from_rrd_filepath

pub fn new_handle(id: StoreId, config: ChunkStoreConfig) -> ChunkStoreHandle

Instantiate a new empty ChunkStore with the given ChunkStoreConfig.

Pre-wraps the result in a ChunkStoreHandle.

See also:

• ChunkStore::from_rrd_filepath

pub fn id(&self) -> StoreId

pub fn set_info(&mut self, info: StoreInfo)

pub fn info(&self) -> Option<&StoreInfo>

pub fn generation(&self) -> ChunkStoreGeneration

Return the current ChunkStoreGeneration. This can be used to determine whether the database has been modified since the last time it was queried.

pub fn config(&self) -> &ChunkStoreConfig

See ChunkStoreConfig for more information about configuration.

pub fn iter_chunks(&self) -> impl Iterator<Item = &Arc<Chunk>> + '_

Iterate over all chunks in the store, in ascending [ChunkId] order.

pub fn chunk(&self, id: &ChunkId) -> Option<&Arc<Chunk>>

Get a chunk based on its ID.

pub fn num_chunks(&self) -> usize

Get the number of chunks.

pub fn lookup_datatype( &self, component_name: &ComponentName ) -> Option<&ArrowDataType>

Lookup the latest arrow [ArrowDataType] used by a specific [re_types_core::Component].

pub fn lookup_column_metadata( &self, entity_path: &EntityPath, component_name: &ComponentName ) -> Option<ColumnMetadata>

Lookup the ColumnMetadata for a specific [EntityPath] and [re_types_core::Component].

impl ChunkStore

pub fn from_rrd_filepath( store_config: &ChunkStoreConfig, path_to_rrd: impl AsRef<Path>, version_policy: VersionPolicy ) -> Result<BTreeMap<StoreId, Self>>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

The stores will be prefilled with the data at the specified path.

See also:

• ChunkStore::new

pub fn handle_from_rrd_filepath( store_config: &ChunkStoreConfig, path_to_rrd: impl AsRef<Path>, version_policy: VersionPolicy ) -> Result<BTreeMap<StoreId, ChunkStoreHandle>>

Instantiate a new ChunkStore with the given ChunkStoreConfig.

Wraps the results in ChunkStoreHandles.

The stores will be prefilled with the data at the specified path.

See also:

• ChunkStore::new_handle

impl ChunkStore

pub fn register_subscriber( subscriber: Box<dyn ChunkStoreSubscriber> ) -> ChunkStoreSubscriberHandle

Registers a ChunkStoreSubscriber so it gets automatically notified when data gets added and/or removed to/from a ChunkStore.

Refer to ChunkStoreEvent’s documentation for more information about these events.

Scope

Registered ChunkStoreSubscribers are global scope: they get notified of all events from all existing ChunkStores, including ChunkStores created after the subscriber was registered.

Use ChunkStoreEvent::store_id to identify the source of an event.

Late registration

Subscribers must be registered before a store gets created to guarantee that no events were missed.

ChunkStoreEvent::event_id can be used to identify missing events.

Ordering

The order in which registered subscribers are notified is undefined and will likely become concurrent in the future.

If you need a specific order across multiple subscribers, embed them into an orchestrating subscriber.

pub fn with_subscriber<V: ChunkStoreSubscriber, T, F: FnMut(&V) -> T>( ChunkStoreSubscriberHandle: ChunkStoreSubscriberHandle, f: F ) -> Option<T>

Passes a reference to the downcasted subscriber to the given FnMut callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_once<V: ChunkStoreSubscriber, T, F: FnOnce(&V) -> T>( ChunkStoreSubscriberHandle: ChunkStoreSubscriberHandle, f: F ) -> Option<T>

Passes a reference to the downcasted subscriber to the given FnOnce callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub fn with_subscriber_mut<V: ChunkStoreSubscriber, T, F: FnMut(&mut V) -> T>( ChunkStoreSubscriberHandle: ChunkStoreSubscriberHandle, f: F ) -> Option<T>

Passes a mutable reference to the downcasted subscriber to the given callback.

Returns None if the subscriber doesn’t exist or downcasting failed.

pub(crate) fn on_events(events: &[ChunkStoreEvent])

Called by ChunkStore’s mutating methods to notify subscriber subscribers of upcoming events.

impl ChunkStore

pub fn insert_chunk( &mut self, chunk: &Arc<Chunk> ) -> ChunkStoreResult<Vec<ChunkStoreEvent>>

Inserts a [Chunk] in the store.

Iff the store was modified, all registered subscribers will be notified and the resulting ChunkStoreEvent will be returned, or None otherwise.

• Trying to insert an unsorted chunk ([Chunk::is_sorted]) will fail with an error.
• Inserting a duplicated [ChunkId] will result in a no-op.
• Inserting an empty [Chunk] will result in a no-op.

fn find_and_elect_compaction_candidate( &mut self, chunk: &Arc<Chunk> ) -> Option<Arc<Chunk>>

Finds the most appropriate candidate for compaction.

The algorithm is simple: for each incoming [Chunk], we take a look at its future neighbors. Each neighbor is a potential candidate for compaction.

Because the chunk is going to be inserted into many different indices – for each of its timelines and components – it will have many direct neighbors. Everytime we encounter a neighbor, it earns points.

The neighbor with the most points at the end of the process is elected.

pub fn drop_entity_path( &mut self, entity_path: &EntityPath ) -> Vec<ChunkStoreEvent>

Unconditionally drops all the data for a given entity_path.

Returns the list of Chunks that were dropped from the store in the form of ChunkStoreEvents.

This is not recursive. The store is unaware of the entity hierarchy.

Trait Implementations

impl Clone for ChunkStore

fn clone(&self) -> Self

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for ChunkStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.