Struct re_dataframe::query::QueryHandle

pub struct QueryHandle<E: StorageEngineLike> {
    pub(crate) engine: E,
    pub(crate) query: QueryExpression,
    state: OnceLock<QueryHandleState>,
}

A handle to a dataframe query, ready to be executed.

Cheaply created via QueryEngine::query.

See QueryHandle::next_row or QueryHandle::into_iter.

Fields

engine: E

Handle to the QueryEngine.

query: QueryExpression

The original query expression used to instantiate this handle.

state: OnceLock<QueryHandleState>

Internal private state. Lazily computed.

It is important that handles stay cheap to create.

Implementations

impl<E: StorageEngineLike> QueryHandle<E>

pub(crate) fn new(engine: E, query: QueryExpression) -> Self

impl<E: StorageEngineLike> QueryHandle<E>

fn init(&self) -> &QueryHandleState

Lazily initialize internal private state.

It is important that query handles stay cheap to create.

fn init_(&self, store: &ChunkStore, cache: &QueryCache) -> QueryHandleState

fn compute_user_selection( &self, view_contents: &[ColumnDescriptor], selection: &[ColumnSelector] ) -> Vec<(usize, ColumnDescriptor)>

fn fetch_view_chunks( &self, store: &ChunkStore, cache: &QueryCache, query: &RangeQuery, view_contents: &[ColumnDescriptor] ) -> (Option<usize>, Vec<Vec<(AtomicU64, Chunk)>>)

fn fetch_clear_chunks( &self, store: &ChunkStore, cache: &QueryCache, query: &RangeQuery, view_contents: &[ColumnDescriptor] ) -> IntMap<EntityPath, Vec<Chunk>>

Returns all potentially relevant clear [Chunk]s for each unique entity path in the view contents.

These chunks take recursive clear semantics into account and are guaranteed to be properly densified. The component data is stripped out, only the indices are left.

fn fetch_chunks<const N: usize>( &self, _store: &ChunkStore, cache: &QueryCache, query: &RangeQuery, entity_path: &EntityPath, component_names: [ComponentName; N] ) -> Option<Vec<(AtomicU64, Chunk)>>

pub fn query(&self) -> &QueryExpression

The query used to instantiate this handle.

pub fn view_contents(&self) -> &[ColumnDescriptor]

Describes the columns that make up this view.

See [QueryExpression::view_contents].

pub fn selected_contents(&self) -> &[(usize, ColumnDescriptor)]

Describes the columns that make up this selection.

The extra usize is the index in Self::view_contents that this selection points to.

See [QueryExpression::selection].

pub fn schema(&self) -> &ArrowSchema

All results returned by this handle will strictly follow this Arrow schema.

Columns that do not yield any data will still be present in the results, filled with null values.

pub fn seek_to_row(&self, row_idx: usize)

Advance all internal cursors so that the next row yielded will correspond to row_idx.

Does nothing if row_idx is out of bounds.

Concurrency

Cursors are implemented using atomic variables, which means calling any of the seek_* while iteration is concurrently ongoing is memory-safe but logically undefined racy behavior. Be careful.

Performance

This requires going through every chunk once, and for each chunk running a binary search if the chunk’s time range contains the index_value.

I.e.: it’s pretty cheap already.

fn seek_to_index_value(&self, index_value: IndexValue)

Advance all internal cursors so that the next row yielded will correspond to index_value.

If index_value isn’t present in the dataset, this seeks to the first index value available past that point, if any.

Concurrency

Cursors are implemented using atomic variables, which means calling any of the seek_* while iteration is concurrently ongoing is memory-safe but logically undefined racy behavior. Be careful.

Performance

This requires going through every chunk once, and for each chunk running a binary search if the chunk’s time range contains the index_value.

I.e.: it’s pretty cheap already.

pub fn num_rows(&self) -> u64

How many rows of data will be returned?

The number of rows depends and only depends on the view contents. The selected contents has no influence on this value.

pub fn next_row(&self) -> Option<Vec<Box<dyn ArrowArray>>>

Returns the next row’s worth of data.

The returned vector of Arrow arrays strictly follows the schema specified by Self::schema. Columns that do not yield any data will still be present in the results, filled with null values.

Each cell in the result corresponds to the latest locally known value at that particular point in the index, for each respective ColumnDescriptor. See [QueryExpression::sparse_fill_strategy] to go beyond local resolution.

Example:

while let Some(row) = query_handle.next_row() {
    // …
}

Pagination

Use Self::seek_to_row:

query_handle.seek_to_row(42);
for row in query_handle.into_iter().take(len) {
    // …
}

pub fn next_row_async( &self ) -> impl Future<Output = Option<Vec<Box<dyn ArrowArray>>>>

where E: 'static + Send + Clone,

Asynchronously returns the next row’s worth of data.

The returned vector of Arrow arrays strictly follows the schema specified by Self::schema. Columns that do not yield any data will still be present in the results, filled with null values.

Each cell in the result corresponds to the latest locally known value at that particular point in the index, for each respective ColumnDescriptor. See [QueryExpression::sparse_fill_strategy] to go beyond local resolution.

Example:

while let Some(row) = query_handle.next_row_async().await {
    // …
}

pub fn _next_row( &self, store: &ChunkStore, cache: &QueryCache ) -> Option<Vec<Box<dyn ArrowArray>>>

pub fn next_row_batch(&self) -> Option<RecordBatch>

Calls Self::next_row and wraps the result in a RecordBatch.

Only use this if you absolutely need a RecordBatch as this adds a lot of allocation overhead.

See Self::next_row for more information.

pub async fn next_row_batch_async(&self) -> Option<RecordBatch>

where E: 'static + Send + Clone,

impl<E: StorageEngineLike> QueryHandle<E>

pub fn iter(&self) -> impl Iterator<Item = Vec<Box<dyn ArrowArray>>> + '_

Returns an iterator backed by Self::next_row.

pub fn into_iter(self) -> impl Iterator<Item = Vec<Box<dyn ArrowArray>>>

Returns an iterator backed by Self::next_row.

pub fn batch_iter(&self) -> impl Iterator<Item = RecordBatch> + '_

Returns an iterator backed by Self::next_row_batch.

pub fn into_batch_iter(self) -> impl Iterator<Item = RecordBatch>

Returns an iterator backed by Self::next_row_batch.