Struct rerun::ChunkStoreConfig

pub struct ChunkStoreConfig {
    pub enable_changelog: bool,
    pub chunk_max_bytes: u64,
    pub chunk_max_rows: u64,
    pub chunk_max_rows_if_unsorted: u64,
}

Everything needed to build custom ChunkStoreSubscribers.

Fields

enable_changelog: bool

If true (the default), the store will emit events when its contents are modified in any way (insertion, GC), that can be subscribed to.

Leaving this disabled can lead to major performance improvements on the ingestion path in some workloads, provided that the subscribers aren’t needed (e.g. headless mode).

chunk_max_bytes: u64

What is the threshold, in bytes, after which a Chunk cannot be compacted any further?

This is a multi-dimensional trade-off:

• Larger chunks lead to less fixed overhead introduced by metadata, indices and such. Good.
• Larger chunks lead to slower query execution on some unhappy paths. Bad.
• Larger chunks lead to slower and slower compaction as chunks grow larger. Bad.
• Larger chunks lead to coarser garbage collection. Good or bad depending on use case.
• Larger chunks lead to less precision in e.g. the time panel. Bad.

Empirical testing shows that the space overhead gains rapidly diminish beyond ~1000 rows, which is the default row threshold. The default byte threshold is set to 8MiB, which is a reasonable unit of work when e.g. sending chunks over the network.

chunk_max_rows: u64

What is the threshold, in rows, after which a Chunk cannot be compacted any further?

This specifically applies to time-sorted chunks. See also ChunkStoreConfig::chunk_max_rows_if_unsorted.

This is a multi-dimensional trade-off:

• Larger chunks lead to less fixed overhead introduced by metadata, indices and such. Good.
• Larger chunks lead to slower query execution on some unhappy paths. Bad.
• Larger chunks lead to slower and slower compaction as chunks grow larger. Bad.
• Larger chunks lead to coarser garbage collection. Good or bad depending on use case.
• Larger chunks lead to less precision in e.g. the time panel. Bad.

Empirical testing shows that the space overhead gains rapidly diminish beyond ~1000 rows, which is the default row threshold. The default byte threshold is set to 8MiB, which is a reasonable unit of work when e.g. sending chunks over the network.

chunk_max_rows_if_unsorted: u64

What is the threshold, in rows, after which a Chunk cannot be compacted any further?

This specifically applies to non time-sorted chunks. See also ChunkStoreConfig::chunk_max_rows.

This is a multi-dimensional trade-off:

• Larger chunks lead to less fixed overhead introduced by metadata, indices and such. Good.
• Larger chunks lead to slower query execution on some unhappy paths. Bad.
• Larger chunks lead to slower and slower compaction as chunks grow larger. Bad.
• Larger chunks lead to coarser garbage collection. Good or bad depending on use case.
• Larger chunks lead to less precision in e.g. the time panel. Bad.

Empirical testing shows that the space overhead gains rapidly diminish beyond ~1000 rows, which is the default row threshold. The default byte threshold is set to 8MiB, which is a reasonable unit of work when e.g. sending chunks over the network.

Implementations

impl ChunkStoreConfig

pub const DEFAULT: ChunkStoreConfig = _

Default configuration, applicable to most use cases, according to empirical testing.

pub const COMPACTION_DISABLED: ChunkStoreConfig = _

Self::DEFAULT, but with compaction entirely disabled.

pub const CHANGELOG_DISABLED: ChunkStoreConfig = _

Self::DEFAULT, but with changelog disabled.

pub const ALL_DISABLED: ChunkStoreConfig = _

All features disabled.

pub const ENV_STORE_ENABLE_CHANGELOG: &'static str = "RERUN_STORE_ENABLE_CHANGELOG"

Environment variable to configure Self::enable_changelog.

pub const ENV_CHUNK_MAX_BYTES: &'static str = "RERUN_CHUNK_MAX_BYTES"

Environment variable to configure Self::chunk_max_bytes.

pub const ENV_CHUNK_MAX_ROWS: &'static str = "RERUN_CHUNK_MAX_ROWS"

Environment variable to configure Self::chunk_max_rows.

pub const ENV_CHUNK_MAX_ROWS_IF_UNSORTED: &'static str = "RERUN_CHUNK_MAX_ROWS_IF_UNSORTED"

Environment variable to configure Self::chunk_max_rows_if_unsorted.

pub fn from_env() -> Result<ChunkStoreConfig, ChunkStoreError>

Creates a new ChunkStoreConfig using the default values, optionally overridden through the environment.

See Self::apply_env.

pub fn apply_env(&self) -> Result<ChunkStoreConfig, ChunkStoreError>

Returns a copy of self, overriding existing fields with values from the environment if they are present.

See Self::ENV_STORE_ENABLE_CHANGELOG, Self::ENV_CHUNK_MAX_BYTES, Self::ENV_CHUNK_MAX_ROWS and Self::ENV_CHUNK_MAX_ROWS_IF_UNSORTED.

Trait Implementations

impl Clone for ChunkStoreConfig

fn clone(&self) -> ChunkStoreConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ChunkStoreConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for ChunkStoreConfig

fn default() -> ChunkStoreConfig

Returns the “default value” for a type.

impl PartialEq for ChunkStoreConfig

fn eq(&self, other: &ChunkStoreConfig) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for ChunkStoreConfig

impl StructuralPartialEq for ChunkStoreConfig