Struct vector::sinks::azure_blob::AzureBlobSinkConfig

pub struct AzureBlobSinkConfig {
    pub connection_string: Option<SensitiveString>,
    pub storage_account: Option<String>,
    pub endpoint: Option<String>,
    pub blob_prefix: Template,
    pub blob_time_format: Option<String>,
    pub blob_append_uuid: Option<bool>,
    pub encoding: EncodingConfigWithFraming,
    pub compression: Compression,
    pub batch: BatchConfig<BulkSizeBasedDefaultBatchSettings>,
    pub request: TowerRequestConfig<AzureBlobTowerRequestConfigDefaults>,
    /* private fields */
}

Configuration for the azure_blob sink.

Fields

connection_string: Option<SensitiveString>

The Azure Blob Storage Account connection string.

Authentication with an access key or shared access signature (SAS) are supported authentication methods. If using a non-account SAS, healthchecks will fail and will need to be disabled by setting healthcheck.enabled to false for this sink

When generating an account SAS, the following are the minimum required option settings for Vector to access blob storage and pass a health check.

Option	Value
Allowed services	Blob
Allowed resource types	Container & Object
Allowed permissions	Read & Create

Either storage_account, or this field, must be specified.

storage_account: Option<String>

The Azure Blob Storage Account name.

Attempts to load credentials for the account in the following ways, in order:

• read from environment variables (more information)
• looks for a Managed Identity
• uses the az CLI tool to get an access token (more information)

Either connection_string, or this field, must be specified.

endpoint: Option<String>

The Azure Blob Storage Endpoint URL.

This is used to override the default blob storage endpoint URL in cases where you are using credentials read from the environment/managed identities or access tokens without using an explicit connection_string (which already explicitly supports overriding the blob endpoint URL).

This may only be used with storage_account and is ignored when used with connection_string.

blob_prefix: Template

A prefix to apply to all blob keys.

Prefixes are useful for partitioning objects, such as by creating a blob key that stores blobs under a particular directory. If using a prefix for this purpose, it must end in / to act as a directory path. A trailing / is not automatically added.

blob_time_format: Option<String>

The timestamp format for the time component of the blob key.

By default, blob keys are appended with a timestamp that reflects when the blob are sent to Azure Blob Storage, such that the resulting blob key is functionally equivalent to joining the blob prefix with the formatted timestamp, such as date=2022-07-18/1658176486.

This would represent a blob_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022 20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders timestamps in seconds since the Unix epoch.

Supports the common strftime specifiers found in most languages.

When set to an empty string, no timestamp is appended to the blob prefix.

blob_append_uuid: Option<bool>

Whether or not to append a UUID v4 token to the end of the blob key.

The UUID is appended to the timestamp portion of the object key, such that if the blob key generated is date=2022-07-18/1658176486, setting this field to true results in an blob key that looks like date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.

This ensures there are no name collisions, and can be useful in high-volume workloads where blob keys must be unique.

encoding: EncodingConfigWithFraming

compression: Compression

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Some cloud storage API clients and browsers handle decompression transparently, so depending on how they are accessed, files may not always appear to be compressed.

batch: BatchConfig<BulkSizeBasedDefaultBatchSettings>

request: TowerRequestConfig<AzureBlobTowerRequestConfigDefaults>

Implementations

impl AzureBlobSinkConfig

pub fn build_processor( &self, client: Arc<ContainerClient>, ) -> Result<VectorSink>

pub fn key_partitioner(&self) -> Result<KeyPartitioner>

Trait Implementations

impl Clone for AzureBlobSinkConfig

fn clone(&self) -> AzureBlobSinkConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Configurable for AzureBlobSinkConfig

fn referenceable_name() -> Option<&'static str>

Gets the referenceable name of this value, if any.

fn metadata() -> Metadata

Gets the metadata for this value.

fn generate_schema( schema_gen: &RefCell<SchemaGenerator>, ) -> Result<SchemaObject, GenerateError>

Generates the schema for this value.

fn is_optional() -> bool

where Self: Sized,

Whether or not this value is optional.

fn validate_metadata(_metadata: &Metadata) -> Result<(), GenerateError>

where Self: Sized,

Validates the given metadata against this type.

fn as_configurable_ref() -> ConfigurableRef

where Self: Sized + 'static,

Create a new configurable reference table.

impl Debug for AzureBlobSinkConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for AzureBlobSinkConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl GenerateConfig for AzureBlobSinkConfig

fn generate_config() -> Value

impl NamedComponent for AzureBlobSinkConfig

fn get_component_name(&self) -> &'static str

Gets the name of the component.

impl Serialize for AzureBlobSinkConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl SinkConfig for AzureBlobSinkConfig

fn build<'life0, 'async_trait>( &'life0 self, _cx: SinkContext, ) -> Pin<Box<dyn Future<Output = Result<(VectorSink, Healthcheck)>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Builds the sink with the given context.

fn input(&self) -> Input

Gets the input configuration for this sink.

fn acknowledgements(&self) -> &AcknowledgementsConfig

Gets the acknowledgements configuration for this sink.

fn resources(&self) -> Vec<Resource>

Gets the list of resources, if any, used by this sink.

impl ToValue for AzureBlobSinkConfig

fn to_value(&self) -> Value

Convert this value into a serde_json::Value. Must not fail.