Struct vector::sinks::aws_s3::S3SinkConfig

pub struct S3SinkConfig {

    pub bucket: String,
    pub key_prefix: String,
    pub filename_time_format: String,
    pub filename_append_uuid: bool,
    pub filename_extension: Option<String>,
    pub options: S3Options,
    pub region: RegionOrEndpoint,
    pub encoding: EncodingConfigWithFraming,
    pub compression: Compression,
    pub batch: BatchConfig<BulkSizeBasedDefaultBatchSettings>,
    pub request: TowerRequestConfig,
    pub tls: Option<TlsConfig>,
    pub auth: AwsAuthentication,
    pub acknowledgements: AcknowledgementsConfig,
    pub timezone: Option<TimeZone>,
}

Configuration for the aws_s3 sink.

Fields

bucket: String

The S3 bucket name.

This must not include a leading s3:// or a trailing /.

key_prefix: String

A prefix to apply to all object keys.

Prefixes are useful for partitioning objects, such as by creating an object key that stores objects under a particular directory. If using a prefix for this purpose, it must end in / to act as a directory path. A trailing / is not automatically added.

filename_time_format: String

The timestamp format for the time component of the object key.

By default, object keys are appended with a timestamp that reflects when the objects are sent to S3, such that the resulting object key is functionally equivalent to joining the key prefix with the formatted timestamp, such as date=2022-07-18/1658176486.

This would represent a key_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022 20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders timestamps in seconds since the Unix epoch.

Supports the common strftime specifiers found in most languages.

When set to an empty string, no timestamp is appended to the key prefix.

filename_append_uuid: bool

Whether or not to append a UUID v4 token to the end of the object key.

The UUID is appended to the timestamp portion of the object key, such that if the object key generated is date=2022-07-18/1658176486, setting this field to true results in an object key that looks like date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.

This ensures there are no name collisions, and can be useful in high-volume workloads where object keys must be unique.

filename_extension: Option<String>

The filename extension to use in the object key.

This overrides setting the extension based on the configured compression.

options: S3Options

region: RegionOrEndpoint

encoding: EncodingConfigWithFraming

compression: Compression

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Some cloud storage API clients and browsers handle decompression transparently, so depending on how they are accessed, files may not always appear to be compressed.

batch: BatchConfig<BulkSizeBasedDefaultBatchSettings>

request: TowerRequestConfig

tls: Option<TlsConfig>

auth: AwsAuthentication

acknowledgements: AcknowledgementsConfig

timezone: Option<TimeZone>

Implementations

impl S3SinkConfig

pub fn build_processor( &self, service: S3Service, cx: SinkContext, ) -> Result<VectorSink>

pub fn build_healthcheck(&self, client: S3Client) -> Result<Healthcheck>

pub async fn create_service(&self, proxy: &ProxyConfig) -> Result<S3Service>

Trait Implementations

impl Clone for S3SinkConfig

fn clone(&self) -> S3SinkConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Configurable for S3SinkConfig

fn referenceable_name() -> Option<&'static str>

Gets the referenceable name of this value, if any.

fn metadata() -> Metadata

Gets the metadata for this value.

fn generate_schema( schema_gen: &RefCell<SchemaGenerator>, ) -> Result<SchemaObject, GenerateError>

Generates the schema for this value.

fn is_optional() -> bool

where Self: Sized,

Whether or not this value is optional.

fn validate_metadata(_metadata: &Metadata) -> Result<(), GenerateError>

where Self: Sized,

Validates the given metadata against this type.

fn as_configurable_ref() -> ConfigurableRef

where Self: Sized + 'static,

Create a new configurable reference table.

impl Debug for S3SinkConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for S3SinkConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl GenerateConfig for S3SinkConfig

fn generate_config() -> Value

impl NamedComponent for S3SinkConfig

fn get_component_name(&self) -> &'static str

Gets the name of the component.

impl Serialize for S3SinkConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl SinkConfig for S3SinkConfig

fn build<'life0, 'async_trait>( &'life0 self, cx: SinkContext, ) -> Pin<Box<dyn Future<Output = Result<(VectorSink, Healthcheck)>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Builds the sink with the given context.

fn input(&self) -> Input

Gets the input configuration for this sink.

fn acknowledgements(&self) -> &AcknowledgementsConfig

Gets the acknowledgements configuration for this sink.

fn resources(&self) -> Vec<Resource>

Gets the list of resources, if any, used by this sink.

impl ToValue for S3SinkConfig

fn to_value(&self) -> Value

Convert this value into a serde_json::Value. Must not fail.