Struct vector::sinks::webhdfs::WebHdfsConfig

pub struct WebHdfsConfig {
    pub root: String,
    pub prefix: String,
    pub endpoint: String,
    pub encoding: EncodingConfigWithFraming,
    pub compression: Compression,
    pub batch: BatchConfig<BulkSizeBasedDefaultBatchSettings>,
    pub acknowledgements: AcknowledgementsConfig,
}

Configuration for the webhdfs sink.

Fields

root: String

The root path for WebHDFS.

Must be a valid directory.

The final file path is in the format of {root}/{prefix}{suffix}.

prefix: String

A prefix to apply to all keys.

Prefixes are useful for partitioning objects, such as by creating a blob key that stores blobs under a particular directory. If using a prefix for this purpose, it must end in / to act as a directory path. A trailing / is not automatically added.

The final file path is in the format of {root}/{prefix}{suffix}.

endpoint: String

An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients.

The endpoint is the HDFS’s web restful HTTP API endpoint.

For more information, see the HDFS Architecture documentation.

encoding: EncodingConfigWithFraming

compression: Compression

batch: BatchConfig<BulkSizeBasedDefaultBatchSettings>

acknowledgements: AcknowledgementsConfig

Implementations

impl WebHdfsConfig

pub fn build_operator(&self) -> Result<Operator>

pub fn build_processor(&self, op: Operator) -> Result<VectorSink>

pub fn key_partitioner(&self) -> Result<KeyPartitioner>

Trait Implementations

impl Clone for WebHdfsConfig

fn clone(&self) -> WebHdfsConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Configurable for WebHdfsConfig

fn referenceable_name() -> Option<&'static str>

Gets the referenceable name of this value, if any.

fn metadata() -> Metadata

Gets the metadata for this value.

fn generate_schema( schema_gen: &RefCell<SchemaGenerator>, ) -> Result<SchemaObject, GenerateError>

Generates the schema for this value.

fn is_optional() -> bool

where Self: Sized,

Whether or not this value is optional.

fn validate_metadata(_metadata: &Metadata) -> Result<(), GenerateError>

where Self: Sized,

Validates the given metadata against this type.

fn as_configurable_ref() -> ConfigurableRef

where Self: Sized + 'static,

Create a new configurable reference table.

impl Debug for WebHdfsConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for WebHdfsConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl GenerateConfig for WebHdfsConfig

fn generate_config() -> Value

impl NamedComponent for WebHdfsConfig

fn get_component_name(&self) -> &'static str

Gets the name of the component.

impl Serialize for WebHdfsConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl SinkConfig for WebHdfsConfig

fn build<'life0, 'async_trait>( &'life0 self, _cx: SinkContext, ) -> Pin<Box<dyn Future<Output = Result<(VectorSink, Healthcheck)>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Builds the sink with the given context.

fn input(&self) -> Input

Gets the input configuration for this sink.

fn acknowledgements(&self) -> &AcknowledgementsConfig

Gets the acknowledgements configuration for this sink.

fn resources(&self) -> Vec<Resource>

Gets the list of resources, if any, used by this sink.

impl ToValue for WebHdfsConfig

fn to_value(&self) -> Value

Convert this value into a serde_json::Value. Must not fail.