Struct vector::sources::file::FileConfig

pub struct FileConfig {

    pub include: Vec<PathBuf>,
    pub exclude: Vec<PathBuf>,
    pub file_key: OptionalValuePath,
    pub start_at_beginning: Option<bool>,
    pub ignore_checkpoints: Option<bool>,
    pub read_from: ReadFromConfig,
    pub ignore_older_secs: Option<u64>,
    pub max_line_bytes: usize,
    pub host_key: Option<OptionalValuePath>,
    pub data_dir: Option<PathBuf>,
    pub offset_key: Option<OptionalValuePath>,
    pub glob_minimum_cooldown_ms: Duration,
    pub ignore_not_found: bool,
    pub message_start_indicator: Option<String>,
    pub multi_line_timeout: u64,
    pub multiline: Option<MultilineConfig>,
    pub max_read_bytes: usize,
    pub oldest_first: bool,
    pub remove_after_secs: Option<u64>,
    pub line_delimiter: String,
    pub encoding: Option<EncodingConfig>,
    pub rotate_wait: Duration,
    /* private fields */
}

Configuration for the file source.

Fields

include: Vec<PathBuf>

Array of file patterns to include. Globbing is supported.

exclude: Vec<PathBuf>

Array of file patterns to exclude. Globbing is supported.

Takes precedence over the include option. Note: The exclude patterns are applied after the attempt to glob everything in include. This means that all files are first matched by include and then filtered by the exclude patterns. This can be impactful if include contains directories with contents that are not accessible.

file_key: OptionalValuePath

Overrides the name of the log field used to add the file path to each event.

The value is the full path to the file where the event was read message.

Set to "" to suppress this key.

start_at_beginning: Option<bool>

Whether or not to start reading from the beginning of a new file.

ignore_checkpoints: Option<bool>

Whether or not to ignore existing checkpoints when determining where to start reading a file.

Checkpoints are still written normally.

read_from: ReadFromConfig

ignore_older_secs: Option<u64>

Ignore files with a data modification date older than the specified number of seconds.

max_line_bytes: usize

The maximum size of a line before it is discarded.

This protects against malformed lines or tailing incorrect files.

host_key: Option<OptionalValuePath>

Overrides the name of the log field used to add the current hostname to each event.

By default, the global log_schema.host_key option is used.

Set to "" to suppress this key.

data_dir: Option<PathBuf>

The directory used to persist file checkpoint positions.

By default, the global data_dir option is used. Make sure the running user has write permissions to this directory.

If this directory is specified, then Vector will attempt to create it.

offset_key: Option<OptionalValuePath>

Enables adding the file offset to each event and sets the name of the log field used.

The value is the byte offset of the start of the line within the file.

Off by default, the offset is only added to the event if this is set.

glob_minimum_cooldown_ms: Duration

The delay between file discovery calls.

This controls the interval at which files are searched. A higher value results in greater chances of some short-lived files being missed between searches, but a lower value increases the performance impact of file discovery.

ignore_not_found: bool

Ignore missing files when fingerprinting.

This may be useful when used with source directories containing dangling symlinks.

message_start_indicator: Option<String>

String value used to identify the start of a multi-line message.

multi_line_timeout: u64

How long to wait for more data when aggregating a multi-line message, in milliseconds.

multiline: Option<MultilineConfig>

Multiline aggregation configuration.

If not specified, multiline aggregation is disabled.

max_read_bytes: usize

Max amount of bytes to read from a single file before switching over to the next file. Note: This does not apply when oldest_first is true.

This allows distributing the reads more or less evenly across the files.

oldest_first: bool

Instead of balancing read capacity fairly across all watched files, prioritize draining the oldest files before moving on to read data from more recent files.

remove_after_secs: Option<u64>

After reaching EOF, the number of seconds to wait before removing the file, unless new data is written.

If not specified, files are not removed.

line_delimiter: String

String sequence used to separate one file line from another.

encoding: Option<EncodingConfig>

rotate_wait: Duration

How long to keep an open handle to a rotated log file. The default value represents “no limit”

Trait Implementations

impl Clone for FileConfig

fn clone(&self) -> FileConfig

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Configurable for FileConfig

fn referenceable_name() -> Option<&'static str>

Gets the referenceable name of this value, if any.

fn metadata() -> Metadata

Gets the metadata for this value.

fn generate_schema( schema_gen: &RefCell<SchemaGenerator>, ) -> Result<SchemaObject, GenerateError>

Generates the schema for this value.

fn is_optional() -> bool

where Self: Sized,

Whether or not this value is optional.

fn validate_metadata(_metadata: &Metadata) -> Result<(), GenerateError>

where Self: Sized,

Validates the given metadata against this type.

fn as_configurable_ref() -> ConfigurableRef

where Self: Sized + 'static,

Create a new configurable reference table.

impl Debug for FileConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for FileConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for FileConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl GenerateConfig for FileConfig

fn generate_config() -> Value

impl NamedComponent for FileConfig

fn get_component_name(&self) -> &'static str

Gets the name of the component.

impl PartialEq for FileConfig

fn eq(&self, other: &FileConfig) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for FileConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl SourceConfig for FileConfig

fn build<'life0, 'async_trait>( &'life0 self, cx: SourceContext, ) -> Pin<Box<dyn Future<Output = Result<Source>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Builds the source with the given context.

fn outputs(&self, global_log_namespace: LogNamespace) -> Vec<SourceOutput>

Gets the list of outputs exposed by this source.

fn can_acknowledge(&self) -> bool

Whether or not this source can acknowledge the events it emits.

fn resources(&self) -> Vec<Resource>

Gets the list of resources, if any, used by this source.

impl ToValue for FileConfig

fn to_value(&self) -> Value

Convert this value into a serde_json::Value. Must not fail.

impl Eq for FileConfig

impl StructuralPartialEq for FileConfig