Struct file_source::FileServer

pub struct FileServer<PP, E: FileSourceInternalEvents>
where
    PP: PathsProvider,
{

    pub paths_provider: PP,
    pub max_read_bytes: usize,
    pub ignore_checkpoints: bool,
    pub read_from: ReadFrom,
    pub ignore_before: Option<DateTime<Utc>>,
    pub max_line_bytes: usize,
    pub line_delimiter: Bytes,
    pub data_dir: PathBuf,
    pub glob_minimum_cooldown: Duration,
    pub fingerprinter: Fingerprinter,
    pub oldest_first: bool,
    pub remove_after: Option<Duration>,
    pub emitter: E,
    pub handle: Handle,
    pub rotate_wait: Duration,
}

FileServer is a Source which cooperatively schedules reads over files, converting the lines of said files into LogLine structures. As FileServer is intended to be useful across multiple operating systems with POSIX filesystem semantics FileServer must poll for changes. That is, no event notification is used by FileServer.

FileServer is configured on a path to watch. The files do not need to exist at startup. FileServer will discover new files which match its path in at most 60 seconds.

Fields

paths_provider: PP

max_read_bytes: usize

ignore_checkpoints: bool

read_from: ReadFrom

ignore_before: Option<DateTime<Utc>>

max_line_bytes: usize

line_delimiter: Bytes

data_dir: PathBuf

glob_minimum_cooldown: Duration

fingerprinter: Fingerprinter

oldest_first: bool

remove_after: Option<Duration>

emitter: E

handle: Handle

rotate_wait: Duration

Implementations

impl<PP, E> FileServer<PP, E>

where PP: PathsProvider, E: FileSourceInternalEvents,

FileServer as Source

The ‘run’ of FileServer performs the cooperative scheduling of reads over FileServer’s configured files. Much care has been taking to make this scheduling ‘fair’, meaning busy files do not drown out quiet files or vice versa but there’s no one perfect approach. Very fast files will be lost if your system aggressively rolls log files. FileServer will keep a file handler open but should your system move so quickly that a file disappears before FileServer is able to open it the contents will be lost. This should be a rare occurrence.

Specific operating systems support evented interfaces that correct this problem but your intrepid authors know of no generic solution.

pub fn run<C, S1, S2>( self, chans: C, shutdown_data: S1, shutdown_checkpointer: S2, checkpointer: Checkpointer, ) -> Result<Shutdown, <C as Sink<Vec<Line>>>::Error>

where C: Sink<Vec<Line>> + Unpin, <C as Sink<Vec<Line>>>::Error: Error, S1: Future + Unpin + Send + 'static, S2: Future + Unpin + Send + 'static,