Log Collection

A centralized Logging Service platform consists of two major components: Log Collection and Log Aggregation. The responsibilities of the platform provider and the user differ in the context of Logging Service.

Log Collection: The responsibility for log collection and its configuration lies with the user. This involves setting up mechanisms to gather log data from various sources within the infrastructure and applications. These mechanisms can include agents, log shippers, or APIs that send log data to a central location for storage and analysis.

Log Aggregation: The Logging Service platform provider provides and manages the log aggregation component. This component involves the centralization of log data from multiple sources, making it accessible for analysis and visualization. The platform handles log storage, indexing, and search functionalities.

Log Agent

Logs must be targeted and collected to be sent to the Logging Service platform for aggregation and analysis. Log agents responsible for collecting and forwarding logs to the central logging platform typically facilitate this process.

While various log agents are available, the Logging Service platform supports the Fluent Bit Log Agent. Fluent Bit is a lightweight and efficient log forwarder that can be installed on Linux, macOS, and Windows systems. For more information, see Fluent Bit's official website. It provides the necessary functionality to collect logs from different sources and push them to the Logging Service platform for further processing and analysis.

Note:

• Fluent Bit installation and configuration vary based on your Log Sources.

• Ensure you follow the instructions provided by the Logging Service platform provider and refer to any additional documentation or guidelines they may offer for integrating Fluent Bit log agent into your logging infrastructure.

Fluent Bit Configuration

To ensure that the logs are shipped correctly and securely, ensure that you configure the following appropriately in Fluent Bit:

• Log Server Endpoint: It refers to the address of your logging pipeline, where the logs will be sent after they are collected. You can obtain this endpoint from the REST API response.

• Tag: To ensure an appropriate synchronization between the agent and the log server, configure a tag in the Fluent Bit log agent. It can be utilized for reporting purposes and aids in identifying and categorizing the logs.

• Key: In addition to the TLS connection, Fluent Bit needs a Shared_Key configuration for authentication purposes. This key ensures that only authorized logs are sent to the logging pipeline. You can obtain a token via the REST API.

Here is an example of a Fluent Bit configuration that needs an endpoint, a tag, and a key:

[OUTPUT]
    Name            forward
    Match           *
    Port            9000
    Tag             <TAG>
    Host            <TCP_ENDPOINT>
    tls             on
    Shared_Key      <KEY>

Note: The user must perform any data masking or sanitization.