Log Collection
A centralized logging platform consists of two major components: Log Collection and Log Aggregation. In the context of the Logging Service, it's important to clarify the responsibilities of the platform provider and the user.
Log Collection: The responsibility for log collection and its configuration lies with the user. This involves setting up mechanisms to gather log data from various sources within the infrastructure and applications. These mechanisms can include agents, log shippers, or APIs that send log data to a central location for storage and analysis.
Log Aggregation: The Logging Service platform provider provides and manages the log aggregation component. This component involves the centralization of log data from multiple sources, making it accessible for analysis and visualization. The platform handles log storage, indexing, and search functionalities.
Log Agent
Logs must be targeted and collected to be sent to a Logging Service platform for aggregation and analysis. Log agents responsible for collecting and forwarding logs to the central logging platform typically facilitate this process.
While various log agents are available, it's mentioned that the supported log agent for the Logging Service platform in question is FluentBit Log Agent. FluentBit is a lightweight and efficient log forwarder that can be installed on Linux, macOS, and Windows systems. It provides the necessary functionality to collect logs from different sources and push them to the Logging Service platform for further processing and analysis.
FluentBit can be installed on Linux, macOS, and Windows. For more information, see FluentBit's official website.
Note that FluentBit installation and configuration is depending on your log sources.
Ensure you follow the instructions provided by the Logging Service platform provider and refer to any additional documentation or guidelines they may offer for integrating FluentBit Log Agent into your logging infrastructure.
FluentBit Configuration
When configuring FluentBit for log shipping, certain pieces of information need to be properly configured to ensure the logs are shipped correctly and securely.
Log Server Endpoint
The Log Server Endpoint refers to the address of your logging pipeline, where the logs will be sent after they are collected. This endpoint can be obtained from the REST API response.
Tag
The Tag is a piece of information that must be configured in the log agent (FluentBit) to ensure synchronization between the agent and the log server. It helps identify and categorize the logs and can also be used for reporting purposes.
Key
In addition to the TLS connection, FluentBit needs to be configured with a Key (SharedKey) for authentication purposes. This key ensures that only authorized logs are sent to the logging pipeline. The token can be obtained via our REST API.
Here is an example of a FluentBit configuration that needs an Endpoint, Tag, and Key:
Note: Any data masking or sanitization must happen on the client's side.
Last updated