A centralized Logging Service platform consists of two major components: Log Collection and Log Aggregation. The responsibilities of the platform provider and the user differ in the context of Logging Service.

Log Collection: The responsibility for log collection and its configuration lies with the user. This involves setting up mechanisms to gather log data from various sources within the infrastructure and applications. These mechanisms can include agents, log shippers, or APIs that send log data to a central location for storage and analysis.

Log Aggregation: The Logging Service platform provider provides and manages the log aggregation component. This component involves the centralization of log data from multiple sources, making it accessible for analysis and visualization. The platform handles log storage, indexing, and search functionalities.

Log Agent

Logs must be targeted and collected to be sent to the Logging Service platform for aggregation and analysis. Log agents responsible for collecting and forwarding logs to the central logging platform typically facilitate this process.

While various log agents are available, the Logging Service platform supports the . Fluent Bit is a lightweight and efficient log forwarder that can be installed on Linux, macOS, and Windows systems. For more information, see . It provides the necessary functionality to collect logs from different sources and push them to the Logging Service platform for further processing and analysis.

Fluent Bit Configuration

To ensure that the logs are shipped correctly and securely, ensure that you configure the following appropriately in Fluent Bit:

• Log Server Endpoint: It refers to the address of your logging pipeline, where the logs will be sent after they are collected. You can obtain this endpoint from the REST API response.

• Tag: To ensure an appropriate synchronization between the agent and the log server, configure a tag in the Fluent Bit log agent. It can be utilized for reporting purposes and aids in identifying and categorizing the logs.

• Key: In addition to the TLS connection, Fluent Bit needs a

Here is an example of a Fluent Bit configuration that needs an endpoint, a tag, and a key:

Scenario 1: Streamlined log centralization and analysis

Precondition

Maintaining log visibility is challenging in a distributed system with numerous services and environments. Logs are scattered across different nodes, making troubleshooting, performance monitoring, and security analysis complex tasks.

Solution

Logging Service offers a streamlined approach to centralizing logs. Easily configure agents on your log sources to collect and forward logs to a central repository. These logs are securely transferred, efficiently stored, and indexed for analysis. You can create custom dashboards for real-time log visualization and analysis, helping you quickly detect and address issues, maintain security standards, and optimize your application's performance.

Scenario 2: Flexible log retention and storage

Precondition

Effectively managing log retention and storage is a critical operational challenge. Storing logs for an extended period can be costly and cumbersome, while inadequate retention policies may lead to losing valuable historical data needed for compliance, auditing, and troubleshooting.

Solution

Logging Service offers flexibility in log retention and storage. Configure retention policies to remove logs based on your organization's requirements automatically. It ensures you retain logs for as long as necessary without incurring high storage costs. Additionally, you can use the service to search and retrieve older logs when needed, simplifying compliance audits and historical analysis.

Scenario 3: Real-time log monitoring during development and deployment

Precondition

DevOps teams rely on real-time visibility into application logs during the development and deployment phases. Timely and continuous access to logs is essential for debugging, identifying issues, and ensuring smooth deployments.

Solution

Logging Service provides DevOps teams with near-real-time log monitoring capabilities. Integrate the service seamlessly into your development and deployment pipelines to capture and analyze logs as applications are built and deployed. As your application components interact and generate logs, Logging Service immediately ingests and makes them available for analysis. DevOps teams can set up alerts and notifications based on specific log events, ensuring rapid response to critical issues. This near-real-time log monitoring helps streamline development and deployment processes, reduces downtime, and ensures the successful release of reliable applications.

Overview

The Usage Reporter gathers two usage numbers for every logging pipeline you have configured and sends them to the Mimir data source in your Grafana Dashboard.

Benefits

• Visibility: See per-pipeline bytes and record counts in Grafana for quick monitoring and investigation.

• Alerting & optimization: Use the metrics for setting alerts and optimizing ingestion.

Metrics

As the ionos_usage_logs_bytes_total and ionos_usage_logs_counts_total metrics are shown per pipeline, the label pipeline_id="<your‑pipeline‑uuid>" lets you separate the values in Grafana.

View the metrics

In your Grafana dashboard, go to Explore → select Mimir as the datasource. Select ionos_usage_logs_bytes_total or ionos_usage_logs_counts_total from the Metric drop-down list and click Run query to view the metrics.

This section lists the key features and benefits of using the Logging Service.

Features

• Scalability: It is designed to handle extensive volumes of logs and scale up or down according to needs.

• Availability: The logs are available 24/7 and can be accessed from anywhere using a web-based interface.

• Security: The logs are secured using necessary encryption, access controls, and audit trails.

• Customization: The service can be customized according to your needs. For example, defining log retention periods, setting up alerts, or creating custom dashboards.

• Sub-user Management: The Logging Service allows the primary account owner to create sub-users and delegate pipeline management responsibilities.

Benefits

• Reduced Costs: It eliminates the need to invest in hardware and software for log management. Also, you only pay for the services utilized; thus, resulting in significant cost savings.

• Increased Efficiency: It automates log management tasks such as log collection, storage, and analysis; thus, reducing the time and effort required to manage logs manually.

• Improved Troubleshooting: It provides real-time access to log data, which facilitates speedy problem identification and resolution.

The Logging Service is a versatile and accessible platform that allows you to conveniently store and manage logs from various sources. This platform offers a seamless log aggregation solution for logs generated within the IONOS Cloud infrastructure, by your bare metal system, or another cloud environment. With its flexibility, you can effortlessly push logs from anywhere, ensuring comprehensive log monitoring and analysis.

The following two encryption mechanisms safeguard all HTTP or TCP communications that push logs:

• Transport Layer Security (TLS)

Logging Service is a cloud-based service that allows you to store, manage, and analyze logs generated by your applications or systems. It is a scalable and cost-effective solution for organizations that need to manage and analyze large volumes of logs.

Logging Service is also an efficient log management solution, which contains several important features and benefits that can help organizations improve their log management practices and troubleshoot issues more quickly.

Log data is the key to an early detection of security incidents; thus, using the Logging Service can improve your operational efficiency, enhance your security, and increase your visibility into your system's performance and errors. Log data is available 24/7, secure, high-performance, and scalable. Your data is encrypted before, during, and after it is transmitted. The Fluent Bit agent gathers log data locally, while the Logging Service forms the basis for analysis and better visualization of your logs with increased security.

The managed Grafana in the Logging Service comes with a pre-configured datasource for the Telemetry API called IONOS Telemetry. You can use this datasource to query metrics from the IONOS Cloud Telemetry API. For more information, see Integration with IONOS Telemetry API.

Logging Service also enables you to configure an unlimited log retention period for your logs. For more information, see .

Components of Logging Service

The architecture of the Logging Service includes the following three main components that can be used to aggregate logs from various sources, analyze the gathered logs, and create visualizations for monitoring and report generation.

• Data Collection: Data is collected from various log sources, such as applications, servers, and network devices, which are sent to a centralized Logging Service platform. Currently, the following log sources are supported: Kubernetes, Docker, Linux Systemd, HTTP (JSON REST API), and Generic.

• Logging Service Platform: The Logging Service platform stores the logs gathered from various sources in a centralized location for easy access. This data can be accessed for analysis and troubleshooting. The platform includes log search, routing, storage, analytics, and visualization features. For more information, see .

The illustration shows the default components of the Logging Service platform and the following is a brief description of the components:

• Systems (1, 2, and n): These are the various log sources with Fluent Bit installed to gather, parse, and redirect logs to the Logging Service platform.

• Logging Service Platform:

  • It contains an in-built log aggregator, Fluentd, easily compatible with various sources and targets. Hence, shipping logs from the source to the destination is hassle-free. It aggregates, processes, and ships data to the stipulated target.

Limitations

The following are the key limitations of Logging Service:

A log pipeline refers to an instance or configuration of the Logging Service you can create using the DCD or the API. To create an instance of the Logging Service, you can request the designated regional endpoint based on your desired location:

• Berlin: https://logging.de-txl.ionos.com/pipelines

• Frankfurt: https://logging.de-fra.ionos.com/pipelines

• London: https://logging.gb-lhr.ionos.com/pipelines

• Worcester: https://logging.gb-bhx.ionos.com/pipelines

• Paris: https://logging.fr-par.ionos.com/pipelines

• Logroño: https://logging.es-vit.ionos.com/pipelines

• Lenexa: https://logging.us-mci.ionos.com/pipelines

• Las Vegas: https://logging.us-las.ionos.com/pipelines

• Newark: https://logging.us-ewr.ionos.com/pipelines

When creating a log pipeline instance, you can define multiple log streams within each pipeline. Each stream functions as a separate log source, allowing you to organize and manage different sources of logs within your logging system.

To differentiate the log sources and enable effective reporting, it is necessary to provide a unique tag for each log source within the pipeline instance. The tag serves as an identifier or label for the log source, allowing you to distinguish and track the logs from different sources easily.

After the pipeline is set up, a unique endpoint is assigned to each pipeline, thus establishing a connection, either HTTP or TCP, with an independent log server. This endpoint serves as the designated destination for sending logs generated by all the log sources within the pipeline. However, to ensure proper categorization and differentiation, each pipeline configuration log source must utilize its designated tag. By adhering to this practice, the logs generated by each source can be accurately identified and traced, even when they are directed to the same endpoint.

Limitations

The Logging Service platform imposes specific limitations on the number of pipelines you can create and the log rate you can send to the log server.

These limitations are determined by your pricing plan and are designed to ensure that all users receive optimal performance and resource allocation.

Ingestion Rate

By default, the platform sets an ingestion rate limit of 50 HTTP requests per second for each pipeline to prevent overloading the log server with excessive log data.

It is essential to identify the origin of the logs before choosing the right approach to installing and configuring the Fluent Bit agent. However, to provide convenient parsing and data labeling, IONOS Cloud accepts logs from the following four log sources: Kubernetes, Docker, Linux Systemd, and HTTP. The configuration of these log sources varies accordingly.

Kubernetes

This method lets you collect and ship your Kubernetes application's logs. offers a wide range of information on how to set it up on your Kubernetes cluster. However, we also recommend you try our before configuring the log source.

Docker

If you have a set of applications on Docker, we recommend trying our . You can also find more information about Docker configuration on .

Linux Systemd

To set up Fluent Bit on a Linux system with systemd or journals, you must install an appropriate package for your Linux distribution. For more information about how to accomplish it, see . We also recommend you try our sample configuration.

HTTP

You can also send logs through the HTTP REST endpoint. You can transfer logs that are in JSON format only via the HTTP REST endpoint.

The following is an example: