Trace an AI Agent that Uses AI Model Hub with the Tracing Service

Overview

This tutorial demonstrates how to trace an AI agent that uses IONOS CLOUD AI Model Hub for inference and export its execution as distributed traces to the IONOS CLOUD Tracing Service (powered by Grafana Tempo). You will build a small Python agent that calls a large language model through the AI Model Hub OpenAI-compatible endpoint, instrument it with the OpenAI OpenTelemetry instrumentation (which emits gen_ai.* span attributes), and view the resulting spans, including the model, token usage, and latency, in Grafana.

This keeps both inference and observability inside IONOS CLOUD: the model runs on AI Model Hub, and the traces stay in your EU-hosted tracing pipeline.

Target audience

This tutorial benefits AI/ML engineers, application developers, and platform teams building LLM-powered applications who want visibility into agent behavior, latency, and cost. Readers benefit from basic familiarity with:

• Python and virtual environments
• LLM/OpenAI-compatible APIs
• OpenTelemetry concepts (spans, exporters)
• The IONOS CLOUD Console and API authentication

What you will learn

• How to create a IONOS CLOUD tracing pipeline and retrieve its ingestion endpoint and key
• How to instrument a Python LLM application with the OpenAI OpenTelemetry instrumentation.
• How to export gen_ai trace spans to your tracing pipeline over the OpenTelemetry Protocol (OTLP) using HTTP.
• How to call a model on AI Model Hub through its OpenAI-compatible endpoint.
• How to explore GenAI traces (model, tokens, prompts, latency) in Grafana with TraceQL.

Before you begin

Ensure you have:

• An active IONOS CLOUD account with the Access and manage Tracing privilege.
• Access to IONOS CLOUD AI Model Hub with a model deployed and its OpenAI-compatible endpoint and API token.
• An IONOS CLOUD API token to create the pipeline. To generate a token, see Token Manager.
• Python 3.9 or later, and outbound HTTPS access on port 443.

Cost considerations

This tutorial creates billable resources: a IONOS CLOUD tracing pipeline (billed by trace data ingested and stored) and AI Model Hub inference usage (billed by token consumption).

Delete the pipeline after you finish, and review your AI Model Hub usage. For current rates, see the IONOS CLOUD price list (EUR).

Architecture

The diagram below shows the data flow:

Architecture: a Python AI agent with OpenTelemetry GenAI instrumentation calls AI Model Hub for chat completion and exports GenAI spans over OTLP/HTTP to the IONOS Tracing pipeline, which ingests and stores them in the Tracing Service (Grafana Tempo) for querying in Grafana.

The agent calls the model on AI Model Hub for inference. The OpenTelemetry instrumentation wraps each model call and produces a gen_ai span, capturing the model, token counts, and (optionally) prompts and responses. These spans are exported to your tracing pipeline and become searchable in Grafana.

Procedure

1

Create a tracing pipeline.

Create a pipeline configured for the otlp-http protocol:

curl --location \
  --request POST 'https://tracing.de-txl.ionos.com/pipelines' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <IONOS_API_TOKEN>' \
  --data '{
    "metadata": {},
    "properties": {
      "name": "ai-agent-tracing",
      "protocol": "otlp-http"
    }
  }'

The create response returns the pipeline id and the ingestion key (key). Save the key immediately: it is returned only once.

Important: Save the key immediately. The Tracing Service returns the ingestion key only once. Store it securely; if it is lost or exposed, rotate it through the Tracing Service API.

The create response does not include the ingestion endpoint. Retrieve the ingestion endpoint (https://<tracing-host>/v1/traces) and the grafanaEndpoint with a follow-up GET request, using the pipeline id:

curl --location \
  --request GET 'https://tracing.de-txl.ionos.com/pipelines/<pipeline-id>' \
  --header 'Authorization: Bearer <IONOS_API_TOKEN>'

2

Set up the Python environment.

Create a virtual environment and install the dependencies:

python3 -m venv .venv && source .venv/bin/activate
pip install openai \
            opentelemetry-sdk \
            opentelemetry-exporter-otlp \
            opentelemetry-instrumentation-openai

Provide the secrets as environment variables. Do not store credentials directly in the script:

export IONOS_TRACING_APIKEY="<tracing pipeline ingestion key>"
export IONOS_MODELHUB_KEY="<AI Model Hub API token>"

Important: Handle tokens like passwords. The AI Model Hub token is a IONOS CLOUD credential. Do not paste it into files, chats, tickets, or version control. If exposed, revoke it immediately in the Token Manager and rotate the tracing pipeline key.

3

Write the instrumented agent.

Create agent_trace.py. Replace the endpoint hosts and model id with your values.

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from openai import OpenAI

# --- Configuration ---
TRACES_ENDPOINT = "https://<tracing-host>/v1/traces"                 # from the pipeline you created
MODELHUB_BASE_URL = "https://openai.inference.<region>.ionos.com/v1" # AI Model Hub OpenAI-compatible endpoint
MODEL = "meta-llama/Llama-3.3-70B-Instruct"                          # a model available on your AI Model Hub

# --- Export gen_ai spans to the Tracing Service ---
provider = TracerProvider(resource=Resource.create({"service.name": "ai-agent-demo"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint=TRACES_ENDPOINT,
    headers={"apikey": os.environ["IONOS_TRACING_APIKEY"]},
)))
trace.set_tracer_provider(provider)

# Auto-instrument the OpenAI client so every model call emits a gen_ai span
OpenAIInstrumentor().instrument()

client = OpenAI(base_url=MODELHUB_BASE_URL, api_key=os.environ["IONOS_MODELHUB_KEY"])
tracer = trace.get_tracer("ai-agent-demo")

# A parent "agent" span wraps the run, so the trace shows the agent and its model call
with tracer.start_as_current_span("agent-run"):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "In one sentence, what is distributed tracing?"},
        ],
        max_tokens=200,
    )
    print(response.choices[0].message.content)

provider.shutdown()   # flush spans before exit

Note: The OpenAI instrumentation captures GenAI attributes such as gen_ai.request.model, gen_ai.usage.input_tokens, and, when input/output recording is activated, gen_ai.prompt.* and gen_ai.completion.* (the actual prompt and response). Recorded prompts and responses may contain sensitive data, so be deliberate before activating this in production.

4

Run the agent.

python3 agent_trace.py

The script prints the model's reply and exports the spans. A macOS LibreSSL/urllib3 warning, if shown, is harmless.

5

View GenAI traces in Grafana.

Open Grafana using the grafanaEndpoint you retrieved, open Explore, and select the Tracing (Tempo) data source. Query for GenAI spans with TraceQL:

{ span.gen_ai.request.model != "" }

Open the agent-run trace and expand the child openai.chat span. The trace shows GenAI attributes such as gen_ai.system, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and the prompt or the completion content when recording is activated.

A GenAI trace in Grafana Tempo: A parent agent span, a child chat completion span, and a span-attributes panel listing the request model, response model, token counts, GenAI system, and prompt content.

Note: Span and service names reflect your own values. The parent span name (agent-run) and service.name (ai-agent-demo) come from the script above, so your trace shows those. The child openai.chat span and its gen_ai.* attributes are emitted by the OpenAI instrumentation. The example is from a validation run, so it shows a different service and span name.

Result: Your AI agent runs now appear as GenAI traces in Grafana. The agent-run trace contains the openai.chat span with gen_ai.* attributes for the model, token usage, latency, and (when recording is activated) the prompt and response content.

Troubleshooting

1. No gen_ai spans: Confirm the pipeline is AVAILABLE, IONOS_TRACING_APIKEY is set, and TRACES_ENDPOINT uses the full .../v1/traces path. provider.shutdown() must run so buffered spans are flushed before the process exits.
2. Authentication errors (401) on ingestion: The apikey header value must match the current pipeline key.
3. Model call fails: Verify the AI Model Hub base URL ends in /v1, the token is valid, and the model id is available on your hub.
4. Doubled endpoint path: If you switch to the OTEL_EXPORTER_OTLP_ENDPOINT base environment variable instead of the explicit endpoint=, drop the /v1/traces suffix; the exporter appends it.

Decommission resources

Delete the tracing pipeline through the API to stop billing, using the pipeline id from the creation response:

curl --location \
  --request DELETE 'https://tracing.de-txl.ionos.com/pipelines/<pipeline-id>' \
  --header 'Authorization: Bearer <IONOS_API_TOKEN>'

Then review your AI Model Hub usage.

Next steps

• Trace n8n Workflow Executions with the Tracing Service
• Tracing Service documentation
• AI Model Hub documentation