> For the complete documentation index, see [llms.txt](https://docs.ionos.com/cloud/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ionos.com/cloud/tutorials/observability/tracing-service/trace-ai-agent-using-ai-model-hub-with-tracing-service.md).
# Trace an AI Agent that Uses AI Model Hub with the Tracing Service
## Overview
This tutorial demonstrates how to trace an AI agent that uses space.vars.ionos\_cloud [AI Model Hub](https://docs.ionos.com/cloud/ai/ai-model-hub) for inference and export its execution as distributed traces to the space.vars.ionos\_cloud Tracing Service (powered by Grafana Tempo). You will build a small Python agent that calls a large language model through the AI Model Hub OpenAI-compatible endpoint, instrument it with the OpenAI OpenTelemetry instrumentation (which emits `gen_ai.*` span attributes), and view the resulting spans, including the model, token usage, and latency, in Grafana.
This keeps both inference and observability inside space.vars.ionos\_cloud: the model runs on AI Model Hub, and the traces stay in your EU-hosted tracing pipeline.
## Target audience
This tutorial benefits AI/ML engineers, application developers, and platform teams building LLM-powered applications who want visibility into agent behavior, latency, and cost. Readers benefit from basic familiarity with:
* Python and virtual environments
* LLM/OpenAI-compatible APIs
* OpenTelemetry concepts (spans, exporters)
* The space.vars.ionos\_cloud Console and API authentication
## What you will learn
* How to create a space.vars.ionos\_cloud tracing pipeline and retrieve its ingestion endpoint and key
* How to instrument a Python LLM application with the OpenAI OpenTelemetry instrumentation.
* How to export `gen_ai` trace spans to your tracing pipeline over the OpenTelemetry Protocol (OTLP) using HTTP.
* How to call a model on AI Model Hub through its OpenAI-compatible endpoint.
* How to explore GenAI traces (model, tokens, prompts, latency) in Grafana with TraceQL.
## Before you begin
Ensure you have:
* An active space.vars.ionos\_cloud account with the **Access and manage Tracing** privilege.
* Access to space.vars.ionos\_cloud [AI Model Hub](https://docs.ionos.com/cloud/ai/ai-model-hub) with a model deployed and its OpenAI-compatible endpoint and API token.
* An space.vars.ionos\_cloud API token to create the pipeline. To generate a token, see [Token Manager](https://docs.ionos.com/cloud/set-up-ionos-cloud/management/identity-access-management/token-manager).
* Python 3.9 or later, and outbound `HTTPS` access on port `443`.
## Cost considerations
This tutorial creates billable resources: a space.vars.ionos\_cloud tracing pipeline (billed by trace data ingested and stored) and AI Model Hub inference usage (billed by token consumption).
Delete the pipeline after you finish, and review your AI Model Hub usage. For current rates, see the [IONOS CLOUD price list (EUR)](https://docs.ionos.com/cloud/support/general-information/price-list/ionos-cloud-eur-en).
## Architecture
The diagram below shows the data flow:
The agent calls the model on AI Model Hub for inference. The OpenTelemetry instrumentation wraps each model call and produces a `gen_ai` span, capturing the model, token counts, and (optionally) prompts and responses. These spans are exported to your tracing pipeline and become searchable in Grafana.
## Procedure
{% stepper %}
{% step %}
**Create a tracing pipeline.**
Create a pipeline configured for the `otlp-http` protocol:
```bash
curl --location \
--request POST 'https://tracing.de-txl.ionos.com/pipelines' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ' \
--data '{
"metadata": {},
"properties": {
"name": "ai-agent-tracing",
"protocol": "otlp-http"
}
}'
```
The create response returns the pipeline **`id`** and the **ingestion key** (`key`). Save the key immediately: it is returned only once.
{% hint style="warning" %}
**Important:** Save the key immediately. The Tracing Service returns the ingestion key only once. Store it securely; if it is lost or exposed, rotate it through the [Tracing Service API](https://api.ionos.com/docs/tracing/v1/).
{% endhint %}
The create response does not include the ingestion endpoint. Retrieve the **ingestion endpoint** (`https:///v1/traces`) and the **`grafanaEndpoint`** with a follow-up `GET` request, using the pipeline `id`:
```bash
curl --location \
--request GET 'https://tracing.de-txl.ionos.com/pipelines/' \
--header 'Authorization: Bearer '
```
{% endstep %}
{% step %}
**Set up the Python environment.**
Create a virtual environment and install the dependencies:
```bash
python3 -m venv .venv && source .venv/bin/activate
pip install openai \
opentelemetry-sdk \
opentelemetry-exporter-otlp \
opentelemetry-instrumentation-openai
```
Provide the secrets as environment variables. Do not store credentials directly in the script:
```bash
export IONOS_TRACING_APIKEY=""
export IONOS_MODELHUB_KEY=""
```
{% hint style="warning" %}
**Important:** Handle tokens like passwords. The AI Model Hub token is a space.vars.ionos\_cloud credential. Do not paste it into files, chats, tickets, or version control. If exposed, revoke it immediately in the [Token Manager](https://docs.ionos.com/cloud/set-up-ionos-cloud/management/identity-access-management/token-manager) and rotate the tracing pipeline key.
{% endhint %}
{% endstep %}
{% step %}
**Write the instrumented agent.**
Create `agent_trace.py`. Replace the endpoint hosts and model id with your values.
```python
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from openai import OpenAI
# --- Configuration ---
TRACES_ENDPOINT = "https:///v1/traces" # from the pipeline you created
MODELHUB_BASE_URL = "https://openai.inference..ionos.com/v1" # AI Model Hub OpenAI-compatible endpoint
MODEL = "meta-llama/Llama-3.3-70B-Instruct" # a model available on your AI Model Hub
# --- Export gen_ai spans to the Tracing Service ---
provider = TracerProvider(resource=Resource.create({"service.name": "ai-agent-demo"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
endpoint=TRACES_ENDPOINT,
headers={"apikey": os.environ["IONOS_TRACING_APIKEY"]},
)))
trace.set_tracer_provider(provider)
# Auto-instrument the OpenAI client so every model call emits a gen_ai span
OpenAIInstrumentor().instrument()
client = OpenAI(base_url=MODELHUB_BASE_URL, api_key=os.environ["IONOS_MODELHUB_KEY"])
tracer = trace.get_tracer("ai-agent-demo")
# A parent "agent" span wraps the run, so the trace shows the agent and its model call
with tracer.start_as_current_span("agent-run"):
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "In one sentence, what is distributed tracing?"},
],
max_tokens=200,
)
print(response.choices[0].message.content)
provider.shutdown() # flush spans before exit
```
{% hint style="info" %}
**Note:** The OpenAI instrumentation captures GenAI attributes such as `gen_ai.request.model`, `gen_ai.usage.input_tokens`, and, when input/output recording is activated, `gen_ai.prompt.*` and `gen_ai.completion.*` (the actual prompt and response). Recorded prompts and responses may contain sensitive data, so be deliberate before activating this in production.
{% endhint %}
{% endstep %}
{% step %}
**Run the agent.**
```bash
python3 agent_trace.py
```
The script prints the model's reply and exports the spans. A macOS `LibreSSL`/`urllib3` warning, if shown, is harmless.
{% endstep %}
{% step %}
**View GenAI traces in Grafana.**
Open Grafana using the `grafanaEndpoint` you retrieved, open **Explore**, and select the Tracing (Tempo) data source. Query for GenAI spans with TraceQL:
```
{ span.gen_ai.request.model != "" }
```
Open the `agent-run` trace and expand the child `openai.chat` span. The trace shows GenAI attributes such as `gen_ai.system`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and the prompt or the completion content when recording is activated.
{% hint style="info" %}
**Note:** Span and service names reflect your own values. The parent span name (`agent-run`) and `service.name` (`ai-agent-demo`) come from the script above, so your trace shows those. The child `openai.chat` span and its `gen_ai.*` attributes are emitted by the OpenAI instrumentation. The example is from a validation run, so it shows a different service and span name.
{% endhint %}
**Result:** Your AI agent runs now appear as GenAI traces in Grafana. The `agent-run` trace contains the `openai.chat` span with `gen_ai.*` attributes for the model, token usage, latency, and (when recording is activated) the prompt and response content.
{% endstep %}
{% endstepper %}
## Troubleshooting
1. **No `gen_ai` spans:** Confirm the pipeline is `AVAILABLE`, `IONOS_TRACING_APIKEY` is set, and `TRACES_ENDPOINT` uses the full `.../v1/traces` path. `provider.shutdown()` must run so buffered spans are flushed before the process exits.
2. **Authentication errors (`401`) on ingestion:** The `apikey` header value must match the current pipeline key.
3. **Model call fails:** Verify the AI Model Hub base URL ends in `/v1`, the token is valid, and the model id is available on your hub.
4. **Doubled endpoint path:** If you switch to the `OTEL_EXPORTER_OTLP_ENDPOINT` base environment variable instead of the explicit `endpoint=`, drop the `/v1/traces` suffix; the exporter appends it.
## Decommission resources
Delete the tracing pipeline through the API to stop billing, using the pipeline `id` from the creation response:
```bash
curl --location \
--request DELETE 'https://tracing.de-txl.ionos.com/pipelines/' \
--header 'Authorization: Bearer '
```
Then review your AI Model Hub usage.
## Next steps
* [Trace n8n Workflow Executions with the Tracing Service](/cloud/tutorials/observability/tracing-service/trace-n8n-workflows-with-tracing-service.md)
* [Tracing Service documentation](https://docs.ionos.com/cloud/observability/tracing-service)
* [AI Model Hub documentation](https://docs.ionos.com/cloud/ai/ai-model-hub)
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.ionos.com/cloud/tutorials/observability/tracing-service/trace-ai-agent-using-ai-model-hub-with-tracing-service.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.