> For the complete documentation index, see [llms.txt](https://docs.ionos.com/cloud/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ionos.com/cloud/ai/ai-model-hub/how-tos/document-collections/migrate-from-document-collections.md).

# Retain Your Document Collections Data in a Self-Managed Vector Store

{% hint style="warning" %}
**Deprecation notice:** The Document Collections feature is being decommissioned on **August 31, 2026**. After that date, your collections will no longer be available through the <code class="expression">space.vars.ionos\_cloud</code> API.
{% endhint %}

This guide shows you how to export your existing Document Collections data, provision a PostgreSQL vector database on <code class="expression">space.vars.ionos\_cloud</code>, re-embed and import your documents, and update your application to query the new store. If you prefer to start fresh instead, see [<mark style="color:blue;">Retrieval Augmented Generation</mark>](/cloud/ai/ai-model-hub/how-tos/retrieval-augmented-generation.md).

Re-embedding is free until August 31, 2026 using dedicated migration model aliases.

## Before you begin

To follow this guide, you need:

| **Requirement**     | **Details**                                                                                                                                                                                              |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **API key**         | <code class="expression">space.vars.ionos\_cloud</code> AI Model Hub API key with access to Document Collections and the embeddings endpoint                                                             |
| **Python**          | 3.10 or later                                                                                                                                                                                            |
| **Target database** | <code class="expression">space.vars.ionos\_cloud</code> PostgreSQL cluster with the `pgvector` extension, provisioned in [<mark style="color:blue;">Step 2</mark>](#step-2-provision-a-vector-database). |
| **Python packages** | `requests`, `psycopg2-binary`                                                                                                                                                                            |

Install the required packages:

```bash
pip install requests psycopg2-binary
```

### What to know before you start

* **Embedding vectors are not exported.** The Document Collections API returns only the original text content. Every document must be re-embedded during import. <code class="expression">space.vars.ionos\_cloud</code> provides dedicated migration model aliases for this purpose. Re-embedding is **free** during the announced migration window. For more information, see [<mark style="color:blue;">Step 3</mark>](#step-3-import-and-re-embed-your-data).
* **Two vector dimensions are in use.** The three supported embedding models produce vectors of either 768 or 1024 dimensions. Check the `embedding.model` field in your export (captured by the export script in [<mark style="color:blue;">Step 1</mark>](#step-1-export-your-data)) to determine which models your collections use.

  | **Model**                                                     | **Dimensions** |
  | ------------------------------------------------------------- | -------------- |
  | `BAAI/bge-m3`                                                 | 1024           |
  | `BAAI/bge-large-en-v1.5`                                      | 1024           |
  | `sentence-transformers/paraphrase-multilingual-mpnet-base-v2` | 768            |
* **Plain text only.** Document Collections stores plain text (`text/plain`). No file conversion is needed.

***

## Step 1: Export your data

The following script exports all your collections and their document text to a local JSON file.

```python
#!/usr/bin/env python3
"""
export_document_collections.py

Exports all collections and their document text from the IONOS CLOUD Document Collections API
to a local JSON file.

Environment variables:
    DC_API_BASE  - base URL, e.g. https://inference.de-txl.ionos.com
    DC_API_KEY   - your IONOS CLOUD AI Model Hub API key
    CONTRACT_NO  - your IONOS CLOUD contract number
    OUTPUT_FILE  - path to write the export JSON (default: dc_export.json)
"""

import base64
import json
import os

import requests

API_BASE = os.environ["DC_API_BASE"].rstrip("/")
API_KEY = os.environ["DC_API_KEY"]
CONTRACT_NO = os.environ["CONTRACT_NO"]
OUTPUT_FILE = os.environ.get("OUTPUT_FILE", "dc_export.json")

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "X-Contract-No": CONTRACT_NO,
}

def paginate(url, params=None):
    items = []
    page = 0
    while True:
        resp = requests.get(url, headers=HEADERS, params={**(params or {}), "page": page, "pageSize": 100})
        resp.raise_for_status()
        data = resp.json()
        batch = data.get("items") or data.get("data") or []
        items.extend(batch)
        if len(batch) < 100:
            break
        page += 1
    return items

def decode_content(doc):
    raw = doc.get("content", "")
    return base64.b64decode(raw).decode("utf-8") if raw else ""

def main():
    print(f"Exporting collections for contract {CONTRACT_NO}...")

    collections = paginate(f"{API_BASE}/collections")
    print(f"  Found {len(collections)} collection(s).")

    export = []
    for col in collections:
        col_id = col["id"]
        print(f"  Exporting collection {col['name']!r} ({col_id})...")

        documents = paginate(f"{API_BASE}/collections/{col_id}/documents")
        doc_records = []
        for doc in documents:
            doc_records.append({
                "id": doc["id"],
                "name": doc.get("name", ""),
                "description": doc.get("description", ""),
                "text": decode_content(doc),
                "labels": doc.get("labels", {}),
            })

        export.append({
            "id": col_id,
            "name": col["name"],
            "description": col.get("description", ""),
            "chunking": col.get("chunking", {}),
            "embedding": col.get("embedding", {}),
            "documents": doc_records,
        })

    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
        json.dump(export, f, ensure_ascii=False, indent=2)

    total_docs = sum(len(c["documents"]) for c in export)
    print(f"Exported {len(export)} collection(s) / {total_docs} document(s) → {OUTPUT_FILE}")

if __name__ == "__main__":
    main()
```

Run the script with your credentials:

```bash
export DC_API_BASE="https://inference.de-txl.ionos.com"
export DC_API_KEY="<your-api-key>"
export CONTRACT_NO="<your-contract-number>"
python export_document_collections.py
```

The script writes a `dc_export.json` file to the current directory containing all your collections and document text.

***

## Step 2: Provision a vector database

The recommended target is <code class="expression">space.vars.ionos\_cloud</code> **PostgreSQL with the pgvector extension**. It is production-ready and provides full ACID guarantees with cosine, L2, and inner-product similarity operators.

### Create a PostgreSQL cluster

Create a PostgreSQL 17 cluster through the <code class="expression">space.vars.ionos\_cloud\_api</code>:

```bash
curl -s -X POST \
  -H "Authorization: Bearer $IONOS_API_TOKEN" \
  -H "Content-Type: application/json" \
  "https://postgresql.de-txl.ionos.com/clusters" \
  --data-raw '{
    "properties": {
      "name": "my-pgvector-cluster",
      "version": "17",
      "instances": {"count": 1, "cores": 2, "ram": 8, "storageSize": 50},
      "connection": {
        "datacenterId": "<your-datacenter-id>",
        "lanId": "<private-lan-id>",
        "primaryInstanceAddress": "10.0.1.10/24"
      },
      "replicationMode": "ASYNCHRONOUS",
      "maintenanceWindow": {"dayOfTheWeek": "Sunday", "time": "04:00:00"},
      "backup": {"location": "eu-central-3", "retentionDays": 7},
      "credentials": {
        "username": "pgmigration",
        "password": "<password>",
        "database": "dc_migration"
      }
    }
  }'
```

{% hint style="info" %}
**External access:** <code class="expression">space.vars.ionos\_cloud</code> managed PostgreSQL clusters use private IPs only. To connect from networks external <code class="expression">space.vars.ionos\_cloud</code>, set up a Network Load Balancer that forwards a public port 5432 to the cluster's private IP address. For more information, see [<mark style="color:blue;">Enable External Access to a PostgreSQL Database</mark>](https://docs.ionos.com/cloud/tutorials/databases/postgresql/managed-postgres-external-access-nlb).
{% endhint %}

Verify the connection after setup:

```bash
PGPASSWORD="<password>" psql -h <nlb-public-ip> -p 5432 -U pgmigration -d dc_migration \
  -c "SELECT version();"
```

### Create the schema

Connect with `psql` and create the required schema. Two tables are needed because the three supported embedding models produce vectors of two different sizes:

```sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS collections (
    id              UUID PRIMARY KEY,
    name            TEXT NOT NULL,
    description     TEXT,
    db_type         TEXT,
    embedding_model TEXT,
    created_at      TIMESTAMPTZ DEFAULT now()
);

-- 768-dimensional (sentence-transformers/paraphrase-multilingual-mpnet-base-v2)
CREATE TABLE IF NOT EXISTS documents_768 (
    id            UUID PRIMARY KEY,
    collection_id UUID NOT NULL REFERENCES collections(id) ON DELETE CASCADE,
    name          TEXT,
    description   TEXT,
    text_content  TEXT NOT NULL,
    embedding     VECTOR(768),
    labels        JSONB,
    created_at    TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_768 ON documents_768 USING hnsw (embedding vector_cosine_ops);

-- 1024-dimensional (BAAI/bge-m3 and BAAI/bge-large-en-v1.5)
CREATE TABLE IF NOT EXISTS documents_1024 (
    id            UUID PRIMARY KEY,
    collection_id UUID NOT NULL REFERENCES collections(id) ON DELETE CASCADE,
    name          TEXT,
    description   TEXT,
    text_content  TEXT NOT NULL,
    embedding     VECTOR(1024),
    labels        JSONB,
    created_at    TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_1024 ON documents_1024 USING hnsw (embedding vector_cosine_ops);
```

***

## Step 3: Import and re-embed your data

### Free migration model aliases

<code class="expression">space.vars.ionos\_cloud</code> provides dedicated model aliases that are **not billed** during the migration window. Use the alias that matches the original embedding model of each collection:

| **Original model**                                            | **Migration alias**                               | **Dimensions** |
| ------------------------------------------------------------- | ------------------------------------------------- | -------------- |
| `BAAI/bge-m3`                                                 | `bge-m3-migration`                                | 1024           |
| `BAAI/bge-large-en-v1.5`                                      | `bge-large-en-v1.5-migration`                     | 1024           |
| `sentence-transformers/paraphrase-multilingual-mpnet-base-v2` | `paraphrase-multilingual-mpnet-base-v2-migration` | 768            |

{% hint style="warning" %}
**Important:** After the migration window closes, switch to the standard model IDs. The migration aliases will no longer be available.
{% endhint %}

### Run the import script

The following script reads your `dc_export.json` file, re-embeds each document using the <code class="expression">space.vars.ionos\_cloud</code> embeddings API, and inserts the results into PostgreSQL. It handles both 768-dimensional and 1024-dimensional collections in a single pass.

```python
#!/usr/bin/env python3
"""
import_to_pgvector.py

Reads dc_export.json produced by the export script,
re-embeds all documents, and inserts into PostgreSQL + pgvector.

Environment variables:
    IONOS_API_TOKEN  — IONOS AI Model Hub API token
    PG_HOST          — PostgreSQL host (NLB public IP or private IP)
    PG_PORT          — PostgreSQL port (default: 5432)
    PG_DB            — database name
    PG_USER          — database user
    PG_PASSWORD      — database password
    INPUT_FILE       — path to export JSON (default: dc_export.json)
"""

import json
import os
import time
import uuid

import psycopg2
import psycopg2.extras
import requests

EMBED_BASE = "https://openai.inference.de-txl.ionos.com"
TOKEN = os.environ["IONOS_API_TOKEN"]
PG = dict(
    host=os.environ["PG_HOST"],
    port=int(os.environ.get("PG_PORT", 5432)),
    dbname=os.environ["PG_DB"],
    user=os.environ["PG_USER"],
    password=os.environ["PG_PASSWORD"],
    connect_timeout=15,
)
INPUT_FILE = os.environ.get("INPUT_FILE", "dc_export.json")

EH = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
BATCH = 64

def embed(texts, model):
    out = []
    for i in range(0, len(texts), BATCH):
        batch = texts[i : i + BATCH]
        for attempt in range(3):
            try:
                r = requests.post(
                    f"{EMBED_BASE}/v1/embeddings",
                    headers=EH,
                    json={"model": model, "input": batch},
                    timeout=60,
                )
                if r.status_code == 429:
                    time.sleep(int(r.headers.get("Retry-After", 30)))
                    continue
                r.raise_for_status()
                out.extend(x["embedding"] for x in r.json()["data"])
                break
            except Exception as exc:
                if attempt == 2:
                    raise
                time.sleep(10 * (2 ** attempt))
                print(f"  retry {attempt + 1} on batch {i // BATCH}: {exc}", flush=True)
    return out

def main():
    psycopg2.extras.register_uuid()
    conn = psycopg2.connect(**PG)
    export = json.load(open(INPUT_FILE, encoding="utf-8"))
    print(f"Loaded {len(export)} collections from {INPUT_FILE}", flush=True)

    for idx, col in enumerate(export):
        cid = col["id"]
        name = col["name"]
        model = col["embedding"]["model"]
        dim = 768 if "paraphrase" in model else 1024
        docs = col["documents"]

        with conn, conn.cursor() as c:
            c.execute(
                "INSERT INTO collections VALUES(%s,%s,%s,%s,%s) ON CONFLICT DO NOTHING",
                (uuid.UUID(cid), name, col.get("description", ""),
                 col.get("db_type", ""), model),
            )

        if not docs:
            continue

        texts = [d.get("text", "") for d in docs]

        t_emb = time.time()
        vectors = embed(texts, model)
        t_emb = time.time() - t_emb

        t_ins = time.time()
        rows = [
            (uuid.UUID(d["id"]), uuid.UUID(cid),
             d.get("name", ""), d.get("description", ""),
             d.get("text", ""), vec, json.dumps(d.get("labels", {})))
            for d, vec in zip(docs, vectors)
        ]
        with conn, conn.cursor() as c:
            psycopg2.extras.execute_values(
                c,
                f"INSERT INTO documents_{dim}"
                "(id,collection_id,name,description,text_content,embedding,labels)"
                " VALUES %s ON CONFLICT DO NOTHING",
                rows,
                template="(%s,%s,%s,%s,%s,%s::vector,%s)",
                page_size=200,
            )
        t_ins = time.time() - t_ins

        print(f"[{idx+1}/{len(export)}] {name[:40]:<40} | {len(docs):5d} docs | "
              f"{dim}d | embed={t_emb:.1f}s insert={t_ins:.1f}s", flush=True)

    with conn.cursor() as c:
        c.execute("SELECT COUNT(*) FROM documents_768"); n768 = c.fetchone()[0]
        c.execute("SELECT COUNT(*) FROM documents_1024"); n1024 = c.fetchone()[0]
        c.execute("SELECT COUNT(*) FROM collections"); ncols = c.fetchone()[0]

    print(f"\n=== Done ===")
    print(f"  Collections:    {ncols}")
    print(f"  documents_768:  {n768}")
    print(f"  documents_1024: {n1024}")
    conn.close()

if __name__ == "__main__":
    main()
```

Run the import with your database and API credentials:

```bash
export IONOS_API_TOKEN="<your-api-key>"
export PG_HOST="<nlb-public-ip>"
export PG_DB="dc_migration"
export PG_USER="pgmigration"
export PG_PASSWORD="<password>"
export INPUT_FILE="dc_export.json"
python3 import_to_pgvector.py
```

***

## Step 4: Query the new vector store

After the import completes, replace Document Collections query calls with direct similarity searches against the new PostgreSQL database. The following example shows the pattern:

```python
import os
import uuid

import psycopg2
import requests

EMBED_API_BASE = "https://openai.inference.de-txl.ionos.com"
EMBED_API_KEY = os.environ["IONOS_API_TOKEN"]
EMBED_MODEL = "sentence-transformers/paraphrase-multilingual-mpnet-base-v2"
EMBED_HEADERS = {"Authorization": f"Bearer {EMBED_API_KEY}", "Content-Type": "application/json"}

def query_collection(conn, collection_id, query_text, top_k=5):
    resp = requests.post(
        f"{EMBED_API_BASE}/v1/embeddings",
        headers=EMBED_HEADERS,
        json={"model": EMBED_MODEL, "input": [query_text]},
    )
    resp.raise_for_status()
    query_vector = resp.json()["data"][0]["embedding"]

    # Use documents_768 or documents_1024 depending on the model used for the collection
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT id, text_content, 1 - (embedding <=> %s::vector) AS score
            FROM documents_768
            WHERE collection_id = %s
            ORDER BY score DESC
            LIMIT %s
            """,
            (query_vector, uuid.UUID(collection_id), top_k),
        )
        rows = cur.fetchall()

    return [{"id": str(r[0]), "text": r[1], "score": float(r[2])} for r in rows]
```

{% hint style="info" %}
To build a full RAG pipeline on top of the new vector store, see the [<mark style="color:blue;">Retrieval Augmented Generation</mark>](/cloud/ai/ai-model-hub/how-tos/retrieval-augmented-generation.md) guide.
{% endhint %}

***

## Step 5: Remove deprecated API parameters

Remove the `collectionId` and `collectionQuery` parameters from any existing chat completion or inference calls that relied on the built-in Document Collections context injection. These parameters will be removed from the API when the feature is fully decommissioned.

***

## Step 6: Confirm successful migration

Before decommissioning your Document Collections data, run the following checks:

1. **Document count:** Compare the row count in `documents_768` and `documents_1024` against the document counts returned by the Document Collections API. The totals must match.
2. **Spot-check retrieval:** Run a known query against both the old Document Collections endpoint and the new PostgreSQL store. Verify that the top results are the same or semantically equivalent.
3. **Application smoke test:** Run your application's test suite with the new database configured to confirm end-to-end behavior.

{% hint style="success" %}
**Result:** Your Document Collections data is now in a self-managed PostgreSQL vector store. You can safely stop using the Document Collections API and decommission your collections before the end-of-life date.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.ionos.com/cloud/ai/ai-model-hub/how-tos/document-collections/migrate-from-document-collections.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.