Retain Your Document Collections Data in a Self-Managed Vector Store

Deprecation notice: The Document Collections feature is being decommissioned on August 31, 2026. After that date, your collections will no longer be available through the IONOS CLOUD API.

This guide shows you how to export your existing Document Collections data, provision a PostgreSQL vector database on IONOS CLOUD, re-embed and import your documents, and update your application to query the new store. If you prefer to start fresh instead, see Retrieval Augmented Generation.

Re-embedding is free until August 31, 2026 using dedicated migration model aliases.

Before you begin

To follow this guide, you need:

Requirement	Details
API key	IONOS CLOUD AI Model Hub API key with access to Document Collections and the embeddings endpoint
Python	3.10 or later
Target database	IONOS CLOUD PostgreSQL cluster with the pgvector extension, provisioned in Step 2.
Python packages	requests, psycopg2-binary

Install the required packages:

pip install requests psycopg2-binary

What to know before you start

• Embedding vectors are not exported. The Document Collections API returns only the original text content. Every document must be re-embedded during import. IONOS CLOUD provides dedicated migration model aliases for this purpose. Re-embedding is free during the announced migration window. For more information, see Step 3.
• Two vector dimensions are in use. The three supported embedding models produce vectors of either 768 or 1024 dimensions. Check the embedding.model field in your export (captured by the export script in Step 1) to determine which models your collections use.

  Model	Dimensions
  BAAI/bge-m3	1024
  BAAI/bge-large-en-v1.5	1024
  sentence-transformers/paraphrase-multilingual-mpnet-base-v2	768

• Plain text only. Document Collections stores plain text (text/plain). No file conversion is needed.

---

Step 1: Export your data

The following script exports all your collections and their document text to a local JSON file.

#!/usr/bin/env python3
"""
export_document_collections.py

Exports all collections and their document text from the IONOS CLOUD Document Collections API
to a local JSON file.

Environment variables:
    DC_API_BASE  - base URL, e.g. https://inference.de-txl.ionos.com
    DC_API_KEY   - your IONOS CLOUD AI Model Hub API key
    CONTRACT_NO  - your IONOS CLOUD contract number
    OUTPUT_FILE  - path to write the export JSON (default: dc_export.json)
"""

import base64
import json
import os

import requests

API_BASE = os.environ["DC_API_BASE"].rstrip("/")
API_KEY = os.environ["DC_API_KEY"]
CONTRACT_NO = os.environ["CONTRACT_NO"]
OUTPUT_FILE = os.environ.get("OUTPUT_FILE", "dc_export.json")

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "X-Contract-No": CONTRACT_NO,
}

def paginate(url, params=None):
    items = []
    page = 0
    while True:
        resp = requests.get(url, headers=HEADERS, params={**(params or {}), "page": page, "pageSize": 100})
        resp.raise_for_status()
        data = resp.json()
        batch = data.get("items") or data.get("data") or []
        items.extend(batch)
        if len(batch) < 100:
            break
        page += 1
    return items

def decode_content(doc):
    raw = doc.get("content", "")
    return base64.b64decode(raw).decode("utf-8") if raw else ""

def main():
    print(f"Exporting collections for contract {CONTRACT_NO}...")

    collections = paginate(f"{API_BASE}/collections")
    print(f"  Found {len(collections)} collection(s).")

    export = []
    for col in collections:
        col_id = col["id"]
        print(f"  Exporting collection {col['name']!r} ({col_id})...")

        documents = paginate(f"{API_BASE}/collections/{col_id}/documents")
        doc_records = []
        for doc in documents:
            doc_records.append({
                "id": doc["id"],
                "name": doc.get("name", ""),
                "description": doc.get("description", ""),
                "text": decode_content(doc),
                "labels": doc.get("labels", {}),
            })

        export.append({
            "id": col_id,
            "name": col["name"],
            "description": col.get("description", ""),
            "chunking": col.get("chunking", {}),
            "embedding": col.get("embedding", {}),
            "documents": doc_records,
        })

    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
        json.dump(export, f, ensure_ascii=False, indent=2)

    total_docs = sum(len(c["documents"]) for c in export)
    print(f"Exported {len(export)} collection(s) / {total_docs} document(s) → {OUTPUT_FILE}")

if __name__ == "__main__":
    main()

Run the script with your credentials:

export DC_API_BASE="https://inference.de-txl.ionos.com"
export DC_API_KEY="<your-api-key>"
export CONTRACT_NO="<your-contract-number>"
python export_document_collections.py

The script writes a dc_export.json file to the current directory containing all your collections and document text.

---

Step 2: Provision a vector database

The recommended target is IONOS CLOUD PostgreSQL with the pgvector extension. It is production-ready and provides full ACID guarantees with cosine, L2, and inner-product similarity operators.

Create a PostgreSQL cluster

Create a PostgreSQL 17 cluster through the IONOS CLOUD API:

curl -s -X POST \
  -H "Authorization: Bearer $IONOS_API_TOKEN" \
  -H "Content-Type: application/json" \
  "https://postgresql.de-txl.ionos.com/clusters" \
  --data-raw '{
    "properties": {
      "name": "my-pgvector-cluster",
      "version": "17",
      "instances": {"count": 1, "cores": 2, "ram": 8, "storageSize": 50},
      "connection": {
        "datacenterId": "<your-datacenter-id>",
        "lanId": "<private-lan-id>",
        "primaryInstanceAddress": "10.0.1.10/24"
      },
      "replicationMode": "ASYNCHRONOUS",
      "maintenanceWindow": {"dayOfTheWeek": "Sunday", "time": "04:00:00"},
      "backup": {"location": "eu-central-3", "retentionDays": 7},
      "credentials": {
        "username": "pgmigration",
        "password": "<password>",
        "database": "dc_migration"
      }
    }
  }'

External access: IONOS CLOUD managed PostgreSQL clusters use private IPs only. To connect from networks external IONOS CLOUD, set up a Network Load Balancer that forwards a public port 5432 to the cluster's private IP address. For more information, see Enable External Access to a PostgreSQL Database.

Verify the connection after setup:

PGPASSWORD="<password>" psql -h <nlb-public-ip> -p 5432 -U pgmigration -d dc_migration \
  -c "SELECT version();"

Create the schema

Connect with psql and create the required schema. Two tables are needed because the three supported embedding models produce vectors of two different sizes:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS collections (
    id              UUID PRIMARY KEY,
    name            TEXT NOT NULL,
    description     TEXT,
    db_type         TEXT,
    embedding_model TEXT,
    created_at      TIMESTAMPTZ DEFAULT now()
);

-- 768-dimensional (sentence-transformers/paraphrase-multilingual-mpnet-base-v2)
CREATE TABLE IF NOT EXISTS documents_768 (
    id            UUID PRIMARY KEY,
    collection_id UUID NOT NULL REFERENCES collections(id) ON DELETE CASCADE,
    name          TEXT,
    description   TEXT,
    text_content  TEXT NOT NULL,
    embedding     VECTOR(768),
    labels        JSONB,
    created_at    TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_768 ON documents_768 USING hnsw (embedding vector_cosine_ops);

-- 1024-dimensional (BAAI/bge-m3 and BAAI/bge-large-en-v1.5)
CREATE TABLE IF NOT EXISTS documents_1024 (
    id            UUID PRIMARY KEY,
    collection_id UUID NOT NULL REFERENCES collections(id) ON DELETE CASCADE,
    name          TEXT,
    description   TEXT,
    text_content  TEXT NOT NULL,
    embedding     VECTOR(1024),
    labels        JSONB,
    created_at    TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_1024 ON documents_1024 USING hnsw (embedding vector_cosine_ops);

---

Step 3: Import and re-embed your data

Free migration model aliases

IONOS CLOUD provides dedicated model aliases that are not billed during the migration window. Use the alias that matches the original embedding model of each collection:

Original model	Migration alias	Dimensions
BAAI/bge-m3	bge-m3-migration	1024
BAAI/bge-large-en-v1.5	bge-large-en-v1.5-migration	1024
sentence-transformers/paraphrase-multilingual-mpnet-base-v2	paraphrase-multilingual-mpnet-base-v2-migration	768

Important: After the migration window closes, switch to the standard model IDs. The migration aliases will no longer be available.

Run the import script

The following script reads your dc_export.json file, re-embeds each document using the IONOS CLOUD embeddings API, and inserts the results into PostgreSQL. It handles both 768-dimensional and 1024-dimensional collections in a single pass.

#!/usr/bin/env python3
"""
import_to_pgvector.py

Reads dc_export.json produced by the export script,
re-embeds all documents, and inserts into PostgreSQL + pgvector.

Environment variables:
    IONOS_API_TOKEN  — IONOS AI Model Hub API token
    PG_HOST          — PostgreSQL host (NLB public IP or private IP)
    PG_PORT          — PostgreSQL port (default: 5432)
    PG_DB            — database name
    PG_USER          — database user
    PG_PASSWORD      — database password
    INPUT_FILE       — path to export JSON (default: dc_export.json)
"""

import json
import os
import time
import uuid

import psycopg2
import psycopg2.extras
import requests

EMBED_BASE = "https://openai.inference.de-txl.ionos.com"
TOKEN = os.environ["IONOS_API_TOKEN"]
PG = dict(
    host=os.environ["PG_HOST"],
    port=int(os.environ.get("PG_PORT", 5432)),
    dbname=os.environ["PG_DB"],
    user=os.environ["PG_USER"],
    password=os.environ["PG_PASSWORD"],
    connect_timeout=15,
)
INPUT_FILE = os.environ.get("INPUT_FILE", "dc_export.json")

EH = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}
BATCH = 64

def embed(texts, model):
    out = []
    for i in range(0, len(texts), BATCH):
        batch = texts[i : i + BATCH]
        for attempt in range(3):
            try:
                r = requests.post(
                    f"{EMBED_BASE}/v1/embeddings",
                    headers=EH,
                    json={"model": model, "input": batch},
                    timeout=60,
                )
                if r.status_code == 429:
                    time.sleep(int(r.headers.get("Retry-After", 30)))
                    continue
                r.raise_for_status()
                out.extend(x["embedding"] for x in r.json()["data"])
                break
            except Exception as exc:
                if attempt == 2:
                    raise
                time.sleep(10 * (2 ** attempt))
                print(f"  retry {attempt + 1} on batch {i // BATCH}: {exc}", flush=True)
    return out

def main():
    psycopg2.extras.register_uuid()
    conn = psycopg2.connect(**PG)
    export = json.load(open(INPUT_FILE, encoding="utf-8"))
    print(f"Loaded {len(export)} collections from {INPUT_FILE}", flush=True)

    for idx, col in enumerate(export):
        cid = col["id"]
        name = col["name"]
        model = col["embedding"]["model"]
        dim = 768 if "paraphrase" in model else 1024
        docs = col["documents"]

        with conn, conn.cursor() as c:
            c.execute(
                "INSERT INTO collections VALUES(%s,%s,%s,%s,%s) ON CONFLICT DO NOTHING",
                (uuid.UUID(cid), name, col.get("description", ""),
                 col.get("db_type", ""), model),
            )

        if not docs:
            continue

        texts = [d.get("text", "") for d in docs]

        t_emb = time.time()
        vectors = embed(texts, model)
        t_emb = time.time() - t_emb

        t_ins = time.time()
        rows = [
            (uuid.UUID(d["id"]), uuid.UUID(cid),
             d.get("name", ""), d.get("description", ""),
             d.get("text", ""), vec, json.dumps(d.get("labels", {})))
            for d, vec in zip(docs, vectors)
        ]
        with conn, conn.cursor() as c:
            psycopg2.extras.execute_values(
                c,
                f"INSERT INTO documents_{dim}"
                "(id,collection_id,name,description,text_content,embedding,labels)"
                " VALUES %s ON CONFLICT DO NOTHING",
                rows,
                template="(%s,%s,%s,%s,%s,%s::vector,%s)",
                page_size=200,
            )
        t_ins = time.time() - t_ins

        print(f"[{idx+1}/{len(export)}] {name[:40]:<40} | {len(docs):5d} docs | "
              f"{dim}d | embed={t_emb:.1f}s insert={t_ins:.1f}s", flush=True)

    with conn.cursor() as c:
        c.execute("SELECT COUNT(*) FROM documents_768"); n768 = c.fetchone()[0]
        c.execute("SELECT COUNT(*) FROM documents_1024"); n1024 = c.fetchone()[0]
        c.execute("SELECT COUNT(*) FROM collections"); ncols = c.fetchone()[0]

    print(f"\n=== Done ===")
    print(f"  Collections:    {ncols}")
    print(f"  documents_768:  {n768}")
    print(f"  documents_1024: {n1024}")
    conn.close()

if __name__ == "__main__":
    main()

Run the import with your database and API credentials:

export IONOS_API_TOKEN="<your-api-key>"
export PG_HOST="<nlb-public-ip>"
export PG_DB="dc_migration"
export PG_USER="pgmigration"
export PG_PASSWORD="<password>"
export INPUT_FILE="dc_export.json"
python3 import_to_pgvector.py

---

Step 4: Query the new vector store

After the import completes, replace Document Collections query calls with direct similarity searches against the new PostgreSQL database. The following example shows the pattern:

import os
import uuid

import psycopg2
import requests

EMBED_API_BASE = "https://openai.inference.de-txl.ionos.com"
EMBED_API_KEY = os.environ["IONOS_API_TOKEN"]
EMBED_MODEL = "sentence-transformers/paraphrase-multilingual-mpnet-base-v2"
EMBED_HEADERS = {"Authorization": f"Bearer {EMBED_API_KEY}", "Content-Type": "application/json"}

def query_collection(conn, collection_id, query_text, top_k=5):
    resp = requests.post(
        f"{EMBED_API_BASE}/v1/embeddings",
        headers=EMBED_HEADERS,
        json={"model": EMBED_MODEL, "input": [query_text]},
    )
    resp.raise_for_status()
    query_vector = resp.json()["data"][0]["embedding"]

    # Use documents_768 or documents_1024 depending on the model used for the collection
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT id, text_content, 1 - (embedding <=> %s::vector) AS score
            FROM documents_768
            WHERE collection_id = %s
            ORDER BY score DESC
            LIMIT %s
            """,
            (query_vector, uuid.UUID(collection_id), top_k),
        )
        rows = cur.fetchall()

    return [{"id": str(r[0]), "text": r[1], "score": float(r[2])} for r in rows]

To build a full RAG pipeline on top of the new vector store, see the Retrieval Augmented Generation guide.

---

Step 5: Remove deprecated API parameters

Remove the collectionId and collectionQuery parameters from any existing chat completion or inference calls that relied on the built-in Document Collections context injection. These parameters will be removed from the API when the feature is fully decommissioned.

---

Step 6: Confirm successful migration

Before decommissioning your Document Collections data, run the following checks:

1. Document count: Compare the row count in documents_768 and documents_1024 against the document counts returned by the Document Collections API. The totals must match.
2. Spot-check retrieval: Run a known query against both the old Document Collections endpoint and the new PostgreSQL store. Verify that the top results are the same or semantically equivalent.
3. Application smoke test: Run your application's test suite with the new database configured to confirm end-to-end behavior.

Result: Your Document Collections data is now in a self-managed PostgreSQL vector store. You can safely stop using the Document Collections API and decommission your collections before the end-of-life date.