Model Source Management API¶

Catalog, CRUD, and gateway-selection API for cloud-hosted LLMs (the "Model Gateway"). A ModelSource is a catalog entry describing one model hosted on AWS Bedrock or Azure OpenAI plus the connection details needed to reach it. A ModelGatewaySelection singleton records which ModelSource currently backs each of the two global slots (default workflow/chat model, and embedding model).

Table of Contents¶

1. Concepts
2. API Route Prefix
3. API Endpoints
4. 3.1. Create Model Source
5. 3.2. List Model Sources
6. 3.3. Get Model Source Detail
7. 3.4. Update Model Source
8. 3.5. Delete Model Source
9. 3.6. Get Gateway Selection
10. 3.7. Set Default Workflow Model
11. 3.8. Set Embedding Model
    • 3.8.1. List Reindex Jobs
12. Access Control
13. Data Models
14. Credentials & Encryption
15. Model Metadata
16. Error Response Format

Concepts¶

• ModelSource — one entry in the model catalog: display metadata + mode (chat or embedding) + a typed, per-provider providerConfig. Stored in the model_sources collection. Soft-deleted (deletedAt), never hard-deleted via the API.
• ModelGatewaySelection — a singleton document (collection model_gateway_selection, always at most one row) holding two pointer fields: defaultWorkflowModelSourceId and embeddingModelSourceId. It is not a usage log — it only records the two current global defaults.
• No ACL — scope only. Unlike Federation / SkillSyncSource, a ModelSource is treated as a system-wide config option, not a per-user-owned resource. There is no RegistryResourceType entry and no per-record ownership: authorization is enforced entirely by ScopePermissionMiddleware against scopes.yml. Anyone with models-read sees every model; anyone with models-write may modify any model.

API Route Prefix¶

/api/v1

(Endpoints are /api/v1/model-sources... and /api/v1/model-gateway/selection....)

API Endpoints¶

1. Create Model Source¶

Endpoint: POST /api/v1/model-sources Scope: models-write

Request Body (AWS Bedrock):

{
  "displayName": "Claude 3.5 Sonnet",
  "description": "Default reasoning model",
  "tags": ["bedrock", "chat"],
  "mode": "chat",
  "providerConfig": {
    "providerType": "aws_bedrock",
    "awsRegion": "us-east-1",
    "modelIdOrArn": "anthropic.claude-3-5-sonnet-20240620-v1:0",
    "baseModelId": "anthropic.claude-3-5-sonnet-20240620-v1:0"
  }
}

Request Body (Azure OpenAI) — apiKey is plaintext on input; the service encrypts it before storage and it is never returned:

{
  "displayName": "Azure GPT-4o",
  "mode": "chat",
  "providerConfig": {
    "providerType": "azure_openai",
    "endpoint": "https://acme.openai.azure.com",
    "deploymentName": "gpt4o-prod",
    "baseModelId": "gpt-4o",
    "apiVersion": "2024-10-21",
    "apiKey": "sk-..."
  }
}

Request Fields: - displayName (required, string, 1–128 chars) - description (optional, string) - tags (optional, string[]) - mode (required, enum): chat | embedding - providerConfig (required, discriminated on providerType): see Data Models

Response: 201 Created → ModelSourceDetailResponse (no metadata on create).

2. List Model Sources¶

Endpoint: GET /api/v1/model-sources Scope: models-read

Query Parameters: - mode (optional, enum): filter by chat | embedding - providerType (optional, enum): filter by aws_bedrock | azure_openai - tag (optional, string): filter by a single tag - query (optional, string): full-text search over displayName / description - page (optional, int, default 1, ≥1) - per_page (optional, int, default 20, 1–100)

Response: 200 OK

{
  "modelSources": [
    {
      "id": "000000000000000000000001",
      "displayName": "Claude 3.5 Sonnet",
      "description": "Default reasoning model",
      "tags": ["bedrock", "chat"],
      "mode": "chat",
      "providerType": "aws_bedrock",
      "createdAt": "2026-09-15T08:16:44Z",
      "updatedAt": "2026-09-15T08:16:44Z"
    }
  ],
  "pagination": { "total": 1, "page": 1, "perPage": 20, "totalPages": 1 }
}

Results are sorted by updatedAt descending and exclude soft-deleted entries. List items are lightweight — no providerConfig and no metadata (fetch detail for those).

3. Get Model Source Detail¶

Endpoint: GET /api/v1/model-sources/{model_source_id} Scope: models-read

Response: 200 OK → ModelSourceDetailResponse, including live metadata resolved from litellm.get_model_info() and a masked providerConfig (hasApiKey: true|false instead of the encrypted key). 404 if not found or soft-deleted.

{
  "id": "000000000000000000000002",
  "displayName": "Azure text-embedding-3-small",
  "mode": "embedding",
  "providerType": "azure_openai",
  "tags": ["azure", "embedding"],
  "createdAt": "2026-09-15T08:20:24Z",
  "updatedAt": "2026-09-15T08:20:24Z",
  "providerConfig": {
    "providerType": "azure_openai",
    "endpoint": "https://acme.openai.azure.com",
    "deploymentName": "embed-small",
    "baseModelId": "text-embedding-3-small",
    "apiVersion": "2024-10-21",
    "hasApiKey": true
  },
  "metadata": {
    "maxInputTokens": 8191,
    "maxOutputTokens": null,
    "inputCostPerToken": 2e-08,
    "outputCostPerToken": 0.0,
    "supportsPromptCaching": null,
    "unavailableReason": null
  },
  "createdBy": "000000000000000000000009",
  "updatedBy": "000000000000000000000009"
}

4. Update Model Source¶

Endpoint: PATCH /api/v1/model-sources/{model_source_id} Scope: models-write

Partial update — only the fields present in the body are changed (displayName, description, tags, mode, providerConfig).

Credential handling: providerConfig is replaced as a whole when supplied. For an Azure config, if apiKey is omitted, the previously stored encrypted key is preserved; if a new apiKey is supplied, it is re-encrypted. (Omitting providerConfig entirely leaves the stored config and key untouched.)

Response: 200 OK → ModelSourceDetailResponse (no metadata). 404 if not found/soft-deleted.

5. Delete Model Source¶

Endpoint: DELETE /api/v1/model-sources/{model_source_id} Scope: models-write

Soft-delete — sets deletedAt; the row remains in Mongo. A subsequent GET/DELETE on the same id returns 404.

Delete guard: returns 409 Conflict when the model is currently referenced by a gateway selection slot (defaultWorkflowModelSourceId or embeddingModelSourceId), or when any WorkflowNode.model_source_id across any WorkflowDefinition references the model.

Response: 200 OK

{ "id": "000000000000000000000001", "deletedAt": "2026-09-15T08:16:44Z" }

6. Get Gateway Selection¶

Endpoint: GET /api/v1/model-gateway/selection Scope: models-read

Response: 200 OK — the current two global default slots. On a fresh deployment (no selection set yet) both are null.

{ "defaultWorkflowModelSourceId": null, "embeddingModelSourceId": null }

7. Set Default Workflow Model¶

Endpoint: PUT /api/v1/model-gateway/selection/default-workflow-model Scope: models-write

Sets defaultWorkflowModelSourceId. Takes effect on the next workflow run — no restart: the effective default is resolved fresh (via Beanie) on every run, so a change is picked up without a pod restart.

Request Body: { "modelSourceId": "<id>" }

Response: 200 OK → the updated selection (same shape as Get Gateway Selection).

{ "defaultWorkflowModelSourceId": "000000000000000000000001", "embeddingModelSourceId": null }

Errors: - 404 — modelSourceId does not resolve (invalid id, not found, or soft-deleted). - 409 — the target ModelSource has mode != chat.

8. Set Embedding Model¶

Endpoint: PUT /api/v1/model-gateway/selection/embedding-model Scope: models-write

Selects an embedding model and re-embeds every existing document against it — no restart, using collection generations. The call first synchronously smoke-tests the target model (builds only its embedding client and runs one embed_query, opening no vector-store connection); only if that passes does it start a background reindex job.

The job re-embeds every document into a brand-new collection generation (<Base>_<jobId>) and then switches the active (embeddingModelSourceId, embeddingCollectionGeneration) pair with a single compare-and-set. Every pod follows that pair and swaps its own embedding client + collections. This means:

• Search is never blocked. Each pod keeps serving its current generation consistently; there is no maintenance window for reads.
• Writes to servers/agents return 503 during the sweep and a short grace period after the switch (so a write can't land in a generation about to be superseded), recovering automatically.
• The switch is atomic and all-or-nothing: on any failure nothing is committed, the live collections and selection are untouched, and the abandoned generation is reclaimed by a background GC.
• The previous generation's collections are dropped later by that GC (not inside the job), so a pod that is briefly behind never reads a dropped collection.

embeddingModelSourceId (and the generation) change only when the reindex completes — so GET /model-gateway/selection keeps showing the previous model until the job finishes.

Request Body: { "modelSourceId": "<id>" }

Response: 202 Accepted → same shape as Get Gateway Selection. The body echoes the current selection (unchanged; it switches only on completion), so it is null on a deployment that has never selected an embedding model. The 202 carries no job id; use List Reindex Jobs to see whether the job is running, committed, or failed.

{ "defaultWorkflowModelSourceId": null, "embeddingModelSourceId": null }

Errors: - 404 — modelSourceId does not resolve (invalid id, not found, or soft-deleted). - 409 — the target ModelSource has mode != embedding, or a reindex is already running, or a federation sync job is in flight (PENDING/SYNCING for any federation). The reindex must wait for federation writes to drain before it commits, so it is refused up front instead of sweeping the corpus and then failing. The detail names up to five such jobs (id, federation, status, created time). A job that never finishes was orphaned by a registry restart and has to be marked FAILED in Mongo before a reindex can start. This check runs before the smoke test. - 502 — the target model failed its pre-flight smoke test (bad credentials, wrong endpoint, network failure). Neither the selection nor a reindex job is created.

Startup resolution behavior (AS-1853): - No embedding ModelSource selected → the process uses the legacy env-var VectorConfig path (unchanged fresh-deployment / pre-migration behavior). - Selected source missing/soft-deleted at startup → logs a warning and falls back to the legacy path. - Selected Azure source with no apiKeyEncrypted → startup fails loudly (Workload Identity is not supported on the embedding path; see Data Models), rather than silently falling back.

8.1. List Reindex Jobs¶

GET /model-gateway/selection/embedding-model/reindex-jobs — read-only, models-read scope. Since the PUT above returns no job id, this is how an operator sees whether a reindex is running or why one failed (AS-1868 leaves retries to the operator).

Query: limit (default 10, range 1–50; out-of-range → 422).

Response: 200 OK → { "jobs": [...] }, newest first (by startedAt), empty when no job has ever run. Each job carries: id, status, targetEmbeddingModelSourceId, previousEmbeddingModelSourceId, previousCollectionGeneration, requestedBy, startedAt, switchedAt, finishedAt, attempts, error, lastError, leaseOwner, leaseExpiresAt, heartbeatAt. ObjectId fields are strings; unset optionals are null.

Reading a job (status wire values are lowercase: running, completed, failed): - status running covers both the corpus sweep and the post-switch grace period. A set switchedAt means the new generation is already committed (grace is measured from there). - error is the terminal failure reason (set once the job ends failed); lastError is the most recent failed attempt's text. - attempts counts lease claims, not operator retries — a job that ran cleanly on its first try shows attempts == 1.

Access Control¶

Model sources are scope-only — no ACL, no per-record ownership.

Per-endpoint scope:

Data Models¶

ModelSource¶

Collection model_sources; indexes: (providerConfig.providerType, mode, updatedAt), (mode, deletedAt), and a text index on (displayName, description).

AwsBedrockModelConfig¶

No credential fields — AWS access uses the pod's IRSA ambient identity.

AzureOpenAIModelConfig¶

Primary Azure auth is Workload Identity (ambient, no field); apiKeyEncrypted is the optional fallback. Note: the embedding path (AS-1853) supports only the API-key fallback, not Workload Identity.

ModelGatewaySelection (singleton)¶

Enums¶

• ModelSourceProviderType: aws_bedrock, azure_openai
• ModelSourceMode: chat, embedding

Credentials & Encryption¶

• Only Azure's apiKey is a secret. It is submitted in plaintext, encrypted by the service with the registry encryption_key (same key SkillSyncSource uses), and stored as apiKeyEncrypted. It is never returned — responses expose hasApiKey: bool.
• AWS Bedrock stores no credentials (IRSA).
• The registry does not verify cloud-side invoke permission at registration time. A model the pod's IAM role / managed identity cannot actually call can still be registered; the failure surfaces later, at invocation time (AS-1852 run / AS-1853 embedding).

Model Metadata¶

GET /model-sources/{id} populates metadata live via litellm.get_model_info() keyed on bedrock/{baseModelId} or azure/{baseModelId}. This reads litellm's in-memory model-cost map (no network I/O) and is not snapshotted onto the document. When a model is not in litellm's map (e.g. an AIP ARN, or an unmapped id), metadata degrades gracefully: the fields are null and unavailableReason carries the reason — the request still returns 200, never 500.

Error Response Format¶

Two shapes — the frontend must handle both.

403 / 404 / 409 / 500 use a structured object detail (error is a machine-readable code, message is human-readable):

{ "detail": { "error": "not_found", "message": "Model source not found" } }

error codes seen on these endpoints: not_found, conflict, invalid_request, internal_error.

422 request-validation errors (invalid body, unknown providerType, missing/blank required field, explicit null on a non-nullable field) come from the framework's validation handler and use a string detail (joined field: message, truncated):

{ "detail": "modelIdOrArn: String should have at least 1 character" }