Task 1 — Onboard AI backends and call the Universal LLM API

Part of the Govern AI models and agents through an API gateway lab. New here? Start with Getting started.

Starting here on its own? You need the Citadel hub deployed (azd up) and a sample spoke deployed, plus the workshop Python environment. If you don’t have them yet, complete Getting started first. From the Labfiles/G-govern-ai-through-a-gateway folder, run python setup/check_env.py --task 1 to confirm you’re ready.

Before a workload can be governed, its models have to reach the gateway. In this task you register the AI model backends with API Management and then call them through the Universal LLM API — the single OpenAI-compatible endpoint that routes to the right backend based on the model field in each request.

What is a backend, and what is the Universal LLM API?

In API Management, a backend is a registered target the gateway can forward traffic to — here, each AI Foundry or Azure OpenAI model endpoint. Backends are grouped into load-balanced pools with circuit breakers so the gateway can spread traffic and fail over.

The Universal LLM API (/models) is the front door workloads call. Instead of a per-deployment URL, a caller sends an OpenAI-style request whose body names the model, and the gateway’s policies route it to the right backend pool. That indirection is what lets you govern every model call in one place.

Onboard the AI backends

The first notebook registers the model backends, backend pools, and routing policy fragments with API Management.

  1. In VS Code, open the workshop folder and open the notebook 1. llm-backend-onboarding-runner.ipynb. Select the workshop/.venv kernel.

  2. Read the Step 0 initialization cell. Notice that it discovers your governance hub resource group and the LLM_BACKENDS_CONFIG value from your azd environment — nothing is hardcoded.

    Tip: You can see the value it reads by running azd env get-value LLM_BACKENDS_CONFIG in a terminal.

  3. Run the cells in order (or use Run All), reviewing each output:

    • Steps 1–2 verify your Azure CLI sign-in and initialize the APIM client.
    • Steps 3–4 extract the current backend-pool configuration and discover the managed identity APIM uses to authenticate to the model backends.
    • Steps 5–6 generate a .bicepparam file describing the backends, then deploy the backends, pools, and policy fragments to APIM.
    • Steps 7–8 verify the deployed backends and confirm the get-available-models policy fragment was created.

    Lab context: For this lab, the same backends already provisioned by azd up are re-provisioned through the notebook. In production you’d provision new backends first, then run this notebook to register them. All backends must be included in a single run — if you register only a new backend, previously registered ones are removed.

Call the models through the gateway

The remaining sections of the same notebook prove traffic actually routes through the gateway.

  1. Run the Test GET /deployments section. This is the endpoint Microsoft Foundry uses to discover available deployments; it exercises the get-available-models fragment you just deployed.

  2. Run the Test via Universal LLM API section (POST /models/chat/completions). The gateway reads the model field in the request body and routes to the matching backend pool. Confirm you get a completion back for each discovered model.

  3. Run the Test via Azure OpenAI API and Test using Azure OpenAI Python SDK sections to see the same governed backends reached through the Azure OpenAI-compatible surface (/openai/deployments/{model}/...), including a streaming response.

  4. For an exhaustive check across every model the gateway serves, open and run 2. citadel-universal-llm-api-all-models-tests.ipynb. It provisions a temporary access contract with no model restriction (allowedModels = ""), dynamically discovers the live model catalogue with GET /models/models, and runs the right operation for each model — chat/completions, embeddings, or the full Responses API trio for GPT models. This confirms the whole catalogue is reachable through the Universal LLM API before you start restricting it in Task 2.

    The temporary contract created here previews the access-contract mechanism you’ll build deliberately in Task 2. Leave its optional cleanup cell set to False for now if you plan to continue.

Why route through the gateway instead of calling the model directly?

Calling a model endpoint directly gives you no central control: no shared spend limits, no data masking, no audit trail, and every team re-implementing the same plumbing. Routing through the gateway means one place enforces access, protection, and metering for every workload. The rest of this lab adds those controls on top of the routing you just proved works.

Explore the gateway in the portal (optional)

To see what you configured, open your API Management resource in the Azure portal and explore:

  • APIs — the Universal LLM API, Azure OpenAI API, and (if enabled) Unified AI API.
  • Backends and Load balancer — the registered model backends and the pools they’re grouped into, including circuit-breaker settings on the LLM backends.
  • APIs → Policy fragments — open set-llm-usage to preview the fragment that captures token counts for the observability you’ll use in Task 3.

✅ Checkpoint: The onboarding notebook completes without errors, and calls to the Universal LLM API and Azure OpenAI API return completions. Every model call is now flowing through the Citadel gateway.

Next: Task 2 — Enforce access contracts with model RBAC and quotas