Task 3 — Observe usage, cost, and throttling

Part of the Govern AI models and agents through an API gateway lab. New here? Start with Getting started.

Starting here on its own? You need the Citadel hub deployed (azd up) and some traffic to look at. The observability views are only meaningful after you’ve sent requests through the gateway, so run Task 1 and ideally Task 2 first. From the Labfiles/G-govern-ai-through-a-gateway folder, run python setup/check_env.py --task 3 to confirm the hub is deployed.

Continuing from a previous task? The load test in Task 2 generated a burst of both successful and throttled calls — perfect data to explore here. Everything below reads existing telemetry; you don’t need to send new traffic, though you can re-run any earlier notebook cell to generate more.

Governance you can’t see isn’t governance. Because every call went through the gateway, Citadel captured a full record of it. In this task you explore that telemetry three ways: API Management (what the gateway saw), Application Insights (end-to-end traces), and Cosmos DB (token-level usage records you can turn into cost).

How does a token usage record get into Cosmos DB?

The set-llm-usage APIM policy fragment (the one you previewed in Task 1) captures token counts and model metadata from every LLM response and emits them to Event Hub. A Logic App workflow ingests those events from Application Insights into the Cosmos DB llm-usage-container. A separate model-pricing container holds per-model prices, so you can join usage against price to compute cost. That’s the whole pipeline: policy → Event Hub → Logic App → Cosmos DB.

Explore API Management analytics and metrics

In the Azure portal, open your API Management resource.
Under Monitoring, open Analytics. Explore the Timeline, Subscriptions, and Language Models tabs to see request volume over time, usage broken down by access contract (APIM subscription), and per-model metrics.
Open Metrics and add the Requests metric, split by ApiId (use a bar chart) to see how traffic distributed across your notebook runs. Add a second chart for Backend Duration, split by ApiId, to compare backend response times.

Query the gateway logs with KQL

Under Monitoring, open Logs and run a few queries against ApiManagementGatewayLogs:

Request summary by response code:

 ApiManagementGatewayLogs
 | where TimeGenerated > ago(6h)
 | summarize count() by ResponseCode, ApiId
 | order by count_ desc

Throttled requests (429s) by product:

 ApiManagementGatewayLogs
 | where TimeGenerated > ago(6h)
 | where ResponseCode == 429
 | summarize ThrottledCount = count() by ProductId, bin(TimeGenerated, 5m)
 | render columnchart

The 429s map directly to the token limits you set on each access contract in Task 2.

Trace requests in Application Insights

From your APIM resource sidebar, select Application Insights to open the connected resource.
Open Application Map to see APIM, the AI Foundry backends, and the dependency calls between them. Then open Transaction search, filter to the time range of your notebook runs, and open a request to see the full end-to-end trace: request → APIM policy execution → backend call → response.

Turn usage into cost in Cosmos DB

The token usage records need two things before you can analyze cost: the Logic App has to ingest them, and the model pricing table has to be loaded.

In the portal, open the Logic Apps service (name starts with logic-), go to Workflows, select the llm-usage-ingestion workflow, and select Run. Check Run History for a successful run.

Grant your user data-plane access to Cosmos DB and import the model pricing data. From the workshop folder with Azure CLI signed in:

 $COSMOS_ACCOUNT = az cosmosdb list --resource-group $(azd env get-value AZURE_RESOURCE_GROUP) --query "[0].name" -o tsv
 $COSMOS_RESOURCE_GROUP = azd env get-value AZURE_RESOURCE_GROUP
 $CURRENT_USER_ID = az ad signed-in-user show --query id -o tsv

 az cosmosdb sql role assignment create `
   --account-name $COSMOS_ACCOUNT `
   --resource-group $COSMOS_RESOURCE_GROUP `
   --role-definition-id 00000000-0000-0000-0000-000000000002 `
   --principal-id $CURRENT_USER_ID `
   --scope "/"

 $COSMOS_ENDPOINT = az cosmosdb show --name $COSMOS_ACCOUNT --resource-group $COSMOS_RESOURCE_GROUP --query "documentEndpoint" -o tsv
 uv run python scripts\import-model-pricing.py --endpoint $COSMOS_ENDPOINT

On macOS or Linux, use the equivalent Bash commands from the workshop readme.

Open your Cosmos DB resource → Data Explorer → the ai-usage-db database. Query the llm-usage-container to see the usage records your calls generated:
```
 SELECT * FROM c
 WHERE c._ts > (GetCurrentTimestamp()/1000 - 7200)
 ORDER BY c._ts DESC
```
Examine a record — note the model name and deployment, the prompt/completion/total token counts, the product (access contract) that made the call, and the timestamp. Then open the model-pricing container to see the per-model prices used to turn those tokens into cost.

ℹ️ Note: If a query is blocked by the Cosmos DB firewall, open the Cosmos DB resource → Settings → Networking, choose Selected networks, and add your current IP address.

✅ Checkpoint: You can see request volume and throttling in APIM, trace an individual call end-to-end in Application Insights, and read token-level usage records — attributed to the right access contract — in Cosmos DB. You’ve now closed the governance loop: enforce, then prove.

Next (optional): Task 4 — Protect sensitive data with PII policies