Citadel AI Governance Hub - Hands-On Lab Guide

Duration: 3 hours
Level: Intermediate
Last updated: June 2026

This hands-on lab deploys a governed AI hub-and-spoke architecture into your Azure subscription:

• Citadel Hub — the shared governance plane centered on API Management, observability, policy enforcement, and core AI platform resources.
• Citadel Spoke — a sample Azure AI Foundry workload that connects to the hub as a governed use case.
• Lab environment — your local machine or the included Devcontainer, where deployment commands, notebooks, and validation traffic originate.

You deploy the hub with azd up, then deploy the sample spoke with workshop/scripts/deploy-spoke-foundry.*.

Lab at a glance

Complete the Core rows first (about 1.5 hours) — setup, deployment, a couple of notebooks, and a first look at the metrics get you a working, governed AI gateway end to end. From there, expand into the Standard rows and the Optional notebooks as time allows. The full lab, including all optional notebooks, takes about 4.5 hours. Use the buttons below to auto-expand the optional notebooks that match the time you have.

⏳ Deployment dominates the clock. azd up takes 30–45 minutes on every path. Read ahead through the review and notebook sections while it runs.

Lab overview

In this hands-on lab you will:

1. Deploy the Citadel Hub to your own Azure subscription using Azure Developer CLI (azd)
2. Deploy a sample Citadel Spoke by running the workshop/scripts/deploy-spoke-foundry.* script after azd up completes
3. Review the deployed Azure services and their configuration — focusing on API Management as the unified AI gateway between clients and AI backends
4. Run validation notebooks from your lab environment to exercise Citadel capabilities (backend onboarding, access contracts, PII processing, etc.)
5. Observe telemetry generated by your notebook runs — explore APIM metrics, Application Insights, Log Analytics, and Cosmos DB usage records

Prerequisites

Complete these steps before the lab day. Before you begin, ensure you have:

• An Azure subscription with Owner (or Contributor plus User Access Administrator) permissions — azd up creates managed identities and assigns RBAC roles.
• Sufficient Azure OpenAI / AI Foundry model quota (GPT-4.1, DeepSeek-R1, etc.) in your target region.
• The lab tools installed — Azure CLI, Azure Developer CLI (azd), Python 3.13+, Git, and VS Code — or the included Devcontainer.
• Basic familiarity with the Azure services Citadel uses (API Management, AI Foundry, Cosmos DB, Key Vault, Event Hub) and with running Jupyter notebooks.

Note: Detailed install links, the required Azure resource-provider registration, the permission rationale, and the Devcontainer option are in the Lab setup guide. Complete those steps before you begin.

Deploy Citadel to your Azure subscription

Goal: Deploy the Citadel Hub first, then deploy a sample Citadel Spoke that connects to it.
Estimated time: 45 minutes

Clone the Repository

git clone https://github.com/mohamedsaif/ai-hub-gateway-solution-accelerator.git
cd ai-hub-gateway-solution-accelerator
git checkout workshop

mkdir citadel-workshop && cd citadel-workshop
azd init --template https://github.com/mohamedsaif/ai-hub-gateway-solution-accelerator.git -e citadel-workshop --branch workshop

Authenticate to Azure

# Login to Azure CLI
az login

# Login to Azure Developer CLI (uses the same browser auth)
azd auth login

# Confirm you are on the correct subscription
az account show --output table

If you need to target a specific tenant:

az login --tenant-id <your-tenant-id>
azd auth login --tenant-id <your-tenant-id>

Create a New Environment

azd env new citadel-workshop

Configure Environment (Optional)

This section is optional and can be skipped for the lab. It is provided in case you want to experiment with specific settings.

azd env set AZURE_LOCATION swedencentral

# Optional — disable components to speed up deployment and reduce costs
azd env set ENABLE_API_CENTER false
azd env set ENABLE_MANAGED_REDIS false
azd env set ENABLE_AZURE_AI_SEARCH false

Deploy

Recommended region: For this lab, Sweden Central (swedencentral) is the recommended region.

azd up

The CLI will prompt you to:

1. Select your Azure subscription
2. Confirm the Azure region

Then the deployment begins. This takes approximately 30-45 minutes.

⏳ While waiting: Work through the companion Microsoft Learn module Govern AI at scale to build the conceptual background behind Citadel. (Preview link — sign-in required; it will be replaced by a public link later.)

💡 While you wait — just-in-time learning. Expand the sections below to understand what azd up is doing behind the scenes while it runs.

🔁 Transient errors: Deployment of some resources, such as model deployments, may occasionally fail with a transient error. If azd up fails, re-run the command — it will pick up where it left off.

If your deployment uses the Developer APIM SKU (APIM_SKU=”Developer”), run the APIM cleanup script immediately after azd up completes. The script checks the live APIM SKU and only removes the default APIM products and subscriptions when the SKU is Developer.

.\workshop\scripts\cleanup-apim-defaults.ps1

./workshop/scripts/cleanup-apim-defaults.sh

Verify Deployment

Once azd up completes:

• The Citadel Hub is deployed.
• The sample Citadel Spoke is not deployed yet.

# View deployment outputs (resource names, endpoints, etc.)
azd env get-values

Deploy the Sample Spoke

After azd up finishes, run the spoke deployment script from the repository root. Pass a unique spoke suffix so the script creates a distinct spoke resource group and unique resource names for the Foundry account, Key Vault, Log Analytics workspace, Application Insights, and Azure Container Registry.

.\workshop\scripts\deploy-spoke-foundry.ps1 -SpokeSuffix 1

./workshop/scripts/deploy-spoke-foundry.sh --spoke-suffix 1

💡 While the spoke deploys — just-in-time learning.

This step deploys a sample Citadel Spoke: a standalone Azure AI Foundry account and project that represent an onboarded workload (use case). For example, if the hub resource group is rg-citadel-demo-1, -SpokeSuffix 1 creates a spoke resource group named rg-citadel-demo-1-spoke-1. To create another spoke without editing the scripts, rerun the same command with a different suffix:

.\workshop\scripts\deploy-spoke-foundry.ps1 -SpokeSuffix 2
.\workshop\scripts\deploy-spoke-foundry.ps1 -SpokeSuffix 10

./workshop/scripts/deploy-spoke-foundry.sh --spoke-suffix 2
./workshop/scripts/deploy-spoke-foundry.sh --spoke-suffix 10

Verify Deployment

Once the script completes, there is no need to note down any values — the notebooks read directly from azd env variables at runtime.

You can verify in the Azure Portal:

1. Go to portal.azure.com
2. Navigate to the hub resource group and confirm the shared Citadel resources are deployed (APIM, Event Hub, Cosmos DB, Key Vault, Logic App, AI Foundry, etc.)
3. Navigate to the spoke resource group created by the script and confirm the sample Azure AI Foundry resources are deployed there

Review deployed services and configuration

Goal: Understand the services Citadel deploys and how they are configured.
Estimated time: 30 minutes

Resource Group Overview

Navigate to your resource group in the Azure Portal. You should see resources including:

Explore AI Foundry

ℹ️ Note: After azd up, the AI Foundry resources in Citadel Hub have public network access disabled by design. Related Foundry projects are not accessible through the Foundry portal, and users will see “Private network access required”. This is expected.

Explore API Management

API Management is the core of Citadel. Explore these areas:

APIs

1. Navigate to your API Management resource → APIs
2. You should see:
   • Azure OpenAI API — Standard Azure OpenAI compatible endpoint
   • Universal LLM API — Citadel’s universal inference endpoint
   • Unified AI API — Multi-provider wildcard API (if enabled)
3. Go to Universal LLM API → Settings. Scroll down to Diagnostics Logs and select Azure Monitor. Observe that “Log LLM messages” is enabled. This is also the case with other APIs — Azure OpenAI API and Unified AI API.

🔎 Optional: Select an API, such as Universal LLM API, open the Test tab, and search for the “chat” operation (“Gets chat completions for the provided chat messages.”). In the request body, change the model field from gpt-4o to gpt-4.1. Click Send and observe the reply.

Products

1. Go to Products — each product represents an “access contract” for a team/application

ℹ️ Note: Additional products will be created when you run the validation notebooks.

Policies

1. Select an API (e.g. Universal LLM API) → Select All operations → Inbound processing → Policy code editor
2. Review the XML policies — pay attention to <include-fragment> policy blocks. Note down the policy block with fragment-id="set-llm-usage". In the next step, go to APIs → Policy Fragments, select the set-llm-usage policy fragment, then go to Settings → Policy editor to review its policy code.

Backends

1. Go to Backends — see registered AI Foundry / OpenAI backends
2. Note the circuit breaker configuration on LLM backends (not all backends have circuit breakers — only the LLM-related ones)
3. Go to Load balancer — see how backends are grouped into pools for load distribution

Explore Supporting Services

Cosmos DB:

1. Go to Settings → Networking and enable Selected networks. Under the Firewall option, add your current IP address and/or accept connections from within public Azure datacenters.
2. Note that this is only for the purposes of this lab. In a real implementation, networking should be configured to fit your specific networking setup.

Key Vault:

1. Review the Key Vault resource in the spoke resource group — note that there are no secrets stored immediately after azd up
2. As part of the notebooks execution, secrets (such as access contract subscription keys) will be created here

Run validation notebooks

Goal: Exercise Citadel’s capabilities by running validation notebooks.
Estimated time: 60 minutes

Set Up Python Environment

If you are using the Devcontainer, the post-create setup installs the lab dependencies automatically. Confirm the selected Python interpreter points to workshop/.venv, then continue to Section 5.2.

Navigate to the workshop folder in your terminal:

cd workshop

# Install uv if you don't have it
# Windows:
# powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
# macOS/Linux:
# curl -LsSf https://astral.sh/uv/install.sh | sh

# Create environment and install dependencies
uv sync

# Create a virtual environment with Python 3.13
# Windows:
py -3.13 -m venv .venv
# macOS/Linux:
# python3.13 -m venv .venv

# Activate it
# Windows:
.venv\Scripts\activate
# macOS/Linux:
# source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Open Notebooks in VS Code

1. Open the workshop folder in VS Code
2. Open a notebook file (.ipynb) - see 5.4 Recommended Notebook Execution Order
3. Select the Python kernel from your virtual environment (.venv or uv managed). In the Devcontainer, use the preconfigured workshop/.venv interpreter.

Configure Notebook Variables

All required variables (resource group name, Azure region, APIM endpoint, etc.) are automatically configured in the notebooks. The initialization cell (Step 0) in each notebook retrieves these values from your azd environment and Azure resource metadata at runtime — you do not need to manually set anything.

Take a moment to review how this works in the code: the notebooks use azd env get-values and Azure CLI commands to discover resource names and endpoints, then inject them as Python variables used throughout the notebook.

Recommended Notebook Execution Order

This lab includes a subset of validation notebooks. Execute them in this order:

Lab context: For the purposes of this lab, the same backends that were already provisioned by azd up will be re-provisioned through the notebook. In a real-world scenario, you would first provision new backends (e.g. a new AI Foundry account) and then use this notebook to register them with APIM. Note that all backends must be included in a single provisioning run — if only one new backend is added via the notebook, previously registered backends will be removed. For example, if backend1 and backend2 are already registered and you want to add backend3, all three must be specified in the notebook.

Optional Notebooks (7–9)

The following three notebooks are optional and can be run after notebooks 1–6. They are net-new labs and do not have equivalent notebooks in the validation/ folder. Unless noted, they share the same pre-requisites as notebooks 1–6 (deployed hub + spoke, Azure CLI login, lab Python environment).

Running Notebooks — Tips

• ✅ Run All is fine — you can use Run All to execute the entire notebook, but review the output of each cell afterwards to understand what happened
• 🔐 Azure CLI auth — ensure you are logged in with az login before running notebooks; this is required for notebooks to discover resources and authenticate
• 🧹 Cleanup — do not clean up resources created by notebooks individually; at the end of the lab, run azd down --purge --force
• 🛠️ Errors? — Check the Troubleshooting section

Observability: Metrics, logs, and telemetry

Goal: After running notebooks, explore the telemetry and usage data generated by your API calls.
Estimated time: 30 minutes

API Management — Built-in Analytics

1. Navigate to your API Management resource

2. Analytics (under Monitoring):
   • Explore the Timeline tab to see request volume over time
   • Switch to the Subscriptions tab to view usage broken down by APIM subscription
   • Open the Language Models tab to observe per-model metrics
3. Metrics:
   • Go to Metrics
   • Add metric: Requests → Split by ApiId — observe how traffic distributed across your notebook runs. Use Bar Chart for visualization.
   • Create new chart, then add a new metric: Backend Duration → Split by ApiId, to see backend response times
4. Logs:
   • Go to Logs (under Monitoring)
   • Run sample queries to explore APIM gateway activity:

   Request summary by response code:

   ApiManagementGatewayLogs
   | where TimeGenerated > ago(6h)
   | summarize count() by ResponseCode, ApiId
   | order by count_ desc
   

   Request volume by API:

   ApiManagementGatewayLogs
   | where TimeGenerated > ago(6h)
   | summarize RequestCount = count() by ApiId, bin(TimeGenerated, 5m)
   | render columnchart
   

   Latency percentiles:

   ApiManagementGatewayLogs
   | where TimeGenerated > ago(2h)
   | summarize 
       P50 = percentile(TotalTime, 50),
       P95 = percentile(TotalTime, 95),
       P99 = percentile(TotalTime, 99)
       by ApiId
   

   Throttled requests (429s):

   ApiManagementGatewayLogs
   | where TimeGenerated > ago(6h)
   | where ResponseCode == 429
   | summarize ThrottledCount = count() by ProductId, bin(TimeGenerated, 5m)
   | render columnchart
   

Application Insights — APIM Telemetry

From your APIM resource, click the Application Insights option in the sidebar to navigate directly to the associated Application Insights resource.

1. Application Map — Visualize how components are connected:
   • Go to Application Map
   • See APIM, AI Foundry backends, and dependency calls
2. Transaction Search — Drill into individual requests:
   • Go to Transaction search
   • Filter by time range covering your notebook runs
   • Click on a request to see full end-to-end trace (request → APIM policy execution → backend call → response)

Cosmos DB — Usage Records

Populate data first: Before querying Cosmos DB, you need to run the Logic App workflow that ingests usage data from Application Insights into Cosmos DB:

1. In the Azure Portal, open the Logic Apps service (the resource name starts with logic-)
2. Navigate to Workflows → Workflows
3. Select the llm-usage-ingestion workflow
4. Click Run to trigger the workflow manually
5. Check Run History — wait until you see a successful run
6. Explore the run results to observe how data flows from App Insights to Cosmos DB

Once you have a successful run, proceed with importing model pricing data.

Import model pricing data: The model-pricing container stores per-model cost information used for usage analytics. This data is imported from a local JSON file (model-pricing.json) included in the workshop/scripts folder. The import script authenticates via Entra ID and requires the Cosmos DB Built-in Data Contributor role on your Cosmos DB account.

1. First, retrieve your Cosmos DB account name and assign the required role to your current user:

   💻 Bash

   # Get the Cosmos DB account name from your deployment
   COSMOS_ACCOUNT=$(az cosmosdb list --resource-group $(azd env get-value AZURE_RESOURCE_GROUP) --query "[0].name" -o tsv)
   COSMOS_RESOURCE_GROUP=$(azd env get-value AZURE_RESOURCE_GROUP)
   CURRENT_USER_ID=$(az ad signed-in-user show --query id -o tsv)
   
   # Assign "Cosmos DB Built-in Data Contributor" role (role definition id: 00000000-0000-0000-0000-000000000002)
   az cosmosdb sql role assignment create \
     --account-name $COSMOS_ACCOUNT \
     --resource-group $COSMOS_RESOURCE_GROUP \
     --role-definition-id 00000000-0000-0000-0000-000000000002 \
     --principal-id $CURRENT_USER_ID \
     --scope "/"
   

   🪟 PowerShell

   $COSMOS_ACCOUNT = az cosmosdb list --resource-group $(azd env get-value AZURE_RESOURCE_GROUP) --query "[0].name" -o tsv
   $COSMOS_RESOURCE_GROUP = azd env get-value AZURE_RESOURCE_GROUP
   $CURRENT_USER_ID = az ad signed-in-user show --query id -o tsv
      
   az cosmosdb sql role assignment create `
      --account-name $COSMOS_ACCOUNT `
      --resource-group $COSMOS_RESOURCE_GROUP `
      --role-definition-id 00000000-0000-0000-0000-000000000002 `
      --principal-id $CURRENT_USER_ID `
      --scope "/"
   

2. Run the import script to load model pricing data into Cosmos DB:

   💻 Bash

   # Get the Cosmos DB endpoint
   COSMOS_ENDPOINT=$(az cosmosdb show --name $COSMOS_ACCOUNT --resource-group $COSMOS_RESOURCE_GROUP --query "documentEndpoint" -o tsv)
   
   # Import model pricing data
   uv run python scripts/import-model-pricing.py --endpoint $COSMOS_ENDPOINT
   

   🪟 PowerShell

   $COSMOS_ENDPOINT = az cosmosdb show --name $COSMOS_ACCOUNT --resource-group $COSMOS_RESOURCE_GROUP --query "documentEndpoint" -o tsv
   uv run python scripts\import-model-pricing.py --endpoint $COSMOS_ENDPOINT
   

   You should see output confirming each model pricing record was upserted.

Data Explorer: Now navigate to Cosmos DB to explore both usage records and model pricing:

1. Navigate to your Cosmos DB resource
2. Open Data Explorer
3. Look for the ai-usage-db database
4. Review the two important containers:
   • llm-usage-container — stores usage records generated by notebook calls
   • model-pricing — stores model pricing data imported by the script
5. Select ai-usage-db database → llm-usage-container

6. Run a query to see usage records generated by your notebook calls:

   SELECT * FROM c 
   WHERE c._ts > (GetCurrentTimestamp()/1000 - 7200) 
   ORDER BY c._ts DESC
   

7. Examine a usage record — note fields like:
   • Model name and deployment
   • Token counts (prompt, completion, total)
   • Product that made the call
   • Timestamp

   💡 Tip: These usage records are generated by the set-llm-usage APIM policy fragment you reviewed in Lab 2 (Section 4.3). That policy fragment captures token counts and model metadata from each LLM call and emits them to Event Hub, which the Logic App workflow then ingests into Cosmos DB.

8. Also explore the model-pricing container to see how model pricing data is stored

ℹ️ Note: If you get a warning that the request is blocked by your Cosmos DB account firewall settings, go to your Cosmos DB resource → Settings → Networking, choose Selected networks, and add your current IP address.

Clean up

After the lab, remove all deployed resources to avoid ongoing costs. Perform these steps in order — tear down the optional MCP lab resources first, then the Citadel hub, then any spokes.

Tear Down MCP Lab Resources (only if you ran optional notebook 9)

If you ran optional notebook 9 (HR MCP via APIM), tear down the HR MCP resources before running azd down for the Citadel hub. The MCP deployment creates resources inside the shared hub VNet (a dedicated ACA subnet, private DNS zone link, etc.), so removing them first avoids leaving orphaned dependencies that can block hub deletion. Run the teardown script from the workshop/ folder:

bash ./mcp-hr/scripts/teardown-hr-mcp.sh

pwsh -File ./mcp-hr/scripts/teardown-hr-mcp.ps1

The teardown is best-effort and idempotent — resources that are already gone are skipped.

⏭️ Skip this step if you did not run notebook 9.

Tear Down the Citadel Hub (azd down)

Remove the hub resources that were deployed via azd up:

# This will delete ALL resources in the resource group and purge soft-deleted resources
azd down --purge --force

🛠️ If azd down fails: Go to the Azure Portal, navigate to your hub resource group, and delete the deployment manually.

⚠️ Important: The --purge flag ensures soft-deleted resources (Key Vault, Cognitive Services) are permanently removed. The --force flag skips confirmation prompts.

Tear Down the Spoke(s) (via Azure Portal)

azd down only removes the hub resources that were deployed via azd up. You must also manually delete any spoke resource groups in the Azure Portal because they were deployed separately by script.

1. Go to portal.azure.com
2. Locate each spoke resource group (e.g. rg-citadel-demo-1-spoke-1)
3. Delete the resource group

Verify Cleanup

Verify in the Azure Portal that the hub resource group and any spoke resource groups have been deleted.

Troubleshooting

Deployment Issues

# Register the missing provider
az provider register --namespace <ProviderNamespace>
# Wait a minute, then retry
azd up

Notebook Issues

# Run in your terminal
az login
# Restart the Jupyter kernel in VS Code

# Ensure you activated the correct virtual environment
# For uv:
uv sync
# For pip:
pip install -r requirements.txt

Observability Issues

Appendix

Key Azure Resources Deployed

Useful azd Commands

azd env get-values          # View all deployment outputs
azd env list                # List all environments
azd monitor --overview      # Open Application Insights in browser
azd monitor --live          # Open Live Metrics in browser
azd monitor --logs          # Open Log Analytics in browser
azd down --purge --force    # Delete all resources

Useful Links

• Citadel Governance Hub Repository
• Quick Deployment Guide
• Full Deployment Guide
• Parameters Usage Guide
• Azure API Management Documentation
• Azure Developer CLI Documentation