Citadel Governance Hub - Testing & Validation Guide

Executive Summary

This testing suite provides a comprehensive, end-to-end validation framework for the Citadel Governance Hub — an enterprise-grade AI gateway built on Azure API Management (APIM). The notebooks in this directory enable platform teams to onboard LLM backends, provision access contracts for different business units, test agent framework integrations, verify PII processing capabilities, validate JWT authentication with role-based access control, and test unified AI API routing patterns — all through guided, reproducible Jupyter workflows.

The recommended execution order is:

Strongly recommended baseline (steps 1–4): these four notebooks together exercise the core gateway plumbing — backend onboarding, full model surface area, access-contract provisioning, and real-world agent consumption. Run them in order on every new Governance Hub deployment before moving on to the optional scenario-specific notebooks.

1. Backend Contracts (LLM Onboarding) — Register AI backends and deploy routing logic into APIM ⭐ strongly recommended
2. Universal LLM API — All-Models Tests — Validate every gateway-configured model (chat / embeddings / Responses API) through /models ⭐ strongly recommended
3. Access Contracts — Create per-team access contracts with Key Vault and Foundry integrations ⭐ strongly recommended
4. Agent Frameworks — Validate agent-based consumption of provisioned contracts (Microsoft Agent Framework, Foundry Agent SDK, LangChain) ⭐ strongly recommended
5. Model Aliases — Validate the shared resolve-model-alias policy fragment across all 3 LLM APIs (priority + weighted strategies, RBAC, discovery)
6. PII Processing — Test PII anonymization, deanonymization, and blocking policies
7. Unified AI API — Test multi-provider routing patterns through the Unified AI Wildcard API
8. JWT Authentication — Validate JWT-enforced and role-based access control across all API endpoints
9. Extended Providers Backend Onboarding — Onboard and validate non-Microsoft-Foundry AI backends (AWS Bedrock, GCP Gemini, Anthropic Claude) through native and OpenAI-compatible routing to confirm multi-cloud support

Each notebook is self-contained with initialization, deployment, testing, visualization, and cleanup stages, enabling both interactive exploration and repeatable validation.

Prerequisites

Before running any notebook, ensure the following are in place:

• Citadel Governance Hub deployed (Full Deployment Guide or Quick Deployment Guide)
• Azure CLI installed and authenticated (az login)
• Python 3.10+ with a virtual environment activated
• Dependencies installed:

  pip install -r ../shared/requirements.txt
  

• VS Code with the Jupyter extension (recommended for running notebooks)

Optional (per notebook)

| Capability | Required By | Details | |—|—|—| | Universal LLM API (models) imported in APIM | Universal LLM All-Models Tests, Model Aliases | Required for /models discovery and per-model operation tests | | Azure Key Vault | Access Contracts, Agent Frameworks | A Key Vault with secrets for LLM endpoint and API key | | Azure AI Foundry | Access Contracts, Agent Frameworks | A Foundry account and project for connection integration | | Azure AI Language Service | PII Processing | PII detection endpoint with managed identity access | | Event Hub | PII Processing | For PII state saving and audit logging | | Unified AI API (unified-ai) imported in APIM | Unified AI API, Model Aliases (full coverage) | Required for the wildcard /unified-ai/** routing patterns | | resolve-model-alias policy fragment + alias-aware backend onboarding | Model Aliases | Re-deployed by the notebook itself via the LLM backend onboarding Bicep with modelAliases populated | | Entra ID App Registration | JWT Authentication | Client credentials (client ID + secret) with app roles configured | | MSAL Library | JWT Authentication | Optional — for interactive device code flow token acquisition | | Google Gemini API | Unified AI API | Optional — for testing Gemini routing pattern | | Unified AI API (unified-ai) imported in APIM | Extended Providers Backend Onboarding | Required for native /bedrock/**, /gemini/**, and /claude/** routing | | AWS Bedrock / GCP Gemini / Anthropic API keys | Extended Providers Backend Onboarding | Optional per provider — fill the REPLACE_* placeholders to enable each backend’s tests | | Azure Key Vault (provider secrets) | Extended Providers Backend Onboarding | Optional — recommended over inline secrets to hold non-Azure provider keys | —

Initializing Variables from azd Environment

Every validation notebook supports a one-line init_from_azd = True toggle in its first code cell that auto-populates Citadel Governance Hub variables (resource group, location, Key Vault name, LLM backend config, Foundry account/project, Entra IDs, Cosmos account, …) directly from your active azd environment.

This is the recommended path when the accelerator was deployed with azd up, because the relevant deployment outputs (AZURE_RESOURCE_GROUP, AZURE_LOCATION, LLM_BACKEND_CONFIG, KEY_VAULT_NAME, AI_FOUNDRY_SERVICES, ENTRA_*, COSMOS_DB_ACCOUNT_NAME, …) are already written to the active azd env file by the Bicep deployment.

How It Works

Each notebook’s init cell follows the same pattern:

init_from_azd = True   # Set False to fill REPLACE values manually below

# Manual fallbacks (used when azd env var is missing OR init_from_azd = False)
governance_hub_resource_group = "REPLACE"
location                      = "REPLACE"
# ... other notebook-specific REPLACE values ...

if init_from_azd:
    loaded = utils.load_azd_env({
        "resource_group": ["AZURE_RESOURCE_GROUP", "GOVERNANCE_HUB_RESOURCE_GROUP"],
        "location":       ["AZURE_LOCATION", "LOCATION"],
        # ... notebook-specific azd vars ...
    }, verbose=False)
    # Manually-set values (anything not equal to the REPLACE sentinel) win over azd values.

The shared helper utils.load_azd_env in shared/utils.py wraps azd env get-value <NAME> per variable, supports fallback name lists, and JSON-decodes complex values (LLM_BACKEND_CONFIG, AI_FOUNDRY_SERVICES).

Pre-flight

# Make sure the right azd environment is selected
azd env list
azd env select <env-name>

# (Optional) inspect what will be auto-loaded
azd env get-values

Important: Manually edited values (anything not equal to the "REPLACE" sentinel string in the init cell) always win over values pulled from azd. Set init_from_azd = False to disable the lookup entirely and force purely manual configuration.

Per-Notebook Variable Map

The table below summarizes which azd env variables each notebook auto-loads when init_from_azd = True. Variables in italics are optional / only used when the notebook actually exercises the corresponding integration.

Multi-environment teams: Run azd env select <env-name> before launching the notebook to switch which deployment the notebook talks to. Each .azure/<env-name>/.env file is fully self-contained.

When to Set init_from_azd = False

• You are validating a Citadel Governance Hub that was not deployed via azd (e.g. deployed via az deployment sub create directly, or via a CI/CD pipeline that doesn’t write azd env vars).
• You want to point a notebook at a different environment than the currently selected azd env.
• You are running the notebook on a workstation where azd is not installed.

In any of these cases, set init_from_azd = False and fill the REPLACE values manually. The shared helpers (utils.azd_env_get, utils.azd_env_get_json, utils.load_azd_env) will silently no-op when azd isn’t available, so leaving init_from_azd = True on a machine without azd is also harmless — it just won’t override your manual values.

Notebooks

1. LLM Backend Onboarding Runner

What It Does

This notebook automates the full lifecycle of registering LLM backends with your APIM gateway. It extracts the current backend configuration, generates a Bicep parameter file with per-model metadata (SKU, capacity, model format, version), deploys the backends and policy fragments, and verifies the deployment through multiple API formats.

Steps

Key Configuration

governance_hub_resource_group = "REPLACE"  # Your Governance Hub resource group
location = "REPLACE"                       # e.g., "eastus", "swedencentral"

llm_backends_config = [
    {
        "backendId": "aif-citadel-primary",
        "backendType": "ai-foundry",           # 'ai-foundry' | 'azure-openai' | 'external'
        "endpoint": "https://...",
        "authScheme": "managedIdentity",        # 'managedIdentity' | 'apiKey' | 'token'
        "supportedModels": [
            { "name": "gpt-4o", "sku": "GlobalStandard", "capacity": 100, "modelFormat": "OpenAI", "modelVersion": "2024-11-20" }
        ],
        "priority": 1,
        "weight": 100
    }
]

Output

• Deployed APIM backends with circuit breaker support
• Backend pools with priority/weight-based load balancing
• set-backend-pools and get-available-models policy fragments
• Verified model routing through both API formats

2. Universal LLM API — All-Models Tests

What It Does

This notebook provisions a single access contract with allowedModels = "" (no model restriction), then dynamically discovers the live model catalogue via GET /models/models and exercises the appropriate OpenAI v1 operation for each model. It is the fastest way to confirm that every onboarded backend pool is end-to-end reachable through the Universal LLM API surface.

Operations Exercised Per Model

Steps

Key Configuration

governance_hub_resource_group = "REPLACE"
location                      = "REPLACE"

targetInferenceApi    = "models"               # Universal LLM API
inference_api_version = "2024-05-01-preview"
openai_api_version    = "2024-12-01-preview"

# 0 = test every discovered model; set a positive int to cap for quick smoke tests
max_models_to_test = 0

# Delay between POST /responses and the subsequent GET /responses/{id} calls
responses_get_delay_seconds = 0

Output

• One Bicep-deployed APIM product + subscription with no model RBAC restriction
• Live discovery of every gateway-configured model via GET /models/models
• Per-model results for chat, embeddings, and (where applicable) Responses API operations
• Summary table highlighting any model that failed an expected operation

3. Citadel Access Contracts Tests

What It Does

This notebook provisions three distinct access contracts, each representing a different integration pattern. It generates the Bicep parameter files, deploys the contracts as APIM products with subscriptions, performs load testing, and visualizes throttling behavior and token bucket dynamics across all contracts.

Access Contracts Created

Steps

Key Configuration

governance_hub_resource_group = "REPLACE"
location = "REPLACE"

# Optional integrations
use_keyvault_integration = True
keyvault_name = "REPLACE"

use_foundry_integration = True
foundry_account_name = "REPLACE"
foundry_project_name = "REPLACE"

Output

• Three deployed APIM products with subscription keys
• Key Vault secrets populated (if enabled)
• Foundry connection created (if enabled)
• Performance charts comparing all contracts
• Token bucket behavior visualization

4. Citadel Agent Frameworks Tests

What It Does

This notebook instantiates three AI agents — each using a different framework and integration pattern — and runs multi-turn conversations through the Citadel gateway. It measures token consumption, retry behavior, and call reliability, then produces comparative visualizations across all three frameworks.

Agent Framework Matrix

Steps

Key Configuration

governance_hub_resource_group = "REPLACE"

# Key Vault (for Sales-Assistant)
keyvault_name = "REPLACE"

# Foundry (for HR-ChatAgent)
foundry_account_name = "REPLACE"
foundry_project_name = "REPLACE"
foundry_connection_name = "HR-ChatAgent-DEV-LLM"

# Models per agent
sales_model_name = "gpt-4.1"
hr_model_name = "gpt-4o"
support_model_name = "phi-4"

Output

• Multi-turn conversation logs for each agent
• Token usage metrics (prompt, completion, total) per framework
• Retry rate and reliability analysis
• Comparative visualizations saved as agent_metrics_comparison.png

5. Citadel Model Aliases Tests

What It Does

This notebook re-runs the LLM backend onboarding Bicep with a modelAliases parameter populated, which (re)deploys the shared resolve-model-alias policy fragment with the configured aliases inlined. It then provisions an access contract scoped to alias names only (least-privilege RBAC) and exercises the same aliases through all 3 LLM API surfaces, inspecting the UAIG-* debug response headers to verify the gateway’s routing decisions end-to-end.

Two alias strategies are validated:

Steps

UAIG-* Response Headers Inspected

Key Configuration

governance_hub_resource_group = "REPLACE"   # or auto-loaded from azd
location                      = "REPLACE"
llm_backends_config           = []          # auto-loaded from LLM_BACKEND_CONFIG when init_from_azd=True

model_aliases = [
    { "name": "adv-gpt",   "models": ["gpt-5.2", "gpt-5.4-mini", "gpt-4.1"], "strategy": "priority" },
    { "name": "gpt-blend", "models": ["gpt-5.4-mini", "gpt-4.1"],            "strategy": "weighted", "weights": [70, 30] },
]

direct_test_model     = "gpt-4.1"            # control test (must exist in llm_backends_config)
inference_api_version = "2024-05-01-preview"

Output

• Re-deployed resolve-model-alias policy fragment with the two aliases inlined as a static JObject
• Access contract scoped to alias names (proves alias-name RBAC works without exposing underlying real models)
• Per-API report of UAIG-* headers showing alias → real-model resolution
• Distribution table for the weighted alias compared against the configured weights
• Filtered GET /deployments response showing aliases as first-class entries with properties.capabilities.description
• 403 negative-test confirmation that unauthorized models are blocked at the gateway

6. Citadel PII Processing Tests

What It Does

This notebook creates two specialized access contracts to test PII processing. The first contract enables PII anonymization and deanonymization with state saving (audit logging to Event Hub). The second contract enables PII blocking, which rejects any request containing detected PII. Both contracts support built-in PII categories and custom regex patterns for domain-specific identifiers.

Use Cases Tested

PII Types Covered

• Person names, email addresses, phone numbers
• Physical addresses, IBANs
• Credit card numbers (custom regex)
• Passport numbers (custom regex)
• Emirates ID (custom regex)
• Multiple PII types in a single request

Steps

Key Configuration

governance_hub_resource_group = "REPLACE"
location = "REPLACE"

# PII detection settings (configured in policy XML)
# - Confidence threshold: 0.75–0.8
# - Entity exclusions: PersonType
# - Custom regex: Credit cards, passport numbers, Emirates ID

Output

• Deployed PII masking and blocking APIM products
• Test results for 14 PII test payloads
• Validation of custom regex pattern detection
• Pass/fail summary for both anonymization and blocking modes

7. Citadel Unified AI API Tests

What It Does

This notebook validates the Unified AI Wildcard API (/unified-ai) that enables API pattern flexibility across multiple LLM providers. It deploys a test access contract, then tests Azure OpenAI, AI Foundry inference, and Gemini routing patterns, validates model discovery endpoints, verifies API key authentication, and runs a load test with throttling visualization.

API Patterns Tested

Steps

Key Configuration

governance_hub_resource_group = "REPLACE"
location = "REPLACE"

# Model configuration per backend
openai_model = "gpt-4o"
foundry_inference_model = "Mistral-Large-3"
gemini_model = "gemini-2.5-flash-lite"

# Test toggles
test_gemini = False  # Set True if Gemini backend is configured

# Load test
test_duration = 30   # Seconds

Output

• Deployed test access contract with model-scoped policy
• Model discovery results from GET /deployments
• Response validation across Azure OpenAI, Foundry, and Gemini patterns
• API key authentication enforcement
• Load test visualization (bar chart with 200/429 status codes over time)

8. Citadel JWT Authentication Tests

What It Does

This notebook tests dual authentication modes (API Key + JWT Bearer token) and role-based authorization using Entra ID app roles. It supports both service-to-service (client credentials) and interactive user (device code flow) token acquisition, and validates the unified security-handler fragment across all three API endpoint flavors (Azure OpenAI, Universal LLM, Unified AI).

Use Cases Tested

Steps

Key Configuration

governance_hub_resource_group = "REPLACE"
location = "REPLACE"

# Entra ID / OAuth
entra_tenant_id = "REPLACE"
entra_client_id = "REPLACE"
entra_client_secret = "REPLACE"

# Model and API versions
model_name = "gpt-4.1"
openai_api_version = "2024-12-01-preview"

# Token source: "client_credentials" or "device_flow"
token_source = "client_credentials"

# Role configuration
requiredRoles = "Models.Read"

Output

• Deployed JWT-enforced and role-enforced APIM products
• Test results for 7 scenarios across 4 API endpoints
• Decoded JWT token inspection (claims, roles, lifetime)
• PASS/FAIL summary table with overall pass rate

9. Extended Providers Backend Onboarding (AWS Bedrock + GCP Gemini + Anthropic Claude)

What It Does

This notebook extends the base LLM backend onboarding flow with three non-Azure backends authenticated by simple API keys (no AWS SigV4 or Workload Identity Federation required for validation). It generates a .bicepparam file with the new backend definitions, re-deploys the onboarding Bicep, provisions an access contract that allows every onboarded model, then validates each backend through its native API surface and its OpenAI-compatible surface. Each test cell self-skips when a provider’s credentials are still REPLACE_*, so partial configurations never produce failures.

Backends Onboarded

API Surfaces Validated

Steps

Key Configuration

init_from_azd = False   # keeps REPLACE_* provider placeholders; set True to merge azd Foundry backends

governance_hub_resource_group = "REPLACE"
location                      = "REPLACE"

# Provider credentials — leave as REPLACE_* to skip a backend's tests
aws_bedrock_region           = "eu-north-1"
aws_bedrock_native_inline    = "REPLACE_AWS_BEDROCK_BEARER_VALUE"   # full "Bearer sk-..."
gemini_native_inline         = "REPLACE_GEMINI_RAW_KEY"            # raw key for x-goog-api-key
anthropic_inline             = "REPLACE_ANTHROPIC_RAW_KEY"         # raw key for x-api-key
anthropic_version            = "2023-06-01"

# Optional Key Vault to hold provider secrets (recommended over inline values)
key_vault_name = "kv-REPLACE"

Secret format note: APIM substitutes `` on a Backend resource only when it is the entire header value (no concatenation). Bearer-auth secrets must therefore store the complete header value (Bearer sk-...), while direct-key headers (x-api-key, x-goog-api-key) store the raw key. A provider with both a native and an OpenAI-compat surface (e.g. Gemini) needs two separate secrets.

Output

• Three native + two OpenAI-compat backends onboarded alongside the existing Foundry backends
• Access contract allowing every onboarded model and alias
• Per-surface validation results (native and OpenAI-compat) with UAIG-* debug headers
• Optional model-alias scenarios (single-cloud weighted, cross-cloud OpenAI-compat, native Anthropic) that self-skip when members are unavailable

Recommended Execution Order

Strongly recommended baseline: run notebooks 1 → 4 in order on every new Citadel Governance Hub deployment. Steps 5 → 8 are optional, scenario-specific validations that can be run independently afterwards.

┌──────────────────────────────────────────────┐
│  1. llm-backend-onboarding-runner            │  ⭐ Onboard LLM backends & routing
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│  2. citadel-universal-llm-api-all-models-    │  ⭐ Smoke-test EVERY onboarded model
│     tests                                    │     (chat / embeddings / Responses API)
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│  3. citadel-access-contracts-tests           │  ⭐ Create access contracts & load test
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│  4. citadel-agent-frameworks-tests           │  ⭐ Test agent frameworks
│                                              │     (uses contracts from step 3)
└──────────────┬───────────────────────────────┘
               │   ── End of strongly recommended baseline ──
               ▼
┌──────────────────────────────────────────────┐
│  5. citadel-model-aliases-tests              │  Optional: alias routing
│                                              │     (priority + weighted strategies)
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│  6. citadel-pii-processing-tests             │  Optional: PII masking & blocking
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│  7. citadel-unified-ai-api-tests             │  Optional: Unified AI wildcard API
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│  8. citadel-jwt-authentication-tests         │  Optional: JWT auth & RBAC
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│  9. llm-backend-onboarding-extended-         │  Optional: AWS Bedrock + GCP Gemini +
│     providers-runner                         │     Anthropic Claude (native + OpenAI-compat)
└──────────────────────────────────────────────┘

Note: Notebooks 5–9 create their own access contracts and can be run independently after backend onboarding. Notebook 5 re-deploys the LLM backend onboarding Bicep with modelAliases populated (the resolve-model-alias fragment is regenerated; full cross-API coverage requires the Unified AI API to be imported), notebook 6 requires PII policy fragments (pii-anonymization, pii-deanonymization, pii-state-saving), notebook 7 requires the Unified AI API (unified-ai) to be imported into APIM, notebook 8 requires JWT configuration plus an Entra ID app registration, and notebook 9 extends backend onboarding with non-Azure providers (native /bedrock/**, /gemini/**, and /claude/** routing requires the Unified AI API to be imported).

Shared Utilities

All notebooks import shared helper modules from the ../shared/ directory:

Cleanup

Each notebook includes an optional cleanup cell at the end that removes the APIM products and subscriptions created during testing. Cleanup is controlled by a cleanup_enabled flag (default: True).

Important: Cleanup does not remove Azure Key Vault secrets, Foundry connections, or LLM backend configurations. Those resources are managed separately.

Troubleshooting