🏔 DPO Fine-Tuning the Adventure Works Trail Guide

🧭 Agenda

This notebook walks through end-to-end Direct Preference Optimization (DPO) using gpt-4.1-mini on Azure OpenAI / Microsoft Foundry, applied to our trail recommendation agent:

1. Dataset & the Tone Problem — preferred vs. rejected response pairs
2. Setup & Connect to Foundry — environment, credentials, file upload
3. Evaluate Base Model — tone pass rate before fine-tuning
4. DPO Training: Launch & Monitor — adjust hyperparameters and rerun!
5. Results & Comparison — did preference fine-tuning fix the tone?
6. Key Takeaways — when to use DPO

📋 Section 1: Dataset & the Tone Problem

The Adventure Works Trail Guide agent was flagged for using inappropriate, dismissive tone when users ask about trails involving real safety risks — such as the Knife's Edge ridge.

Users rated responses as unhelpful or condescending when they asked reasonable questions like:

"I'm a moderate hiker. Is the Knife's Edge trail suitable for me?"

DPO requires a preference dataset of paired responses for each prompt:

• preferred_output — a response with appropriate tone: safety-aware, respectful, and actionable
• non_preferred_output — a response with poor tone: dismissive, condescending, or overly alarming

📦 DPO Training Data Format

Each record in training.jsonl follows this structure:

{
  "input": {
    "messages": [
      {"role": "system", "content": "You are an Adventure Works trail guide..."},
      {"role": "user", "content": "I'm a moderate hiker. Is Knife's Edge suitable for me?"}
    ]
  },
  "preferred_output":     [{"role": "assistant", "content": "...respectful, helpful response..."}],
  "non_preferred_output": [{"role": "assistant", "content": "...dismissive or condescending response..."}]
}

💡 Why DPO for This Problem?

The agent's core hiking knowledge is correct — it's only the tone that's wrong. DPO is ideal here because:

• We have clear human judgements: preferred vs. non-preferred tone
• There is no single "correct" answer to rewrite — many phrasings work as long as tone is right
• DPO directly optimises for the preference signal without altering factual knowledge

✅ DPO is the right tool when you know which response is better, but not exactly what the response should say.

🗂 Example Training Pair

🔌 Section 2: Setup & Connect to Microsoft Foundry

We use the azure-ai-projects SDK to connect to Microsoft Foundry and the azure-ai-evaluation SDK to score model responses against our tone criteria.

MICROSOFT_FOUNDRY_PROJECT_ENDPOINT=<your-endpoint>
AZURE_SUBSCRIPTION_ID=<your-subscription-id>
AZURE_RESOURCE_GROUP=<your-resource-group>
AZURE_AOAI_ACCOUNT=<your-foundry-account-name>
MODEL_NAME=gpt-4.1-mini
AZURE_OPENAI_ENDPOINT=<your-azure-openai-endpoint>
AZURE_OPENAI_KEY=<your-azure-openai-api-key>
DEPLOYMENT_NAME=gpt-4.1-mini

📊 Section 3: Evaluating the Base Model Tone

Before fine-tuning, we measure how often the base gpt-4.1-mini model produces an appropriate tone on trail questions. We score responses on three dimensions:

The Tone Pass Rate is computed by a model-based evaluator that classifies each response as passing or failing the tone standard defined in our DPO preference pairs.

🚀 Section 4: DPO Training — Configure, Launch & Monitor

With our preference pairs uploaded, we launch a DPO fine-tuning job on gpt-4.1-mini. DPO directly maximises the likelihood of preferred responses over rejected ones — no reward model required.

🎛 DPO Hyperparameters

🎛 Try it: Adjust the hyperparameters below, click ⚙ Apply, then re-run the training cells to compare outcomes!

🧠 Section 5: Deploy, Test & Evaluate the Fine-Tuned Model

Training is complete. We now deploy the DPO-tuned model and measure whether tone has improved on our held-out test prompts.

Did your hyperparameter choices pay off? Run the cells below to find out!

🔁 Bonus: Continual Fine-Tuning

Once you have a deployed DPO-tuned model, you can continue improving it iteratively — using the fine-tuned model as the new base for a subsequent DPO run with fresh preference pairs.

When to use continual fine-tuning

• You have collected new preference data from production feedback
• Your agent is expanding to new trail categories that have different tone challenges
• You want to iterate quickly without reprocessing the original training set

🎯 Key Takeaways

Run the evaluation cells above to see your results, then this section will summarise what you learned.