Create a custom copilot that uses your own data

Retrieval Augmented Generation (RAG) is a technique used to build applications that integrate data from custom data sources into a prompt for a generative AI model. RAG is a commonly used pattern for developing custom copilots - chat-based applications that use a language model to interpret inputs and generate appropriate responses.

In this exercise, you’ll use Azure AI Foundry portal to integrate custom data into a generative AI prompt flow.

This exercise takes approximately 45 minutes.

Create an Azure AI Search resource

Your copilot solution will integrate custom data into a prompt flow. To support this integration, you’ll need an Azure AI Search resource with which to index your data.

In a web browser, open the Azure portal at https://portal.azure.com and sign in using your Azure credentials.
On the home page, select + Create a resource and search for Azure AI Search. Then create a new Azure AI Search resource with the following settings:
- Subscription: Select your Azure subscription
- Resource group: Select or create a resource group
- Service name: Enter a unique service name
- Location: Make a random choice from any of the following regions*
  - Australia East
  - Canada East
  - East US
  - East US 2
  - France Central
  - Japan East
  - North Central US
  - Sweden Central
  - Switzerland
- Pricing tier: Standard
* Later, you’re going to create an Azure AI Hub (which includes an Azure OpenAI service) in the same region as your Azure AI Search resource. Azure OpenAI resources are constrained at the tenant level by regional quotas. The listed regions include default quota for the model type(s) used in this exercise. Randomly choosing a region reduces the risk of a single region reaching its quota limit in scenarios where you are sharing a tenant with other users. In the event of a quota limit being reached later in the exercise, there’s a possibility you may need to create another Azure AI hub in a different region.
Wait for your Azure AI Search resource deployment to be completed.

Create an Azure AI project

Now you’re ready to create an Azure AI Foundry project and the Azure AI resources to support it.

In a web browser, open Azure AI Foundry portal at https://ai.azure.com and sign in using your Azure credentials.
In the home page, select + Create project.
In the Create a project wizard you can see all the Azure resources that will be automatically created with your project. Select Customize and connect to your Azure AI Search resource:
- Hub name: A unique name
- Azure Subscription: Your Azure subscription
- Resource group: Select the resource group containing your Azure AI Search resource
- Location: The same location as your Azure AI Search resource
- Connect Azure AI Services or Azure OpenAI: (New) Autofills with your selected hub name
- Connect Azure AI Search: Select your Azure AI Search resource
Select Next and review your configuration.
Select Create and wait for the process to complete.

Deploy models

You need two models to implement your solution:

An embedding model to vectorize text data for efficient indexing and processing.
A model that can generate natural language responses to questions based on your data.

In the Azure AI Foundry portal, in your project, in the navigation pane on the left, under My assets, select the Models + endpoints page.
Create a new deployment of the text-embedding-ada-002 model with the following settings by selecting Customize in the Deploy model wizard:
- Deployment name: text-embedding-ada-002
- Deployment type: Standard
- Model version: Select the default version
- AI resource: Select the resource created previously
- Tokens per Minute Rate Limit (thousands): 5K
- Content filter: DefaultV2
- Enable dynamic quota: Disabled
Repeat the previous steps to deploy a gpt-35-turbo-16k model with the deployment name gpt-35-turbo-16k.

Note: Reducing the Tokens Per Minute (TPM) helps avoid over-using the quota available in the subscription you are using. 5,000 TPM is sufficient for the data used in this exercise.

Add data to your project

The data for your copilot consists of a set of travel brochures in PDF format from the fictitious travel agency Margie’s Travel. Let’s add them to the project.

Download the zipped archive of brochures from https://github.com/MicrosoftLearning/mslearn-ai-studio/raw/main/data/brochures.zip and extract it to a folder named brochures on your local file system.
In Azure AI Foundry portal, in your project, in the navigation pane on the left, under My assets, select the Data + indexes page.
Select + New data.
In the Add your data wizard, expand the drop-down menu to select Upload files/folders.
Select Upload folder and select the brochures folder.
Select Next and set the data name to brochures.
Wait for the folder to be uploaded and note that it contains several .pdf files.

Create an index for your data

Now that you’ve added a data source to your project, you can use it to create an index in your Azure AI Search resource.

In Azure AI Foundry portal, in your project, in the navigation pane on the left, under My assets, select the Data + indexes page.
In the Indexes tab, add a new index with the following settings:
- Source location:
  - Data source: Data in Azure AI Foundry portal
    - Select the brochures data source
- Index configuration:
  - Select Azure AI Search service: Select the AzureAISearch connection to your Azure AI Search resource
  - Vector index: brochures-index
  - Virtual machine: Auto select
- Search settings:
  - Vector settings: Add vector search to this search resource
  - Azure OpenAI connection: Select the default Azure OpenAI resource for your hub.
Wait for the indexing process to be completed, which can take several minutes. The index creation operation consists of the following jobs:
- Crack, chunk, and embed the text tokens in your brochures data.
- Create the Azure AI Search index.
- Register the index asset.

Test the index

Before using your index in a RAG-based prompt flow, let’s verify that it can be used to affect generative AI responses.

In the navigation pane on the left, select the Playgrounds page.
On the Chat page, in the Setup pane, ensure that your gpt-35-turbo-16k model deployment is selected. Then, in the main chat session panel, submit the prompt Where can I stay in New York?
Review the response, which should be a generic answer from the model without any data from the index.
In the Setup pane, expand the Add your data field, and then add the brochures-index project index and select the hybrid (vector + keyword) search type.

Note: Some users are finding newly created indexes unavailable right away. Refreshing the browser usually helps, but if you’re still experiencing the issue where it can’t find the index you may need to wait until the index is recognized.
After the index has been added and the chat session has restarted, resubmit the prompt Where can I stay in New York?
Review the response, which should be based on data in the index.

Use the index in a prompt flow

Your vector index has been saved in your Azure AI Foundry project, enabling you to use it easily in a prompt flow.

In Azure AI Foundry portal, in your project, in the navigation pane on the left, under Build and customize, select the Prompt flow page.
Create a new prompt flow by cloning the Multi-Round Q&A on Your Data sample in the gallery. Save your clone of this sample in a folder named brochure-flow.
Troubleshooting tip: Permissions error

If you receive a permissions error when you create a new prompt flow, try the following to troubleshoot:
- In the Azure portal, select the AI Services resource.
- Under Resource Management, in the Identity tab, confirm that it is system assigned managed identity.
- Navigate to the associated Storage Account. On the IAM page, add role assignment Storage blob data reader.
- Under Assign access to, choose Managed Identity, + Select members, select the All system-assigned managed identities, and select your Azure AI services resource.
- Review and assign to save the new settings and retry the previous step.
When the prompt flow designer page opens, review brochure-flow. Its graph should resemble the following image:

The sample prompt flow you are using implements the prompt logic for a chat application in which the user can iteratively submit text input to chat interface. The conversational history is retained and included in the context for each iteration. The prompt flow orchestrate a sequence of tools to:
- Append the history to the chat input to define a prompt in the form of a contextualized form of a question.
- Retrieve the context using your index and a query type of your own choice based on the question.
- Generate prompt context by using the retrieved data from the index to augment the question.
- Create prompt variants by adding a system message and structuring the chat history.
- Submit the prompt to a language model to generate a natural language response.
Use the Start compute session button to start the runtime compute for the flow.

Wait for the runtime to start. This provides a compute context for the prompt flow. While you’re waiting, in the Flow tab, review the sections for the tools in the flow.
In the Inputs section, ensure the inputs include:
- chat_history
- chat_input
The default chat history in this sample includes some conversation about AI.
In the Outputs section, ensure that the output includes:
- chat_output with value ${chat_with_context.output}
In the modify_query_with_history section, select the following settings (leaving others as they are):
- Connection: The default Azure OpenAI resource for your AI hub
- Api: chat
- deployment_name: gpt-35-turbo-16k
- response_format: {“type”:”text”}
Wait for the compute session to start, then in the lookup section, set the following parameter values:
- mlindex_content: Select the empty field to open the Generate pane
  - index_type: Registered Index
  - mlindex_asset_id: brochures-index:1
- queries: ${modify_query_with_history.output}
- query_type: Hybrid (vector + keyword)
- top_k: 2
In the generate_prompt_context section, review the Python script and ensure that the inputs for this tool include the following parameter:
- search_result (object): ${lookup.output}
In the Prompt_variants section, review the Python script and ensure that the inputs for this tool include the following parameters:
- contexts (string): ${generate_prompt_context.output}
- chat_history (string): ${inputs.chat_history}
- chat_input (string): ${inputs.chat_input}
In the chat_with_context section, select the following settings (leaving others as they are):
- Connection: Default_AzureOpenAI
- Api: Chat
- deployment_name: gpt-35-turbo-16k
- response_format: {“type”:”text”}
Then ensure that the inputs for this tool include the following parameters:
- prompt_text (string): ${Prompt_variants.output}
On the toolbar, use the Save button to save the changes you’ve made to the tools in the prompt flow.
On the toolbar, select Chat. A chat pane opens with the sample conversation history and the input already filled in based on the sample values. You can ignore these.
In the chat pane, replace the default input with the question Where can I stay in London? and submit it.
Review the response, which should be based on data in the index.
Review the outputs for each tool in the flow.
In the chat pane, enter the question What can I do there?
Review the response, which should be based on data in the index and take into account the chat history (so “there” is understood as “in London”).
Review the outputs for each tool in the flow, noting how each tool in the flow operated on its inputs to prepare a contextualized prompt and get an appropriate response.

Deploy the flow

Now that you have a working flow that uses your indexed data, you can deploy it as a service to be consumed by a copilot application.

Note: Depending on the region and datacenter load, deployments can sometimes take a while and will sometimes throw an error when interacting with the deployment. Feel free to move on to the challenge section below while it deploys or skip the testing of your deployment if you’re short on time.

On the toolbar, select Deploy.
Create a deployment with the following settings:
- Basic settings:
  - Endpoint: New
  - Endpoint name: Use the default unique endpoint name
  - Deployment name: Use the default deployment endpoint name
  - Virtual machine: Standard_DS3_v2
  - Instance count: 3
  - Inferencing data collection: Selected
- Advanced settings:
  - Use the default settings
In Azure AI Foundry portal, in your project, in the navigation pane on the left, under My assets, select the Models + endpoints page.
Keep refreshing the view until the brochure-endpoint-1 deployment is shown as having succeeded under the brochure-endpoint endpoint (this may take a significant period of time).
When the deployment has succeeded, select it. Then, on its Test page, enter the prompt What is there to do in San Francisco? and review the response.
Enter the prompt Where else could I go? and review the response.
View the Consume page for the endpoint, and note that it contains connection information and sample code that you can use to build a client application for your endpoint - enabling you to integrate the prompt flow solution into an application as a custom copilot.

Challenge

Now you’ve experienced how to integrate your own data in a copilot built with the Azure AI Foundry portal, let’s explore further!

Try adding a new data source through the Azure AI Foundry portal, index it, and integrate the indexed data in a prompt flow. Some data sets you could try are:

A collection of (research) articles you have on your computer.
A set of presentations from past conferences.
Any of the datasets available in the Azure Search sample data repository.

Be as resourceful as you can to create your data source and integrate it in your prompt flow. Try out the new prompt flow and submit prompts that could only be answered by the data set you chose!

Clean up

To avoid unnecessary Azure costs and resource utilization, you should remove the resources you deployed in this exercise.

If you’ve finished exploring Azure AI Foundry, return to the Azure portal at https://portal.azure.com and sign in using your Azure credentials if necessary. Then delete the resources in the resource group where you provisioned your Azure AI Search and Azure AI resources.