Build Native Search Over a Product Catalog in C#

In this exercise, you add full-text and vector search to a CosmicWorks product catalog in Azure Cosmos DB for NoSQL. You load searchable text and embeddings, run keyword and similarity queries, and use the change feed to refresh an embedding after renaming a product.

The setup script configures the container’s full-text and vector policies and indexes. Your application prepares the searchable text and generates the embeddings.

This exercise takes approximately 60 minutes to complete.

Before you start

To complete this exercise, you need an Azure subscription with permission to create resources and assign roles.

If your lab environment isn’t set up yet, follow Set up your lab environment to install Visual Studio Code, Git, the Azure CLI, and PowerShell 7.

You also need the .NET 10 SDK or later installed.

The setup script creates a Microsoft Foundry resource, project, and text-embedding-3-small deployment. Your subscription needs quota for that model in the Foundry region. You don’t need an existing Foundry project.

Set up your Azure Cosmos DB resources

This exercise creates its own Azure Cosmos DB account and deletes it at the end, so it doesn’t use the shared account from the rest of this learning path.

Start Visual Studio Code.
If you don’t have the lab code yet, clone the repository for DP-420: open the command palette with Ctrl+Shift+P, run Git: Clone, and enter the following URL. Choose a local folder when prompted. Otherwise, open the folder from your previous clone.
```
 https://github.com/microsoftlearning/dp-420-cosmos-db-dev
```
Once the repository is cloned, open that local folder in Visual Studio Code.
In the Explorer pane, browse to the Allfiles/Labs/Shared folder.
Open the context menu for the folder and select Open in Integrated Terminal. If the terminal isn’t PowerShell, select the dropdown beside the + in the terminal toolbar and choose PowerShell.
Sign in to the Azure CLI and follow the sign-in prompt.
```
 az login
```
Set variables for the resource group and region. If your lab environment provides a resource group, use that name. Otherwise, use a new group that contains only this exercise’s resources.
```
 $resourceGroup = "ResourceGroup1"
 $location = "eastus"
```

Run the setup script.

 ./setup.ps1 -ResourceGroup $resourceGroup -Location $location -NamePrefix dp420lab16 -LabProfile search -AccountOnly -EnableFoundry -EmbeddingOnly -FoundryLocation $location

The script generates a globally unique account name, such as dp420lab16a7f3k9.

Wait for the script to finish. The whole script takes 5-10 minutes to run.
Record the Account name, Account endpoint, Foundry account, and OpenAI endpoint values the script prints. The Cosmos DB endpoint looks like https://<your-account-name>.documents.azure.com:443/. The script also saves the Foundry values in logs/foundry-<account-name>.json.
Set the account name the script printed, then open that account in the Azure portal.
```
 $accountName = "<your-account-name>"
```
Under Settings, select Features. Confirm Vector Search for NoSQL API is enabled, then enable Full Text & Hybrid Search for NoSQL API. Allow up to 15 minutes for enrollment. Confirm both features are enabled before continuing.

Complete setup and verify the resources using the account name you recorded.

 ./setup.ps1 -ResourceGroup $resourceGroup -Location $location -AccountName $accountName -LabProfile search -SearchFeaturesReady -EnableFoundry -EmbeddingOnly -FoundryLocation $location
 ./verify.ps1 -ResourceGroup $resourceGroup -AccountName $accountName -LabProfile search -EnableFoundry -EmbeddingOnly

Continue after both commands succeed.

After both stages, the script creates the following resources:

Resource	Configuration
Azure Cosmos DB account	API for NoSQL, with key-based authentication disabled and the `EnableNoSQLVectorSearch` capability
`cosmicworks` database	Holds the container this exercise uses
`productSearch` container	Partitioned on `/categoryId`, autoscale up to 1,000 RU/s, empty. Full-text policy and index on `/searchText`, vector policy and `diskANN` index on `/embedding` at 1,536 dimensions with cosine distance
Role assignment	Cosmos DB Built-in Data Contributor, granted to your signed-in identity
Microsoft Foundry resource and project	Key-based authentication disabled; project named `dp420`
Embedding deployment	`text-embedding-3-small`, version `1`, Standard deployment, 30 capacity units; produces 1,536-dimensional embeddings by default
Foundry role assignment	Foundry User, granted to your signed-in identity on the Foundry resource

The applications use DefaultAzureCredential for Microsoft Entra ID authentication. It can select another configured identity before your Azure CLI identity, so ensure the selected identity has the roles listed above.

⚠ Use only the resource group and account this script creates. Vector indexing and search can’t be disabled on a container after enablement, and vector policy settings can’t be edited directly. Never point this exercise at a shared training or production account.

Set variables for the account name and its endpoint so later commands can use them.

 $accountName = "<your-account-name>"
 $cosmosEndpoint = "https://$accountName.documents.azure.com:443/"

Task 1: Confirm the embedding deployment

Confirm that the text-embedding-3-small deployment is ready before loading the catalog.

In the Allfiles/Labs/Shared terminal, load the saved settings.

 $foundry = Get-Content "./logs/foundry-$accountName.json" -Raw | ConvertFrom-Json
 $openAiName = $foundry.FoundryAccountName
 $openAiEndpoint = $foundry.OpenAiEndpoint
 $foundry | Format-List FoundryAccountName, OpenAiEndpoint, EmbeddingDeployment, EmbeddingDimensions

Check the deployed model’s state.

 az cognitiveservices account deployment list `
     --name $openAiName `
     --resource-group $resourceGroup `
     --query "[].{Deployment:name,Model:properties.model.name,Version:properties.model.version,State:properties.provisioningState}" `
     --output table

Confirm that text-embedding-3-small reports Succeeded. Use OpenAiEndpoint, not the project endpoint, in the application code that follows. The model’s default output length matches the container’s 1,536 dimensions.

📝 New role assignments can take several minutes to take effect.

Task 2: Load the catalog with searchable text and embeddings

Combine each product’s name and category into searchText, then generate an embedding from that text. These fields provide more useful search terms than the catalog’s generated descriptions. Store the text in /searchText and the vector in /embedding, which are the paths configured in the container’s indexes.

Create a project and open its folder in the integrated terminal.
```
 dotnet new console --name search-lab
 cd search-lab
```

Install the required packages.

 dotnet add package Microsoft.Azure.Cosmos
 dotnet add package Azure.Identity
 dotnet add package Newtonsoft.Json
 dotnet add package Azure.AI.OpenAI

Replace the contents of Program.cs with the following code. Replace the two placeholder values with your endpoints.

 using System.Security.Cryptography;
 using System.Text;
 using Azure.AI.OpenAI;
 using Azure.Identity;
 using Microsoft.Azure.Cosmos;
 using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
 using OpenAI.Embeddings;

 string cosmosEndpoint = "<cosmos-endpoint>";
 string openAiEndpoint = "<openai-endpoint>";
 string deployment = "text-embedding-3-small";
 string dataUrl = "https://raw.githubusercontent.com/AzureCosmosDB/CosmicWorks/main/data/database-v4/product";

 // Reuse a credential for both services; each SDK requests its own token audience.
 DefaultAzureCredential credential = new DefaultAzureCredential();

 EmbeddingClient embeddingClient = new AzureOpenAIClient(new Uri(openAiEndpoint), credential)
     .GetEmbeddingClient(deployment);

 Container container = new CosmosClient(cosmosEndpoint, credential)
     .GetContainer("cosmicworks", "productSearch");

 using HttpClient http = new HttpClient();
 JArray products = JArray.Parse(await http.GetStringAsync(dataUrl));

 Console.WriteLine($"Downloaded {products.Count} products.");

 // Batch embedding requests, then write the enriched products individually.
 // This isn't a Cosmos DB transactional batch.
 for (int start = 0; start < products.Count; start += 50)
 {
     List<JObject> batch = products.Skip(start).Take(50).Cast<JObject>().ToList();
     List<string> texts = batch.Select(BuildSearchText).ToList();

     OpenAIEmbeddingCollection embeddings = await embeddingClient.GenerateEmbeddingsAsync(texts);

     // Keep each vector with its source text and original product identity.
     for (int i = 0; i < batch.Count; i++)
     {
         batch[i]["searchText"] = texts[i];
         batch[i]["contentHash"] = ContentHash(texts[i]);
         batch[i]["embedding"] = JArray.FromObject(embeddings[i].ToFloats().ToArray());

         await container.UpsertItemAsync(
             batch[i], new PartitionKey(batch[i]["categoryId"]!.ToString()));
     }

     Console.WriteLine($"Loaded {start + batch.Count} of {products.Count}.");
 }

 // Both keyword search and embeddings use this same text representation.
 static string BuildSearchText(JObject product) =>
     $"{product["name"]} {product["categoryName"]}";

 // Detect changes to the embedding input without comparing entire vectors.
 static string ContentHash(string text) =>
     Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text)));

Run the application.

 dotnet run

 Downloaded 295 products.
 Loaded 50 of 295.
 Loaded 100 of 295.
 Loaded 150 of 295.
 Loaded 200 of 295.
 Loaded 250 of 295.
 Loaded 295 of 295.

Each product now has searchText, an embedding, and a contentHash. Task 5 uses the hash to detect changes to the embedding input.

Task 3: Search the catalog with BM25 ranking

Use full-text functions to match search terms and rank products by relevance.

In the Azure portal, open your Azure Cosmos DB account, select Data Explorer, and expand the cosmicworks database.

Right-click the productSearch container and select New SQL Query. Run the following query, which finds every item whose searchable text contains the term.

 SELECT c.name, c.categoryName, c.price
 FROM c
 WHERE FullTextContains(c.searchText, "helmet")

The catalog holds exactly three helmets, all in the same category and all at the same price.

 [
   { "name": "Sport-100 Helmet, Black", "categoryName": "Accessories, Helmets", "price": 34.99 },
   { "name": "Sport-100 Helmet, Blue",  "categoryName": "Accessories, Helmets", "price": 34.99 },
   { "name": "Sport-100 Helmet, Red",   "categoryName": "Accessories, Helmets", "price": 34.99 }
 ]

Note that the category reads Helmets while the search term is helmet. Stemming is why the plural matches the singular.

Run the following query, which requires both terms.
```
 SELECT VALUE COUNT(1)
 FROM c
 WHERE FullTextContainsAll(c.searchText, "mountain", "helmet")
```
```
 [
   0
 ]
```
No CosmicWorks product is both a mountain product and a helmet, so requiring both terms eliminates the whole catalog.
Change the function to FullTextContainsAny and run it again.
```
 SELECT VALUE COUNT(1)
 FROM c
 WHERE FullTextContainsAny(c.searchText, "mountain", "helmet")
```
This query counts products that match either term, including mountain products and the three helmets. Record the count.

Rank results by relevance instead of filtering them.

 SELECT TOP 10 c.name, c.categoryName
 FROM c
 ORDER BY RANK FullTextScore(c.searchText, "mountain", "frame")

Items that carry both terms sort above items that carry one.

Run the following query, which adds FullTextScore to the SELECT list.

 SELECT TOP 10 c.name, c.categoryName,
     FullTextScore(c.searchText, "mountain", "frame") AS score
 FROM c
 ORDER BY RANK FullTextScore(c.searchText, "mountain", "frame")

This query is expected to fail because FullTextScore can appear only in an ORDER BY RANK clause.

Remove FullTextScore from the SELECT list and keep it in ORDER BY RANK. Run the corrected query.
```
 SELECT TOP 10 c.name, c.categoryName
 FROM c
 ORDER BY RANK FullTextScore(c.searchText, "mountain", "frame")
```
The query returns products ranked by relevance without including the score in the results.
Compare the request charges for full-text search and substring matching. Run each of the following queries, and after each one open the Query Stats tab and record the Request Charge.
```
 SELECT VALUE COUNT(1) FROM c WHERE FullTextContains(c.searchText, "helmet")
```
```
 SELECT VALUE COUNT(1) FROM c WHERE CONTAINS(c.searchText, "Helmet")
```
Request charges can be similar for this small catalog. FullTextContains uses the full-text index to match tokens, while CONTAINS matches literal substrings, so the results can differ.

Task 4: Search the catalog by meaning

Search for products by meaning rather than exact keywords.

Replace the contents of Program.cs with the following code, keeping the same project and packages and replacing the two placeholder values with your endpoints.

 using Azure.AI.OpenAI;
 using Azure.Identity;
 using Microsoft.Azure.Cosmos;
 using OpenAI.Embeddings;

 string cosmosEndpoint = "<cosmos-endpoint>";
 string openAiEndpoint = "<openai-endpoint>";
 string deployment = "text-embedding-3-small";

 DefaultAzureCredential credential = new DefaultAzureCredential();

 EmbeddingClient embeddingClient = new AzureOpenAIClient(new Uri(openAiEndpoint), credential)
     .GetEmbeddingClient(deployment);

 Container container = new CosmosClient(cosmosEndpoint, credential)
     .GetContainer("cosmicworks", "productSearch");

 string searchText = "something to protect my head";

 // Use the same embedding deployment as ingestion so query and stored vectors
 // share a vector space and match the container's configured dimensions.
 OpenAIEmbedding embedding = await embeddingClient.GenerateEmbeddingAsync(searchText);
 float[] queryVector = embedding.ToFloats().ToArray();

 // Although the alias says similarityScore, the projected value is a distance.
 // ORDER BY returns the nearest vectors first; it isn't a confidence percentage.
 QueryDefinition query = new QueryDefinition(@"
     SELECT TOP 5 c.name, c.categoryName,
         VectorDistance(c.embedding, @queryVector) AS similarityScore
     FROM c
     ORDER BY VectorDistance(c.embedding, @queryVector)")
     .WithParameter("@queryVector", queryVector);

 Console.WriteLine($"Query: \"{searchText}\"");

 // Consume every page; RequestCharge below is the charge for that page.
 using FeedIterator<dynamic> feed = container.GetItemQueryIterator<dynamic>(query);

 while (feed.HasMoreResults)
 {
     FeedResponse<dynamic> page = await feed.ReadNextAsync();

     foreach (dynamic item in page)
     {
         Console.WriteLine($"  {item.similarityScore:F4}  {item.name}");
     }

     Console.WriteLine($"Request charge: {page.RequestCharge} RUs");
 }

Run the application.
```
 dotnet run
```

Record the five product names and distance scores. The phrase doesn’t contain helmet, but helmet products should appear among the closest matches. Lower distances indicate closer matches.

Add a metadata filter by replacing the SQL text inside the existing QueryDefinition with the following query. The new @categoryId parameter is bound in the next step.

 SELECT TOP 5 c.name, c.price,
     VectorDistance(c.embedding, @queryVector) AS similarityScore
 FROM c
 WHERE c.categoryId = @categoryId
 ORDER BY VectorDistance(c.embedding, @queryVector)

Bind @categoryId to the helmet category identifier. Replace the existing .WithParameter("@queryVector", queryVector); line with the following code, which keeps the query vector parameter and adds the category parameter.
```
     .WithParameter("@queryVector", queryVector)
     .WithParameter("@categoryId", "14A1AD5D-59EA-4B63-A189-67B077783B0E");
```
Run the application again with dotnet run and compare the request charge with the unfiltered query.

Filtering on categoryId, the partition key, limits the search to one logical partition. The request charge might not decrease on this small dataset.

📝 The diskANN index requires at least 1,000 vectors. With only 295 vectors, these queries use a full scan.

Task 5: Refresh an embedding from the change feed

Renaming a product doesn’t update its stored searchText or embedding. The full-text index updates when the application writes a new searchText value. In this task, you rename a product, then run a change feed consumer that finds the mismatch and repairs it.

In the portal Data Explorer, run the following query and record the identifier and category of the first result.
```
 SELECT c.id, c.categoryId, c.name, c.searchText
 FROM c
 WHERE c.name = "Sport-100 Helmet, Red"
```
In the Data Explorer tree, expand the productSearch container under cosmicworks and select Items.
In the filter field, enter the following filter, replacing <product-id> with the id you recorded from the query result. Select Apply.
```
 WHERE c.id = "<product-id>"
```
Select the matching item in the filtered list. In the JSON editor, change only its name value to Sport-100 Head Protector, Red, then select Update.
Confirm the mismatch. Run the following query in Data Explorer and compare name against searchText.
```
 SELECT c.id, c.categoryId, c.name, c.searchText
 FROM c
 WHERE c.name = "Sport-100 Head Protector, Red"
```
The name changed, but searchText still contains the old name. The stored embedding hasn’t been regenerated either.

Replace the contents of Program.cs with the following code, keeping the same project and packages and replacing the two placeholder values with your endpoints.

 using System.Net;
 using System.Security.Cryptography;
 using System.Text;
 using Azure.AI.OpenAI;
 using Azure.Identity;
 using Microsoft.Azure.Cosmos;
 using Newtonsoft.Json.Linq;
 using OpenAI.Embeddings;

 string cosmosEndpoint = "<cosmos-endpoint>";
 string openAiEndpoint = "<openai-endpoint>";
 string deployment = "text-embedding-3-small";

 DefaultAzureCredential credential = new DefaultAzureCredential();

 EmbeddingClient embeddingClient = new AzureOpenAIClient(new Uri(openAiEndpoint), credential)
     .GetEmbeddingClient(deployment);

 Container container = new CosmosClient(cosmosEndpoint, credential)
     .GetContainer("cosmicworks", "productSearch");

 int scanned = 0;
 int refreshed = 0;

 // This one-off latest-version scan starts at the beginning without
 // persisting a continuation token for a later run.
 FeedIterator<JObject> changeFeed = container.GetChangeFeedIterator<JObject>(
     ChangeFeedStartFrom.Beginning(), ChangeFeedMode.LatestVersion);

 while (changeFeed.HasMoreResults)
 {
     FeedResponse<JObject> page = await changeFeed.ReadNextAsync();

     // HTTP 304 means the reader is caught up for now, not permanently finished.
     if (page.StatusCode == HttpStatusCode.NotModified) { break; }

     foreach (JObject change in page)
     {
         scanned++;
         string text = BuildSearchText(change);

         // Skip unchanged embedding inputs, including this refresher's own writes.
         if (ContentHash(text) == change["contentHash"]?.ToString()) { continue; }

         OpenAIEmbedding embedding = await embeddingClient.GenerateEmbeddingAsync(text);

         // Refresh text, hash, and vector together so the search fields agree.
         change["searchText"] = text;
         change["contentHash"] = ContentHash(text);
         change["embedding"] = JArray.FromObject(embedding.ToFloats().ToArray());

         await container.UpsertItemAsync(
             change, new PartitionKey(change["categoryId"]!.ToString()));

         refreshed++;
         Console.WriteLine($"Refreshed: {change["name"]}");
     }
 }

 Console.WriteLine($"Scanned {scanned} changes, refreshed {refreshed}.");

 static string BuildSearchText(JObject product) =>
     $"{product["name"]} {product["categoryName"]}";

 static string ContentHash(string text) =>
     Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text)));

Run the application.
```
 dotnet run
```
Example output (the scanned count can vary):
```
 Refreshed: Sport-100 Head Protector, Red
 Scanned 295 changes, refreshed 1.
```
Latest-version mode returns the latest version of each changed item, not every intermediate write. The scan can include the consumer’s own update, so the scanned count can exceed 295. Only the renamed product should need a refresh.
Run the consumer a second time without changing anything.
```
 Scanned 295 changes, refreshed 0.
```
The stored hashes now match the product text, so no embeddings are regenerated. The hash check also prevents unnecessary embedding calls when the consumer reads its own updates.
Confirm the repair. In the portal Data Explorer, run the following query.
```
 SELECT c.id, c.categoryId, c.name, c.searchText
 FROM c
 WHERE c.name = "Sport-100 Head Protector, Red"
```
Check that searchText now contains the new name, Sport-100 Head Protector, Red.

Clean up resources

Delete the resource group only if you created it for this exercise and it contains no resources you need to keep. This removes the Cosmos DB account, Foundry resource, project, and model deployment:

az group delete --name $resourceGroup --yes --no-wait

If your lab environment provided the resource group, keep it. Delete only the Cosmos DB and Foundry resources you created for this exercise, after confirming that no other application uses them:

az cosmosdb delete --name $accountName --resource-group $resourceGroup --yes
az cognitiveservices account delete --name $openAiName --resource-group $resourceGroup