Enrich an AI search index with custom classes

You’ve built a search solution and now want to add Azure AI Services for language enrichments to your indexes.

In this exercise, you’ll create an Azure AI Search solution and enrich an index with the results from a Language Studio custom text classification project. You’ll create a function app to connect search and your classification model together.

Note To complete this exercise, you will need a Microsoft Azure subscription. If you don’t already have one, you can sign up for a free trial at https://azure.com/free.

Set up your development environment with Python, VS Code and VS Code Extensions

Install these tools to complete this exercise. You can still follow along with the steps without these tools.

1. Install VS Code
2. Install Azure Core Functions Tool
3. Install Azure Tools Extensions for VS Code
4. Install Python 3.8 for your operating system.
5. Install Python Extension for VS Code

Set up your Azure resources

To save you time, select this Azure ARM template to create resources you’ll need later in the exercise.

Deploy a pre-built ARM template

1. select this link to create your starting resources. You might need to copy and paste the direct link into your search bar.

   

2. In Resource group, select Create new, name it cog-search-language-exe.
3. In Region, select a supported region that is close to you.
4. The Resource Prefix needs to be globally unique, enter a random numeric and lower-case character prefix, for example acs18245.
5. In Location, select the same region you chose above.
6. Select Review + create.

7. Select Create.

   Note There’s an error shown, You will need to Agree to the terms of service below to create this resource successfully., by selecting Create you are agreeing to them.

8. Select Go to resource group to see all the resources that you’ve created.

   You’ll be setting up an Azure Cognitive Search index, creating an Azure function, and creating a Language Studio project to identify movie genres from their summaries.

Upload sample data to train language services

This exercise uses 210 text files that contain a plot summary for a movie. The text files name is the movie title. The folder also contains a movieLabels.json file that maps the genres of a movie to the file, for each file there’s a JSON entry like this:

{
    "location": "And_Justice_for_All.txt",
    "language": "en-us",
    "classifiers": [
        {
            "classifierName": "Mystery"
        },
        {
            "classifierName": "Drama"
        },
        {
            "classifierName": "Thriller"
        },
        {
            "classifierName": "Comedy"
        }
    ]
},

1. Navigate to Labfiles/04-enrich-custom-classes and extract the movies summary.zip folder containing all the files.

   Note You use these files to train a model in Language Studio, and will also index all the files in Azure AI Search.

2. In the Azure portal, select Resource groups, then select your resource group.
3. Select the storage account you created, for example acs18245str.

4. Select Configuration from the left pane, select the Enable option for the Allow Blob anonymous access setting and then select Save at the top of the page.

   

5. Select Containers from the left, then select + Container.
6. In the New container pane, in Name, enter language-studio-training-data.
7. In Anonymous access level, choose Container (anonymous read access for containers and blobs) and select Create.
8. Select the new container you just created, language-studio-training-data.
9. Select Upload at the top of the pane.
10. In the Upload blob pane, select Browse for files.
11. Navigate to where you extracted the sample files, select all the text (.txt) and json (.json) files.
12. Select Upload in the pane.
13. Close the Upload blob pane.

Create a language resource

1. In the breadcrumb link at the top of the page, select Home.
2. Select + Create a resource and search for Language service.
3. Select Create under Language Service.
4. Select the option that includes Custom text classification and Custom named entity recognition.
5. Select Continue to create your resource.
6. In Resource group, choose cog-search-language-exe.
7. In Region, select the region you used above.
8. In Name, enter learn-language-service-for-custom-text. This needs to be unique globally, so you might need to add a random numbers or characters at the end of it.
9. In Pricing tier, select S.
10. In New/Existing storage account, select Existing storage account.
11. In Storage account in current selected subscription and resource region, select the storage account you created, for example acs18245str.
12. Agree to the Responsible AI Notice terms, then select Review + create.
13. Select Create.
14. Wait for the resources to be deployed, then select Go to resource group.

15. Select learn-language-service-for-custom-text.

    

16. Scroll down on the Overview pane, and select Get started with Language Studio.
17. Sign in the language studio. If you are prompted to choose a Language resource select the resource you created earlier.

Create a custom text classification project in Language Studio

1. On the Language Studio home page, select Create new, then select Custom text classification.

   

2. Select Next.

   

3. Select Multi label classification, then select Next.

   

4. In Name, enter movie-genre-classifier.
5. In Text primary language, select English (US).
6. Select Yes, enable multi-lingual dataset.
7. In Description, enter A model that can identify a movie genre from the summary.

8. Select Next.

   

9. In Blob storage container, choose language-studio-training-data.
10. Select Yes, my documents are already labeled and I have a correctly formatted JSON labels file.
11. In Label documents, choose movieLabels.
12. Select Next.
13. Select Create project.

Train your custom text classification AI model

1. On the left, select Training jobs.

   

2. Select + Start a training job.

   

3. In Train a new modal, enter movie-genre-classifier.
4. Select Train.
5. Training the classifier model should take less than 10 minutes. Wait for the status to change to Training succeeded.

Deploy your custom text classification AI model

1. On the left, select Deploying a model.

   

2. Select Add a deployment.

   

3. In Create a new deployment name, enter test-release.
4. In Model, select movie-genre-classifier.
5. Select Deploy.

Leave this web page open for later in this exercise.

Create an Azure AI Search index

Create a search index that you can enrich with this model, you’ll index all the text files that contain the movie summaries you’ve already downloaded.

1. In the Azure portal, select Resource groups, select your resource group, then select the storage account you created, for example acs18245str.
2. Select Containers from the left, then select + Container.
3. In the New container pane, in Name, enter search-data.
4. In Anonymous access level, choose Container.
5. Select Create.
6. Select the new container you just created, search-data.
7. Select Upload at the top of the pane.
8. In the Upload blob pane, select Browse for files.
9. Navigate to where you downloaded the sample files, select ONLY the text (.txt) files.
10. Select Upload in the pane.
11. Close the Upload blob pane.

Import documents into Azure AI Search

1. On the left, select Resource groups, select your resource group, then select your search service.

2. Select Import data.

   

3. In Data Source, select Azure Blob Storage.
4. In Data source name, enter movie-summaries.
5. Select Choose an existing connection, select your storage account, then select the container you just created, search-data.
6. Select Add cognitive skills (optional).

7. Expand the Attach AI Services section, then select the Azure AI service you created earlier.

   

8. Expand the Add enrichments section.

   

9. Leave all the fields with their default values, then select Extract people names.
10. Select Extract key phrases.
11. Select Detect language.

12. Select Next: Customize target index.

    

13. Leave all the fields with their default values, for metadata_storage_name select Retrievable and Searchable.
14. Select Next: Create an indexer.
15. Select Submit.

The indexer will run and create an index of the 210 text files. You don’t need to wait for it to continue with the next steps.

Create a function app to enrich your search index

You’ll now create a Python function app that your cognitive search custom skillset will call. The function app will use your custom text classifier model to enrich your search index.

1. Download required files and extract the folder containing all the files.

2. Open Visual Studio Code, open the movie-genre-function folder you’ve just downloaded.

   

3. If you’ve installed all the required extensions, you’re prompted to optimize the project. Select Yes.
4. Select your Python interpreter, it should be version 3.8.
5. The workspace will be updated, if you’re asked to connect it to the workspace folder, select Yes.

6. Press F5 to debug the app.

   If the app is running you should see a localhost URL that you can use for local testing.

7. Stop debugging the app, press SHIFT + F5.

Deploy your local function app to Azure

1. In Visual Studio Code, press F1 to open the command palette.
2. In the command palette, search for and select Azure Functions: Create Function App in Azure....
3. Enter a globally unique name for your function app, for example acs13245str-function-app.
4. In Select a runtime stack, select Python 3.8.

5. Select the same location you used above.

6. In the left navigation, select the Azure extension.
7. Expand Resources, expand Function App under your subscription, then right-click on the function, for example acs13245-function-app.
8. Select Deploy to Function App. Wait for the app to be deployed.
9. Expand the app, right-click on Application Settings, select Download Remote Settings.

10. On the left, select Explorer, then select local.settings.json.

    The function app needs to be connected to your custom text classification model. Follow these steps to get the configuration settings.

11. In your browser, navigate to Language Studio, you should be on the Deploying a model page.

    

12. Select your model. Then select Get prediction URL.
13. Select the copy icon next to the Prediction URL.
14. In Visual Studio Code, at the bottom of local.settings.json, paste the prediction URL.

15. In Language Studio, on the left, select Project settings.

    

16. Select the copy icon next to the Primary key.
17. In Visual Studio Code, at the bottom of local.settings.json, paste the primary key.

18. Edit the settings to add these four lines at the bottom, copy the endpoint into the TA_ENDPOINT value.

     ,
     "TA_ENDPOINT": " [your endpoint] ",
     "TA_KEY": " [your key] ",
     "DEPLOYMENT": "test-release",
     "PROJECT_NAME": "movie-genre-classifier"
    

19. Copy the primary key into the TA_KEY value.

     {
       "IsEncrypted": false,
       "Values": {
         "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=...",
         "FUNCTIONS_EXTENSION_VERSION": "~4",
         "FUNCTIONS_WORKER_RUNTIME": "python",
         "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING": "DefaultEndpointsProtocol=https;AccountName=...",
         "WEBSITE_CONTENTSHARE": "acs...",
         "APPINSIGHTS_INSTRUMENTATIONKEY": "6846...",
         "TA_ENDPOINT": "https://learn-languages-service-for-custom-text.cognitiveservices.azure.com/language/analyze-text/jobs?api-version=2022-05-01",
         "TA_KEY": "7105e938ce1...",
         "DEPLOYMENT": "test-release",
         "PROJECT_NAME": "movie-genre-classifier"
       }
     }
    
    

    The settings should look like above, with the values of your project.

20. Press CTRL+S to save your local.settings.json changes.
21. In the left navigation, select the Azure extension.
22. Expand Resources, and under your subscription, expand Function App, then right-click on Application Settings, select Upload Local Settings.

Test your remote function app

There’s a sample query you can use to test that your function app and classifier model are working correctly.

1. On the left, select Explorer, expand the customtextcla folder, then select sample.dat.

   

2. Copy the contents of the file.

3. On the left, select the Azure extension.

   

4. Under the Function App, expand Functions, right-click on customtextcla, then select Execute Function now.

5. In Enter request body, paste the sample data you copied, then press Enter.

   The function app will respond with JSON results.

6. Expand the notification to see the whole results.

   The JSON response should look like this:

    {"values": 
        [
            {"recordId": "0", 
            "data": {"text": 
            [
                {"category": "Action", "confidenceScore": 0.99}, 
                {"category": "Comedy", "confidenceScore": 0.96}
            ]}}
        ]
    }
   

Add a field to your search index

You need a place to store the enrichment returned by your new function app. Follow these steps to add a new compound field to store the text classification and confidence score.

1. In the Azure portal, go to the resource group that contains your search service, then select the cognitive search service you created, for example acs18245-search-service.
2. On the Overview pane, select Indexes.
3. Select azurebob-index.
4. Select Edit JSON.

5. Add the new fields to the index, paste the JSON below the content field.

    {
      "name": "textclass",
      "type": "Collection(Edm.ComplexType)",
      "analyzer": null,
      "synonymMaps": [],
      "fields": [
        {
          "name": "category",
          "type": "Edm.String",
          "facetable": true,
          "filterable": true,
          "key": false,
          "retrievable": true,
          "searchable": true,
          "sortable": false,
          "analyzer": "standard.lucene",
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "synonymMaps": [],
          "fields": []
        },
        {
          "name": "confidenceScore",
          "type": "Edm.Double",
          "facetable": true,
          "filterable": true,
          "retrievable": true,
          "sortable": false,
          "analyzer": null,
          "indexAnalyzer": null,
          "searchAnalyzer": null,
          "synonymMaps": [],
          "fields": []
        }
      ]
    },
   

   Your index should now look like this.

   

6. Select Save.

Edit the custom skillset to call your function app

The cognitive search index needs a way to have these new fields populated. Edit the skillset you created earlier to call your function app.

1. At the top of the page, select the search service link, for example **acs18245-search-service

   Indexes**.

2. On the Overview pane, select Skillsets.

   

3. Select azureblob-skillset.

4. Add the custom skillset definition below, by pasting it as the first skillset.

    {
      "@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
      "name": "Genre Classification",
      "description": "Identify the genre of your movie from its summary",
      "context": "/document",
      "uri": "URI",
      "httpMethod": "POST",
      "timeout": "PT30S",
      "batchSize": 1,
      "degreeOfParallelism": 1,
      "inputs": [
        {
          "name": "lang",
          "source": "/document/language"
        },
        {
          "name": "text",
          "source": "/document/content"
        }
      ],
      "outputs": [
        {
          "name": "text",
          "targetName": "class"
        }
      ],
      "httpHeaders": {}
    },
   

You need to change the "uri": "URI" to point to your function app.

1. In Visual Studio Code, select the Azure extension.

   

2. Under Functions, right-click customtextcla, then select Copy Function Url.
3. On the Azure portal, replace the URI with the copied function URL.
4. Select Save.

Edit the field mappings in the indexer

You now have fields to store the enrichment, a skillset to call your function app, the last step is to tell the cognitive search where to put the enrichment.

1. At the top of the page, select the search service, for example, **acs18245-search-service

   Skillsets** link.

   

2. On the Overview pane, select Indexers.
3. Select azureblob-indexer.
4. Select Indexer Definition (JSON).

5. Add a new output field mapping, by pasting this field definition to the top of the output field section.

    {
      "sourceFieldName": "/document/class",
      "targetFieldName": "textclass"
    },
   

   The indexer JSON definition should now look like this:

   

6. Select Save.
7. Select Reset, then select Yes.

8. Select Run, then select Yes.

   Your Azure cognitive search service runs your updated indexer. The indexer uses the edited custom skillset. The skillset calls your function app with the document being indexed. The custom text classifier model uses the text in the document to try and identify the genre of the movie. The model returns a JSON document with genres and confidence levels. The indexer maps the JSON results to the fields in your index using the new output field mapping.

9. Select Execution history.

10. Check that the indexer has successfully run against the 210 documents.

    You might need to select Refresh to update the status of the indexer.

Test your enriched search index

1. At the top of the page, select the search service, for example **acs18245-search-service

   Indexers**.

2. On the Overview pane, select Indexes.

3. Select azurebob-index.

   

4. Select Search.
5. Explore the search results.

Each document in the index should have a new textclass field that can be searched. It contains a category field with the movies genres. It can be more than one. It also shows how confident the custom text classification model is about the identified genre.

Now that you’ve completed the exercise, delete all the resources you no longer need.

Clean-up

1. In the Azure portal, go to the Home page, and select Resource groups.
2. Select the resource groups you don’t need, then select Delete resource group.