Analyze images with Azure Content Understanding

Azure Content Understanding is a capability available in Microsoft Azure AI Foundry that uses generative AI to analyze and interpret different types of unstructured content, including documents, images, audio, and video. By applying AI models to this content, the service can generate structured outputs that follow a user-defined schema. These structured outputs make it easier to integrate extracted information into automation, analytics, and search workflows.

One common challenge organizations face is managing large collections of visual content. Images often contain valuable information, but that information can be difficult to search or organize without descriptive metadata. Azure Content Understanding can analyze images and generate structured descriptions that help classify and index visual content, making it easier to locate relevant images and integrate them into search systems.

In this exercise, you’ll explore how to create and use an image analyzer in the Microsoft Azure Portal. You’ll run the analyzer on sample images and review the generated descriptions that can be used as metadata for indexing and search. By the end of this lab, you’ll understand how AI-generated image descriptions can help make visual content more searchable and useful in data-driven applications.

This exercise will take approximately 30 minutes.

Note: Some of the technologies used in this exercise are in preview or in active development. You may experience some unexpected behavior, warnings, or errors. Video generation can take 1 to 5 minutes to complete depending on your settings.

Prerequisites

To complete this exercise, you need:

• An Azure subscription with permissions to create AI resources.
• Visual Studio Code installed on your local machine.
• Python 3.13 or later installed on your local machine.
• Azure CLI installed on your local machine.

Create a Microsoft Foundry project

Microsoft Foundry uses projects to organize models, resources, data, and other assets used to develop an AI solution.

1. In a web browser, open the Microsoft Foundry portal at https://ai.azure.com and sign in using your Azure credentials. Close any tips or quick start panes that are opened the first time you sign in, and if necessary use the Foundry logo at the top left to navigate to the home page.

2. If it is not already enabled, in the tool bar the top of the page, enable the New Foundry option. Then, if prompted, create a new project with a unique name; expanding the Advanced options area to specify the following settings for your project:
   • Foundry resource: Use the default name for your resource (usually {project_name}-resource)
   • Subscription: Your Azure subscription
   • Resource group: Create or select a resource group
   • Region: Select any available region
3. Select Create. Wait for your project to be created.

4. On the home page for your project, view the project details.

   Creating a Foundry project also creates a Foundry resource group in Azure that is linked to your project. This resource group will connect to the Azure Content Understanding service and any other AI services you choose to deploy for use in your Foundry project.

Deploy required models for Content Understanding

Now that you have a Foundry project, you can deploy the AI models needed for content understanding.

1. Navigate to the Content Understanding settings page at https://contentunderstanding.ai.azure.com/settings.

2. Select the Add resource button.

3. Select your subscription and the Foundry resources that match your Foundry project name.

4. Check the box for Enable auto-deployment for required models if no default deployment available.

5. Select Next, then select Save to deploy the required models.

   The deployment process can take several minutes. Once the models are deployed, the resource will appear under Connected Azure AI Foundry Resources. Note the name of the resource.

Try a pre-built image analyzer in the Content Understanding Studio

1. Select Home from the upper-right of the page to navigate to the homepage.

   The Get started with Content Understanding page will appear with options to analyze different types of content.

2. Select Explore all pre-built analyzers.

   A list of pre-built analyzers will appear. These analyzers are designed to extract structured information from different types of content, such as documents, images, audio, and video.

3. Select Modality from the filter options, and then select Image.

4. Select Try it on the Image search analyzer.

   Observe the sample image analysis results. The analyzer generates a summary of the image content, identifying objects and concepts in the image. The JSON result is also available. You can also try uploading your own images to see the generated descriptions.

Create an image analyzer application

Now that you’ve explored the playground, let’s build a Python application that programmatically analyzes images using the Content Understanding analyzers.

Get application files from GitHub

The initial application files you’ll need to develop the translation application are provided in a GitHub repo.

1. Open Visual Studio Code.

2. Open the command palette (Ctrl+Shift+P) and use the Git:clone command to clone the https://github.com/microsoftlearning/mslearn-ai-vision repo to a local folder (it doesn’t matter which one). Then open it.

   You may be prompted to confirm you trust the authors.

3. In Visual Studio Code, view the Extensions pane; and if it is not already installed, install the Python extension.

4. In the Command Palette, use the command python:select interpreter. Then select an existing environment if you have one, or create a new Venv environment based on your Python 3.1x installation.

   Tip: If you are prompted to install dependencies, you can install the ones in the requirements.txt file in the /labfiles/content-understanding/python folder; but it’s OK if you don’t - we’ll install them later!

   Tip: If you prefer to use the terminal, you can create your Venv environment with python -m venv labenv, then activate it with \labenv\Scripts\activate.

Prepare the application configuration

1. After the repo has been cloned, open the folder in VS Code (File > Open Folder), and navigate to the /labfiles/content-understanding/python folder.

2. In the VS Code Explorer pane, review the files in the folder:

   • .env - A configuration file for application settings.
   • image-app.py - The Python code file for the image analyzer application.
   • requirements.txt - A file listing the package dependencies.
   • images - A folder containing images for analysis.

3. In the Explorer pane, in the python folder, select the .env file to open it. Then update the endpoint value to match the endpoint for your Foundry resource.

   Important:Be sure to add the https://{YOUR-RESOURCE-NAME}.services.ai.azure.com Foundry resource endpoint, not the project endpoint or Azure OpenAI endpoint!

   Save the modified configuration file.

4. In the Explorer pane, right-click the python folder containing the application files, and select Open in integrated terminal (or open a terminal in the Terminal menu and navigate to the /labfiles/content-understanding/python folder.)

   Note: Opening the terminal in Visual Studio Code will automatically activate the Python environment. You may need to enable running scripts on your system.

5. Ensure that the terminal is open in the /labfiles/content-understanding/python* folder with the prefix (.venv) to indicate that the Python environment you created is active.

6. Install the required Python packages by running the following command:

    pip install -r requirements.txt
   

Write code to analyze images and generate descriptions

Tip: As you add code, be sure to maintain the correct indentation.

1. In VS Code, open the image-app.py file.

2. Find the comment Add references and add the following code for the necessary imports:

   # Add references
   import requests
   import time
   from azure.identity import DefaultAzureCredential
   

3. Find the comment Get an access token for the Azure Cognitive Services resource and add the following code:

   # Get an access token for the Azure Cognitive Services resource
   credential = DefaultAzureCredential()
   token = credential.get_token("https://cognitiveservices.azure.com/.default")
   analyzer_url = f"{endpoint}/contentunderstanding/analyzers/prebuilt-imageSearch:analyze?api-version=2025-11-01"
   headers = {
       "Authorization": f"Bearer {token.token}",
       "Content-Type": "application/json"
   }
   

4. Find the comment Send the image to the analyzer and get the initial response and add the following code:

   # Send the image to the analyzer and get the initial response
   payload = {
       "inputs": [
           {"url": image_url}
       ]
   }
   response = requests.post(analyzer_url, headers=headers, json=payload)
   print(f"Status Code: {response.status_code}")
   result = response.json()
   

5. Find the comment Check the status of the analysis and poll if it’s still running and add the following code:

   # Check the status of the analysis and poll if it's still running
   if result.get("status") in ("Running", "NotStarted"):
       request_id = result.get("id")
       results_url = f"{endpoint}/contentunderstanding/analyzerResults/{request_id}?api-version=2025-11-01"
          
       # Poll until complete
       while result.get("status") not in ("Succeeded", "Failed"):
           time.sleep(2)
           poll_response = requests.get(results_url, headers=headers)
           result = poll_response.json()
           print(f"Status: {result.get('status')}")
   

6. Save the file (Ctrl+S).

Sign into Azure and run the app

1. In the VS Code terminal, sign into Azure:

    az login
   

   You must sign into Azure to authenticate with your Azure OpenAI resource.

   Note: In most scenarios, just using az login will be sufficient. However, if you have subscriptions in multiple tenants, you may need to specify the tenant by using the –tenant parameter.

2. When prompted, follow the instructions to open the sign-in page in a new tab and enter the authentication code provided and your Azure credentials.

3. After you have signed in, run the application:

    python image-app.py
   

4. Observe the output, which should be similar to the following:

    ==================================================
    Image Analysis Summaries:
    ==================================================
   
    Image 1 Summary:
    The image shows a single apple placed on a light-colored fabric surface. The apple has a mix of red and greenish-yellow hues with some natural blemishes and a slightly irregular shape. The background is plain and out of focus, highlighting the apple as the main subject.
   
    Image 2 Summary:
    The image shows a single ripe banana placed on a white textured surface. The banana is mostly yellow with some brown spots and a greenish stem, indicating it is fresh and ready to eat.
   
    Image 3 Summary:
    The image is a 3D pie chart showing the distribution of hours in four categories. The largest segment is '60+ hours' at 37.8%, followed closely by '50-60 hours' at 36.6%. The '40-50 hours' category accounts for 18.9%, and the smallest segment is '1-39 hours' at 6.7%.
   

   Image analysis typically takes a few seconds. Notice that the descriptions include details about the objects in the images, including text, numbers, and other visual features. These detailed descriptions can help make the images more searchable and easier to organize in a content management system or search index.

Summary

In this exercise, you created a Foundry resource and deployed the necessary models for content understanding. You explored the pre-built image analyzer in the Content Understanding Studio, and then built a Python application that sends images to the analyzer and retrieves generated descriptions. Great work!

Clean up

When you finish exploring content understanding in Foundry, you should delete the resources you’ve created to avoid unnecessary Azure costs.

• Navigate to the Azure portal at https://portal.azure.com.
• In the Azure portal, on the Home page, select Resource groups.
• Select the resource group that you created for this exercise.
• At the top of the Overview page for your resource group, select Delete resource group.
• Enter the resource group name to confirm you want to delete it, and select Delete.