Analyze images in Azure AI Foundry portal

Azure AI Vision includes numerous capabilities for understanding image content and context and extracting information from images. In this exercise, you will use Azure AI Vision in Azure AI Foundry portal, Microsoft’s platform for creating intelligent applications, to analyze images using the built-in try-it-out experiences.

Suppose the fictitious retailer Northwind Traders has decided to implement a “smart store”, in which AI services monitor the store to identify customers requiring assistance, and direct employees to help them. By using Azure AI Vision, images taken by cameras throughout the store can be analyzed to provide meaningful descriptions of what they depict.

Create a project in Azure AI Foundry portal

  1. In a web browser, open the Azure AI Foundry portal at https://ai.azure.com and sign in using your Azure credentials. Close any tips or quick start panes that are opened the first time you sign in.

  2. In the browser, navigate to https://ai.azure.com/managementCenter/allResources and select Create. Then choose the option to create a new AI hub resource.

  3. In the Create a project wizard, enter a valid name for your project, and if an existing hub is suggested, select the option to create a new one.

  4. Expand Advanced options to specify the following settings for your project:
    • Subscription: Your Azure subscription
    • Resource group: Create or select a resource group
    • Region: Select one of the following locations:
      • East US
      • France Central
      • Korea Central
      • West Europe
      • West US

    Wait for your project and hub to be created.

  5. When the project is created, you will be taken to an Overview page of the project details. Select AI services on the left-hand menu.

  6. On the AI Services page, select the Vision + Document tile to try out Azure AI Vision and Document capabilities.

    Screenshot of the Vision + Document tile in Azure AI Foundry.

Generate captions for an image

Let’s use the image captioning functionality of Azure AI Vision to analyze images taken by a camera in the Northwind Traders store. Image captions are available through the Caption and Dense Captions features.

  1. On the Vision + Document page, scroll down and select Image under View all other vision capabilities. Then select the Image captioning tile.

    Screenshot of the image captioning tile in the image section of the Vision and Document page.

  2. On the Add captions to images page, select the Azure AI services resource you created.

  3. On the Add captions to images page, review the resource you are connected to which is listed under the Try It Out subheading. You should not have to make changes. (Note: if you did not customize a valid resource location earlier during resource creation, you may be asked to create a new Azure AI services resource that is in a valid region. You will need to create the new resource to continue with the lab.)

  4. Select https://aka.ms/mslearn-images-for-analysis to download image-analysis.zip. Open the folder on your computer and locate the file named store-camera-1.jpg; which contains the following image:

    An image of a parent using a cellphone camera to take a picture of a child in in a store

  5. Upload the store-camera-1.jpg image by dragging it to the Drag and drop files here box, or by browsing to it on your file system.

  6. Observe the generated caption text, visible in the Detected attributes panel to the right of the image.

    The Caption functionality provides a single, human-readable English sentence describing the image’s content.

  7. Next, use the same image to perform Dense captioning. Return to the Vision + Document page by selecting the back arrow at the top of the page. On the Vision + Document page, select the Image tab, then select the Dense captioning tile.

    The Dense Captions feature differs from the Caption capability in that it provides multiple human-readable captions for an image, one describing the image’s content and others, each covering the essential objects detected in the picture. Each detected object includes a bounding box, which defines the pixel coordinates within the image associated with the object.

  8. Hover over one of the captions in the Detected attributes list and observe what happens within the image.

    The image and its captions are displayed.

    Move your mouse cursor over the other captions in the list, and notice how the bounding box shifts in the image to highlight the portion of the image used to generate the caption.

Tagging images

The next feature you will try is the Extract Tags functionality. Extract tags is based on thousands of recognizable objects, including living beings, scenery, and actions.

  1. Return to the Vision + Document page of Azure AI Foundry, then select the Image tab, and select the Common tag extraction tile.

  2. Open the folder containing the images you downloaded and locate the file named store-image-2.jpg, which looks like this:

    An image of person with a shopping basket in a supermarket

  3. Upload the store-camera-2.jpg file.

  4. Review the list of tags extracted from the image and the confidence score for each in the detected attributes panel. Here the confidence score is the likelihood that the text for the detected attribute describes what is actually in the image. Notice in the list of tags that it includes not only objects, but actions, such as shopping, selling, and standing.

    A screenshot of the detect attributes panel in the Vision Studio with text and confidence scores displayed next to the original image.

Object detection

In this task, you use the Object detection feature of Image Analysis. Object detection detects and extracts bounding boxes based on thousands of recognizable objects and living beings.

  1. Return to the Vision + Document page of Azure AI Foundry, then select the Image tab, and select the Common object detection tile.

  2. Open the folder containing the images you downloaded and locate the file named store-camera-3.jpg, which looks like this:

    An image of person with a shopping cart

  3. Upload the store-camera-3.jpg file.

  4. In the Detected attributes box, observe the list of detected objects and their confidence scores.

  5. Hover your mouse cursor over the objects in the Detected attributes list to highlight the object’s bounding box in the image.

  6. Move the Threshold value slider until a value of 70 is displayed to the right of the slider. Observe what happens to the objects in the list. The threshold slider specifies that only objects identified with a confidence score or probability greater than the threshold should be displayed.

Clean up

If you don’t intend to do more exercises, delete any resources that you no longer need. This avoids accruing any unnecessary costs.

  1. Open the Azure portal and select the resource group that contains the resources you created.
  2. Select the resource and select Delete and then Yes to confirm. The resource is then deleted.

Learn more

To learn more about what you can do with this service, see the Azure AI Vision page.