Explore computer vision

In this exercise, you’ll use a chat playground to interact with a generative AI solution that can analyze and interpret images. The goal of this exercise is to explore a common pattern for combining text and visual input in a prompt for a generative AI model.

Image of Anton.
Hi, I’m Anton.
I’ll be here to help you with hints and tips as you work through this lab. You can also find me in the Ask Anton app; which runs a generative AI model and agent in your browser, with a Basic mode option for older or lower-spec computers.

Ask Anton is not a supported Microsoft product or a component of Microsoft Learn or AI Skills Navigator. Just an experimental AI agent for you to explore as you learn about AI.

To complete this exercise, you need a modern browser on a computer with sufficient hardware resources to load and run the models used by Chat Playground the app. On older or low-spec computers, the app may run very slowly or experience errors.

Minimum spec
If your computer does not meet these requirements, the AI model may not run successfully. However, the app does support a failsafe Basic mode in which no model is used; which provides simpler, but faster responses.

  • 64-bit CPU, 8 cores
  • GPU (recommended)
  • 8+ GB system RAM (16 GB recommended)
  • Enough storage to cache ~300MB–800MB model assets
  • Latest Chrome / Edge / Firefox with WASM SIMD enabled/available (WebGPU support is recommended; a WASM-based fallback is provided)
  • Audio hardware (mic and speaker) required for speech functionality

This exercise should take approximately 15 minutes to complete.

Prepare for image-based chat

In this exercise you use a generative AI model in a chat playground to respond to prompts that include image data.

If your browser supports WebGPU, the chat playground uses the Microsoft Phi 3.5 Mini model running on your computer’s GPU. If not, the model runs on CPU - with reduced response-generation quality. If that fails, a basic mode with no model and responses retrieved from Wikipedia is activated. Performance may vary depending on the available memory in your computer and your network bandwidth to download the model. After opening the app, use the ? (About this app) icon in the chat area to find out more.

  1. In a web browser, open the Chat Playground at https://aka.ms/chat-playground.

  2. Wait for the model to download and initialize.

    Image of Anton.
    Tip: The first time you download a model, it may take a few minutes. Subsequent downloads will be faster.

    If the model is taking a long time to load, you can cancel and start in basic mode. You can switch between available models at any time in the Model list.

  3. While waiting for the model to download, open a new browser tab, and download images.zip from https://aka.ms/ai-images to your local computer.
  4. Extract the downloaded archive in a local folder to see the files it contains. These files are the images you will use AI to analyze.

  5. Return to the browser tab containing the chat playground and ensure a language model has downloaded. Then, in the configuration pane on the left pane, in the Vision section, enable Image analysis and wait for the computer vision model to be downloaded and initialized.

    Screenshot of the Image analysis option.

    Image of Anton.
    Tip: You can switch between light and dark themes using the ☼ / ☾ toggle at the top right.

    In the chat interface, an Upload image (📎) button is enabled.

  6. In the configuration pane update the Instructions to the following system prompt:

    You are an AI assistant that helps people identify vintage computer hardware.

  7. Click the Upload image button, and browse to select one of the images you extracted on your computer.

    A thumbnail of the image is added to the prompt input area.

  8. Enter a prompt like What can you tell me about this?. The image is included in the message.

    Screenshot of the chat app with an image in a prompt.

    The MobileNetV2 model is used to determine the likely subject of the image, and the results of that analysis is included in the prompt to the Phi language model. The result should be a reponse that uses the image information to answer the question.

  9. Submit prompts that include the other images, such as What is this? or Tell me about this.

    Image of Anton.
    Tip: Responses will vary in quality depending on the selected mode; but the image classificaton model should correctly identify the images.

  10. If you want to explore further, you can upload your own images and enter appropriate prompts. The combination of a small language model and a limited computer vision model means that the quality of the responses may be highly variable compared to a true multimodal large language model!

Summary

In this exercise, you explored the use of computer vision with a generative AI model in a chat playground.

The app used in this exercise is based on a simplified version of the chat playground in the Microsoft Foundry portal. Microsoft Foundry supports a range of multimodal models that can accept combined image and text input, enabling significantly more complex image interpretation than this simple example. Additionally, you can use the Azure Content Understanding tool to analyze images.

Anton avatar.
If you used the Ask Anton app during this lab, we’d love you to tell us about your experience with it!