Analyze forms with custom Azure AI Document Intelligence models

Suppose a company currently requires employees to manually purchase order sheets and enter the data into a database. They would like you to utilize AI services to improve the data entry process. You decide to build a machine learning model that will read the form and produce structured data that can be used to automatically update a database.

Azure AI Document Intelligence is an Azure AI service that enables users to build automated data processing software. This software can extract text, key/value pairs, and tables from form documents using optical character recognition (OCR). Azure AI Document Intelligence has pre-built models for recognizing invoices, receipts, and business cards. The service also provides the capability to train custom models. In this exercise, we will focus on building custom models.

Create a Azure AI Document Intelligence resource

To use the Azure AI Document Intelligence service, you need a Azure AI Document Intelligence or Azure AI Services resource in your Azure subscription. You’ll use the Azure portal to create a resource.

In a browser tab, open the Azure portal at https://portal.azure.com, signing in with the Microsoft account associated with your Azure subscription.
On the Azure portal home page, navigate to the top search box and type Document Intelligence and then press Enter.
On the Document Intelligence page, select Create.
On the Create Document Intelligence page, create a new resource with the following settings:
- Subscription: Your Azure subscription.
- Resource group: Create or select a resource group
- Region: Any available region
- Name: A valid name for your Document Intelligence resource
- Pricing tier: Free F0 (if you don’t have a Free tier available, select Standard S0).
When the deployment is complete, select Go to resource to view the resource’s Overview page.

Prepare to develop an app in Cloud Shell

You’ll develop your text translation app using Cloud Shell. The code files for your app have been provided in a GitHub repo.

In the Azure Portal, use the [>_] button to the right of the search bar at the top of the page to create a new Cloud Shell in the Azure portal, selecting a PowerShell environment. The cloud shell provides a command line interface in a pane at the bottom of the Azure portal.

Note: If you have previously created a cloud shell that uses a Bash environment, switch it to PowerShell.
Size the cloud shell pane so you can see both the command line console and the Azure portal. You’ll need to use the the split bar to switch as you switch between the two panes.
In the cloud shell toolbar, in the Settings menu, select Go to Classic version (this is required to use the code editor).

Ensure you've switched to the classic version of the cloud shell before continuing.
In the PowerShell pane, enter the following commands to clone the GitHub repo for this exercise:
```
 rm -r mslearn-ai-info -f
 git clone https://github.com/microsoftlearning/mslearn-ai-information-extraction mslearn-ai-info
```
Tip: As you paste commands into the cloudshell, the ouput may take up a large amount of the screen buffer. You can clear the screen by entering the cls command to make it easier to focus on each task.
After the repo has been cloned, navigate to the folder containing the application code files:
```
cd mslearn-ai-info/Labfiles/custom-doc-intelligence
```

Gather documents for training

You’ll use the sample forms such as this one to train a test a model:

An image of an invoice used in this project.

In the command line, run ls ./sample-forms to list the content in the sample-forms folder. Notice there are files ending in .json and .jpg in the folder.

You will use the .jpg files to train your model.

The .json files have been generated for you and contain label information. The files will be uploaded into your blob storage container alongside the forms.
In the Azure portal and navigate to your resource’s Overview page if you’re not already there. Under the Essentials section, note the Resource group, Subscription ID, and Location. You will need these values in subsequent steps.
Run the command code setup.sh to open setup.sh in a code editor. You will use this script to run the Azure command line interface (CLI) commands required to create the other Azure resources you need.
In the setup.sh script, review the commands. The program will:
- Create a storage account in your Azure resource group
- Upload files from your local sampleforms folder to a container called sampleforms in the storage account
- Print a Shared Access Signature URI
Modify the subscription_id, resource_group, and location variable declarations with the appropriate values for the subscription, resource group, and location name where you deployed the Document Intelligence resource.

If the expiry_date variable is in the past, update it to a future date. This variable is used when generating the Shared Access Signature (SAS) URI. In practice, you will want to set an appropriate expiry date for your SAS. You can learn more about SAS here.
After you’ve replaced the placeholders, within the code editor, use the CTRL+S command or Right-click > Save to save your changes and then use the CTRL+Q command or Right-click > Quit to close the code editor while keeping the cloud shell command line open.
Enter the following commands to make the script executable and to run it:
```
chmod +x ./setup.sh
./setup.sh
```
When the script completes, review the displayed output.
In the Azure portal, refresh your resource group and verify that it contains the Azure Storage account just created. Open the storage account and in the pane on the left, select Storage browser. Then in Storage Browser, expand Blob containers and select the sampleforms container to verify that the files have been uploaded from your local custom-doc-intelligence/sample-forms folder.

Train the model using Document Intelligence Studio

Now you will train the model using the files uploaded to the storage account.

Open a new browser tab, and navigate to the Document Intelligence Studio at https://documentintelligence.ai.azure.com/studio.
Scroll down to the Custom models section and select the Custom extraction model tile.
If prompted, sign in with your Azure credentials.
If you are asked which Azure AI Document Intelligence resource to use, select the subscription and resource name you used when you created the Azure AI Document Intelligence resource.
Under My Projects, Create a new project with the following configuration:
- Enter project details:
  - Project name: A valid name for your project
- Configure service resource:
  - Subscription: Your Azure subscription
  - Resource group: The resource group where you deployed your Document Intelligence resource
  - Document intelligence resource Your Document Intelligence resource (select the Set as default option and use the default API version)
- Connect training data source:
  - Subscription: Your Azure subscription
  - Resource group: The resource group where you deployed your Document Intelligence resource
  - Storage account: The storage account that was created by the setup script (select the Set as default option, select the sampleforms blob container, and leave the folder path blank)
When your project is created, on the top right of the page, select Train to train your model. Use the following configurations:
- Model ID: A valid name for your model (you’ll need the model ID name in the next step)
- Build Mode: Template.
Select Go to Models.
Training can take some time. Wait until the status is succeeded.

Test your custom Document Intelligence model

Return to the browser tab containing the Azure Portal and cloud shell. In the command line, run the following command to change to the folder for your preferred language:

Python
```
 cd Python
```
C#
```
 cd C-Sharp
```

Install the Document Intelligence package by running the appropriate command for your language preference:

Python:

 python -m venv labenv
./labenv/bin/Activate.ps1
pip install -r requirements.txt azure-ai-formrecognizer==3.3.3

C#:

 dotnet add package Azure.AI.FormRecognizer --version 4.1.0

Enter the following command to edit the configuration file that has been provided:

Python
```
code .env
```
C#
```
code appsettings.json
```
In the pane containing the Azure portal, on the Overview page for your Document Intelligence resource, select Click here to manage keys to see the endpoint and keys for your resource. Then edit the configuration file with the following values:
- Your Document Intelligence endpoint
- Your Document Intelligence key
- The Model ID you specified when training your model
After you’ve replaced the placeholders, within the code editor, use the CTRL+S command to save your changes and then use the CTRL+Q command to close the code editor while keeping the cloud shell command line open.
Enter the following command to edit the app file that has been provided:

Python
```
code document-analysis.py
```
C#
```
code Program.cs
```
The file is opened in a code editor.
Open the code file for your client application (code Program.cs for C#, code test-model.py for Python) and review the code it contains, particularly that the image in the URL refers to the file in this GitHub repo on the web. Close the file without making any changes.
In the command line, and enter the following command to run the program:

Python
```
 python test-model.py
```
C#
```
 dotnet run
```
View the output and observe how the output for the model provides field names like Merchant and CompanyPhoneNumber.

Clean up

If you’re done with your Azure resource, remember to delete the resource in the Azure portal to avoid further charges.

More information

For more information about the Document Intelligence service, see the Document Intelligence documentation.