I'm trying to perform OCR on an image with Power Automate

[tareenmj]

As the title describes, I'm trying to do OCR on an image which a user submits from Microsoft Forms. The flow is as follows: (i) User submits a form (which includes questions and prompts the user to upload an image), (ii) Gather responses and perform OCR on the image, and (iii) Send an email with the responses and the parsed image. To accomplish this, I setup the following flow: 
The Flow Process

With this, I am receiving the following error: 

Error message

I'm not sure what the error is since I double checked and I did submit a png image. Any ideas on what I can do differently? 

[JoeF-MSFT · Accepted solution]

Hi @tareenmj - thanks for the question and cool use case! I didn't know files can now be submitted in Microsoft Forms! 💚

The add a screenshot... response contains a number of metadata on the file. There are two steps you need to add to your flow to get the image content.

1. A Parse JSON action. Select the add a screenshot... field for Content. And add the following for Schema: 

{
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "name": {
                "type": "string"
            },
            "link": {
                "type": "string"
            },
            "id": {
                "type": "string"
            },
            "type": {},
            "size": {
                "type": "integer"
            },
            "referenceId": {
                "type": "string"
            },
            "driveId": {
                "type": "string"
            },
            "status": {
                "type": "integer"
            },
            "uploadSessionUrl": {}
        },
        "required": [
            "name",
            "link",
            "id",
            "type",
            "size",
            "referenceId",
            "driveId",
            "status",
            "uploadSessionUrl"
        ]
    }
}

2. Add a Get file content action from the OneDrive for Business connector to get the actual image that was uploaded on the Form. Use the id output coming from the 'Parse JSON' action for the File input.

3. You can then use the file content output from the previous OneDrive action in the AI Builder OCR action. 

Let us know if this works for you!

[tareenmj]

Such a thoughtful and well-written answer. I didn't realize I needed to perform actions upon the uploaded file but thank you so much for the help! It worked out perfectly. Thanks!

[JoeF-MSFT]

Glad to hear you got it working - congrats @tareenmj! 🙂

I'm curious to learn more about your scenario (and if you can share) - what type of images are your users submitting through the Microsoft Form you are doing OCR? Is it a photo of a document or similar? 

[tareenmj]

Thanks! Absolutely, the problem I am trying to solve is actually a bit more complicated than just doing OCR. I'm trying to read OCR of screenshots: 

The problem I'm trying to solve is a bit more complicated than just OCR. I need to store the values in the screenshot as variables. So for example, the TAC of 3107 into a separate variable. Also, since the screenshots can vary quite a bit, I can't just build an AI function to extract certain pieces of the screenshot. Some screenshots will have more information than others. 

[JoeF-MSFT]

This is really interesting. Thanks for sharing @tareenmj!

Do the screenshots vary jut in the number of rows? If so, you could look at building an AI Builder Form Processing model and extract the table: https://docs.microsoft.com/en-us/ai-builder/create-form-processing-model#tag-tables

You could also look at combining OCR with AI Builder Entity Extraction. You can train an Entity Extraction model to extract certain elements from a body of text: Create an entity extraction custom AI model - AI Builder | Microsoft Docs

[tareenmj]

Not a problem, I'm glad you found it interesting. I was thinking of using an AI Builder Entity Extraction but thing is since the screenshots can vary, I'm not sure if this is the right step. The screenshots can vary in not only number of rows but also the title (for example screenshot A might contain value1 = 35 while screenshot B will contain value1-trio = 26C). I was thinking that I could parse the string obtained from OCR. In the case of the previously mentioned example, find the substring position at "value1=" and then adding a certain number to be able to get the value 35. Similarly, I would do another substring search for "value1-trio =" and then offset to get the value of "26C". Do you think this would be a better approach or could I get this done using the custom extraction?