Search or filter specific area/coordinates with recognize text from PDF AI

[askingquestion9]

I am using the default action "Recognize text in an image or a PDF document" from AI Builder

Is there a way to add like a quadrant for each image/document?

Or is there a way to filter this output to only include coordinates in the lower left quadrant?

[
  {
    "@odata.type": "#Microsoft.Dynamics.CRM.expando",
    "page": 1,
    "lines@odata.type": "#Collection(Microsoft.Dynamics.CRM.crmbaseentity)",
    "lines": [
      {
        "@odata.type": "#Microsoft.Dynamics.CRM.expando",
        "text": "10000 FARM- ANIMALS MILK",
        "boundingBox": {
          "@odata.type": "#Microsoft.Dynamics.CRM.expando",
          "left": 0.020272931185041865,
          "top": 0.04817338060721935,
          "width": 0.2313408557779566,
          "height": 0.020837619041856763,
          "polygon": {
            "@odata.type": "#Microsoft.Dynamics.CRM.expando",
            "coordinates@odata.type": "#Collection(Microsoft.Dynamics.CRM.crmbaseentity)",
            "coordinates": [
              {
                "@odata.type": "#Microsoft.Dynamics.CRM.expando",
                "x": 0.020272931185041865,
                "y": 0.05078715860550104
              },
              {
                "@odata.type": "#Microsoft.Dynamics.CRM.expando",
                "x": 0.25161378696299846,
                "y": 0.04817338060721935
              },
              {
                "@odata.type": "#Microsoft.Dynamics.CRM.expando",
                "x": 0.25161378696299846,
                "y": 0.06770411064993527
              },
              {
                "@odata.type": "#Microsoft.Dynamics.CRM.expando",
                "x": 0.020272931185041865,
                "y": 0.06901099964907612
              }
            ]
          }
        }
      },

[takolota · Accepted solution]

@askingquestion9 

Yes. You will want to use 1 Filter array action with a more complex condition expression or 2 sequential Filter array actions with simpler condition expressions to filter to only the items below a certain Y coordinate on a scale of 0 to 1 and to only the items below a certain X coordinate on a 0 to 1 scale.

Each object/item there has 4 sets of x-y coordinates and they are ordered as shown below

 Where there is a bounding box around each piece of text where the 1st coordinates are the top left point of the box, the 2nd are the top right, 3rd are bottom right, & fourth are bottom left.

You can probably just base the coordinates filter on the 1st top left coordinates for each item.

Looks like for the From input of the 1st or only Filter you will need to reference the lines array property so like body(‘OCRaction…’)?[‘lines’]
Then for the condition expression you will need to reference the 1st coordinate set with something similar to

item()?[‘boundingBox’]?[‘coordinates’][0][‘x’]

And

item()?[‘boundingBox’]?[‘coordinates’][0][‘y’]

So you can set what coordinates the items returned should be greater than or less than.

[askingquestion9]

Thank you! I will try to implement that

Also thanks for the CSV to Dataset, I found it today and improved the speed of my flows so much!

I need to go over all your posts, they are a gold mine

[JStell]

Awesome! This is exactly what I was looking for!

I appreciate it! 👍

[askingquestion9]

@takolota  I am trying to filter the array after apply to each 5 using the lines output, but it doesnt even let me add the "item()?['boundingBox']?['coordinates'][0]['x']" in the expression

below is the flow without the filter array

I

[takolota]

@askingquestion9 

I don’t know what you’re doing, but that looks like an unnecessary Apply to Each loop on the Lines when you need to use a Filter array action. 

[askingquestion9]

@takolota I managed to get it  to work but it doesn't seem optimal

[JStell]

@takolota, would you be able to help with how to reference the specific lines? For instance, after you see "body":[ which leads down to "lines": [, how would I reference the specific call outs below this point in order to pull the desired information? Would each instance of open and closed brackets following this be considered an idividual line? Let me know if this doesn't make sense so I can try and re word it.

"body": [

        {

            "@odata.type": "#Microsoft.Dynamics.CRM.expando",

            "page": 1,

            "lines@odata.type": "#Collection(Microsoft.Dynamics.CRM.crmbaseentity)",

            "lines": [

[takolota]

@JStell Can you share more of the output? Like maybe paste it into a text file & attach it?

[JStell]

@takolota, I'll attach the output here. I removed the irrelevant information leaving on the text that I would like to extract. It is in the Format of Title then Text to be extracted. For example, the first one you will see is "text": "QC Package ID#" which would be the title of the field, followed by "text": "SW-001" which would be the extract value. In total, I have 8 fields I would like to extract.

[takolota]

@askingquestion9 & @JStell 

Here is a picture that I think may help better answer the original question of the thread asking for the lower-left quadrant

[takolota]

@JStell 
To extract specific fields you will either need
-All the fields to be findable/filterable with simple logic like if it starts with, ends with, or contains specific characters
-Regex logic

-Pass all the text for the given document to a GPT prompt asking it to extract the data fields like in this template: https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Extract-Data-From-PDFs-and-Images-With-G... 

[JStell]

I tried an example like this with multiple apply to each fields, but it failed after running for 3 minutes. However I din't put it in this location so maybe that would make a difference?

The trouble seems to be in defining the 3rd or lower right side coordinates. The "left" or "top" fields make it easier to reference the first set in many ways, but not the other three. Would it be possible to add some math functions to add the width of the box to first "left" coordinates to give a field a "less than" side? I can't seem to actually box in the values while avoiding the apply to each trouble.

I did use all four fields for my filter but it is still pulling alot of unnecessary information. Heres what that looks like, let me know if you see any thing that could be improved. I have been at it all day so my brain is not up to par :D.

@Or(and(greaterOrEquals(item()?['boundingBox']?['left'], 0.4023454545454545),lessOrEquals(item()?['boundingBox']?['top'], 0.14263529411764705),lessOrEquals(item()?['boundingBox']?['width'], 0.033009090909091),lessOrEquals(item()?['boundingBox']?['height'], 0.012058823529411)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.3215272727272727),lessOrEquals(item()?['boundingBox']?['top'], 0.18784705882352942),lessOrEquals(item()?['boundingBox']?['width'], 0.126127272727273),lessOrEquals(item()?['boundingBox']?['height'], 0.010141176470588)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.3221363636363636),lessOrEquals(item()?['boundingBox']?['top'], 0.20941176470588235),lessOrEquals(item()?['boundingBox']?['width'], 0.122336363636364),lessOrEquals(item()?['boundingBox']?['height'], 0.01015294117647)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.3934545454545455),lessOrEquals(item()?['boundingBox']?['top'], 0.09150588235294119),lessOrEquals(item()?['boundingBox']?['width'], 0.0505272727272727),lessOrEquals(item()?['boundingBox']?['height'], 0.012199999999999989)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.028445454545454548),lessOrEquals(item()?['boundingBox']?['top'], 0.2804941176470588),lessOrEquals(item()?['boundingBox']?['width'], 0.0493545454545455),lessOrEquals(item()?['boundingBox']?['height'], 0.575941176470589)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.8116363636363637),lessOrEquals(item()?['boundingBox']?['top'], 0.28054117647058824),lessOrEquals(item()?['boundingBox']?['width'], 0.143772727272727),lessOrEquals(item()?['boundingBox']?['height'], 0.575894117647059)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.08896363636363637),lessOrEquals(item()?['boundingBox']?['top'], 0.20941176470588235),lessOrEquals(item()?['boundingBox']?['width'], 0.152254545454545),lessOrEquals(item()?['boundingBox']?['height'], 0.012823529411765)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.4629818181818182),lessOrEquals(item()?['boundingBox']?['top'], 0.28036470588235296),lessOrEquals(item()?['boundingBox']?['width'], 0.044804545454545),lessOrEquals(item()?['boundingBox']?['height'], 0.570658823529413)),and(greaterOrEquals(item()?['boundingBox']?['left'], 0.0882),lessOrEquals(item()?['boundingBox']?['top'], 0.25372941176470587),lessOrEquals(item()?['boundingBox']?['width'], 0.149381818181818),lessOrEquals(item()?['boundingBox']?['height'], 0.010141176470589)))

[takolota]

@JStell 

Okay I thought the top & left may be easier for you to create a json reference for, but to avoid needing to add field box widths / heights to things you may want to change the references to one of the 4 x & y coordinates in the polygon section of the json mentioned at the start of this thread.

But really if you have multiple pieces of text boxed around the same area of the screen it may be hard to pinpoint to the exact text box you want. You may need Regex or the GPT piece to extract the specific field from the text.

[JStell]

@takolota,

It took some time, but I was able to make it work. It is pulling all 8 areas of interest without fail! If you'd like, ill send it over for you to exam and see what could be improved / simplified.

Thanks!