06-15-2023 08:01 AM - last edited 07-12-2024 09:59 AM
Extract Data From PDFs & Images With GPT
This template uses AI Builder's OCR for PDFs & Images to extract the text present in a file, replicates the file in a text (txt) format, then passes it off to a GPT prompt action for things like data extraction.
Seems to have a 85% or greater reliability for returning requested data fields from most PDFs. It's likely good enough to do more direct data entry on some use-cases with well formatted, clean PDFs, and in many other cases it is good at doing a 1st pass on a file & providing a default / pre-fill value for fields before a person then checks & completes something with the data.
It does not require training on different formats, styles, wording, etc. It works on multiple pages at once. And you can always adjust the prompt to extract the different data you want on different documents & adjust how you want the data to be represented in the output.
It also...
-Runs in less than a minute, usually 10-35 seconds, so it can respond in time for a Power Apps call.
-Handles 10-20 document pages at a time given the recent Create text with GPT update to a 16k model.
-Does not use additional 3rd party services, maintaining better data privacy.
The AI Builder Recognize text action returns a JSON array of each piece of text found in the PDF or image.
The Convert to txt loop goes through each vertical line in the PDF or image & creates a line of text to approximately match both the text & spacing between text for that line.
Each vertical line of text is then combined into a single block of text like a big txt file in the final Compose action, before it is then passed to GPT through the AI Builder Create text action.
Example
Demonstration Invoice Example...
The AI Builder action uses optical character recognition (OCR) on this invoice PDF to return each piece of text & its associated x, y coordinates.
Then the Convert to txt loop produces this output shown in the final Compose...
And if we copy that output over to a text (txt) notebook, then this is what it looks like...
That is then fed into this GPT action prompt...
Which produced this output...
{
"Invoice Date": "2022-09-20",
"Invoice Number": "8304933707",
"Purchase Order (PO) Number": "PO10022556-NIMR",
"Incoterms": "DAP",
"Delivery Or Ship To Address": "Dr The Mission Director, [REDACTED]",
"Consignee Address": "CHEMONICS INTERNATIONAL INC, ATT: ACC PAYABLE, GLOBAL HEALTH SUPPLY CHAIN (PSM), 1275 New Jersey Ave SE, Suite 200, WASHINGTON, DC, 20003 USA 20006, UNITED STATES OF AMERICA",
"Mode Of Shipment": "N/A",
"Product Lines": [
{
"Product Name": "KIT COBAS 58/68/8800 LYS, REAGENT IVD",
"Product Quantity": "49",
"Product Unit Price": "213.00",
"Product Line Total or Amount": "10,437.00",
"Manufacturer": "[REDACTED]"
},
{
"Product Name": "KIT COBAS 58/68/8800 MGP, IVD",
"Product Quantity": "165",
"Product Unit Price": "50.00",
"Product Line Total or Amount": "8,250.00",
"Manufacturer": "[REDACTED]"
},
{
"Product Name": "KIT COBAS 6800/8800 HIV 96T, IVD",
"Product Quantity": "5",
"Product Unit Price": "838.95",
"Product Line Total or Amount": "4,194.75",
"Manufacturer": "[REDACTED]"
},
{
"Product Name": "KIT COBAS 6800/8800 HIV 96T, IVD",
"Product Quantity": "313",
"Product Unit Price": "838.95",
"Product Line Total or Amount": "262,591.35",
"Manufacturer": "[REDACTED]"
},
{
"Product Name": "KIT COBAS 6800/8800 HIV 96T, IVD",
"Product Quantity": "65",
"Product Unit Price": "838.95",
"Product Line Total or Amount": "54,531.75",
"Manufacturer": "[REDACTED]"
},
{
"Product Name": "KIT COBAS HBV/HCV/HIV-1, CONTROL CE-IVD",
"Product Quantity": "72",
"Product Unit Price": "290.00",
"Product Line Total or Amount": "20,880.00",
"Manufacturer": "[REDACTED]"
}
],
"Invoice Total": "360,884.85",
"Banking Details": "[REDACTED]"
}
And remember you can always adjust the prompt to extract the different data you want on different documents & adjust how you want the data to be represented in the output. You can also often improve the output with more data specifications like "A PO number is always 2 letters followed by 8 digits. Only return those 2 letters & 8 digits."
Also if you are working with some Word/.docx files, there are built in OneDrive actions to convert them to .pdf files. So you should be able to process PDF, Image, and/or Word documents on the same type of set-up.
Also if you need something that can handle much larger files with a better page text filter/search set-up & larger GPT context window, check out this Query Large PDFs With GPT RAG template.
Remember, you may need AI Builder credits for the OCR & GPT actions in the flow to work. Each Power Automate premium licenses already come with 5000 credits that can be assigned to your environment. Depending on your license & organization, you may already have a few credits assigned to the environment.
If you are new, you can get a trial license to test things out: https://learn.microsoft.com/en-us/ai-builder/administer-licensing
Lastly, Microsoft recently started requiring approval actions after every GPT action. If you want to get around this requirement, see this post on setting the approval step to automatically succeed & move to the next action.
Version 1.7 simplifies some expressions. Download this version if you are just trying to initially understand the programming of the flow, & don't care as much about speed or efficiency.
Version 1.8 adds a PageNumbers compose action that allows one to input specific pages of a PDF or image packet to pass on to the text conversion & GPT prompt. This could be useful for scenarios where the relevant data is always on the 1st couple of pages or for scenarios where one must filter to only the relevant pages/images because the full packet of PDF page data or image data would exceed the GPT prompt token / character limit.
Version 2 redesigns the Convert to txt section of the flow to use several clever Select actions & expressions to avoid an additional level of Apply to each looping. So for an example 3 page document with 50 lines per page, instead of taking 15-20 seconds and 156 action calls, it takes 1 second and 21 action calls to create the text replica document.
This makes the entire flow 2X faster (15 seconds vs. 30 seconds) and 7X more efficient for daily action limits.
This makes some use-cases like real-time processing on a Power Apps document upload or processing of larger batches of documents each day much more viable.
Version 2.5 More changes to the Convert to txt component to create a little more accurate text replicas and a change to the placeholder prompt to make the message a little more concise & more accurate. Also moved the spaces & line-break into a single Compose called StaticVariables & changed the variable name to the now more accurate EachPage.
The Convert to txt piece now calculates the minimum X coordinate so it can subtract that number from all X coordinates & thus remove additional spaces on the left margin, helping to reduce the characters fed to the GPT prompt.
The Convert to txt piece also now has a ZoomX parameter in the StaticPageVariables action which sets the spaces multiple, or the number of spaces, per coordinate point. So for example, 200=More Accurate Text Alignment, but 100=Less GPT Tokens. So there may be some trade-offs here. (The recognize text bounding box coordinates around longer pieces of text seem to be dis-proportionatly larger than on smaller pieces of text & mess up the text alignment for rows/lines with multiple boxes / text entries.)
In addition, the Convert to txt piece will now include line-breaks for blank Y coordinate rows/lines to more accurately replicate the vertical spacing of pieces of text. I figured since each line should be just a line-break character, it shouldn't add much to the character / token count for the GPT prompt.
So overall 2.5 adds some better options for increased extraction accuracy or for decreased characters/tokens per page & thus for slightly larger file capacity.
Version 2.7 Another adjustment to the conversion from OCR coordinates to the text (txt) replica.
It now calculates the X coordinates of a piece of text from the mid-point between X coordinates 0 & 1. So along with the Y coordinates that were already being calculated from the mid-point between Y coordinates 0 & 3, this now registers the position of each piece of text from the center point of each coordinates box.
I also set it to start using an estimate of the length of text characters instead of the length of the overall coordinates box to calculate the whitespace / number of spaces between each piece of text.
Overall this makes this set-up even more accurate, improving text alignment, improving performance on more tilted pages, & adjusting the spacing/alignment for different font / text sizes on the same line.
Version 2.9 Adjustment For New MS Approval Requirement & Adjust Retry Policy
I added in the automatic approval step to get around the new MS approval action requirement. I also set the retry policy on the GPT action to retry every 5 seconds up to 7 times so it will fail less if wrongful 429 too many request errors occur.
If the standard import of the flow-only packages below do not work for you, you can also try importing the flows through a Power Apps solution package here: Re: Extract Data From PDFs and Images With GPT - Power Platform Community (microsoft.com)
Microsoft is deprecating the original Create text with GPT action this template relies on.
Users may need to use the new “Create text with GPT using a prompt” action & create a custom prompt on that action instead.
https://learn.microsoft.com/en-us/ai-builder/use-a-custom-prompt-in-flow
See this post for an example set-up: https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Extract-Data-From-PDFs-and-Images-With-G...
The ExtractPDFImageDataWithGPT_1_0_0_x Power Apps solution package contains a version of the flow where this is outlined.
Thanks for any feedback,
Please subscribe to my YouTube channel (https://youtube.com/@tylerkolota?si=uEGKko1U8D29CJ86).
And reach out on LinkedIn (https://www.linkedin.com/in/kolota/) if you want to hire me to consult or build more custom Microsoft solutions for you.
watch?v=mcQr-JsGj6Q
Thank you for the suggestion! The file has about 9 pages, but I need data only from 4 pages, so I limited the pages in the PageNumbers step to 4 pages and it appears to have worked!
I have a few questions about scaling up, is there a way to make the flow run when the PDF is uploaded to OneDrive instead of manually uploading the PDF file as an input in the Get_file_metadata step? In an ideal scenario, the end user won't need to interact with the flow at all, just upload the PDF's to OneDrive and the flow will get triggered and run, instead of interacting with the flow and upload the PDF to the Get_metadata_step..
The other question is whether the flow can be edited so it can process multiple PDF's in one run? As far as I can tell, the flow can process only 1 pdf in each run
You can delete the manual trigger & use a OneDrive trigger, then delete the other OneDrive action and in the OneDrive Get content action use the file Id from the OneDrive trigger action.
And yes, the flow only does one pdf at a time, but it can be triggered multiple times for multiple different PDFs per the changes I suggested above. Do you have a scenario where triggering & running for each file uploaded to OneDrive would not be sufficient?
I am guessing that the One Drive trigger should be "when a file is created," and it looks likeI have to choose a path for the files, which could be the attachments directory, is this the correct approach?
For the other One Drive actions, I see there are 2 actions in the original flow, one is Get file metadata, and the other is Get file content. According to your suggestion, the Get file metadata action is the one that needs to be deleted, is this correct?
a particular scenario is one specific use case we have, we want our solution/ flow to process tens of PDF's on daily basis, so I think it could be time consuming for the end user to keep run the flow multiple times individually for every single pdf file
Yes
Yes
And if a user or another automation uploads 10 pdfs to the folder you specify in the trigger of the flow, then the flow will run for each of them. Even if all 10 are uploaded in the same action / file directory selection. And I hope you don't mean people will be manually using the JSON at the end of the flow rather than automatically inserting/using it somewhere like a SharePoint list, Excle file, SQL database, etc.
Hi
I am getting below error
"{"operationStatus":"Error","error":{"type":"Error","code":"InvalidPredictionInput","message":"Parameters JSON string could not be properly deserialized","properties":{"BackendErrorCode":"InvalidInferenceInput","DependencyHttpStatusCode":"400"},"innerErrors":[{"scope":"Generic","target":null,"code":"InvalidModelParameters","type":"Error","properties":{"MlIssueCode":"InvalidModelParameters"}}]},"predictionId":null}"
Could you please assist me here?
@RameshNimmala
What is that from? I don't have a "Predict" action in my template for AI Builder, only "Recognize text in an image or a PDF document" and "Create text with GPT"
Could you please help if I am missing something or using incorrect action?
Where are you located?
Create text with GPT is available in the US, but may not be available in your country yet.
I am from India, so this is not available here? If not, what could be the alternative to use? Please ssuggest