Entire process from OCR to Process Extracted Data to Verify Processed Data

[rswayne]

Hi. I am a experienced developer (primarily C#) but fairly new to Azure and to OCR. I work for a large insurance company that uses a legacy OCR extraction and processing tool.  We are examining current technology alternatives.  

I know this is a long post but I wanted to provide a total picture of what we are trying to accomplish.

- We process 100s of 1000s of documents per month, millions of pages.

- Most of our documents contain multiple pages.

- Many of our documents contain multiple document types (Policy, Cancellations, Bills, etc.).

- We service 1000s of clients, and 10s of 1000s of document layouts, and many of the document layouts change often.

- Building and maintaining models (templates) is not an option for us (we gave up on templates a long time ago)

We are working on a proof of concept (POC). Currently, we use the Forms Recognizer (FR) API to extract data using the General Document process and save the data to SQL Server tables (keyvalue, pagelines, pagewords, paragraphs, tables) in our data center.  Then we run stored procedures against the extracted data (post-extraction process) to select the specific values we need.  Sometimes the Key/Value pairs give us what we need but often we must further process the extracted data.  For example, parse street address, city, state, zip from a string that contains this information. Or find effective and expiration dates that were not picked up by the Key/Value pair data.

Most of the examples I have found deal with Pre-built or Custom Forms FR processes where the structure of the document is known and specific documents may be used to train the extract model.  Any General Document articles/videos only go as far as extracting the data, but not performing any post-extract processing or any decent Verification process. There is a Verify process using Cognitive Services but it is not real usable for documents with dense content, or for multi-page documents. Also, all demos use pristine, single-page documents that produce way-too-clean results.

The process we need to replace:

- Extract text from documents (mostly pdf and tif), but possibly other image types (jpg, png, etc)

   Data is extracted and saved to on-prem (data centers) SQL Server tables.

   Data contains x/y coordinates that point to the location of the extracted data.

- Process the extracted data by searching for "KeyWords" that are mapped (associated) to our FieldNames in extracted text

   Example: Policy Number (FieldName) has many KeyWords mapped to it (Policy No, Policy No:, Policy Nbr, Policy Code etc.).

- Verify the "processed" data - a human process

   Our (legacy) OCR application has a module that shows the document and the FieldNames that have been "processed" for the document. 

   When a FieldName is selected (clicked), its source is brought into focus outlined on the document.

   The Verifier (human) may approve the processed data (FieldName) or select a different value from the document by highlighting the desired replacement text (which is now copied) and selecting Replace.

We would like to keep this POC to Azure-provide features if possible.

If anyone is still reading (😊), I am hoping I can get pointed in the right direction.

THANK you.

[Nived_Nambiar]

Hi @rswayne 

Does this helps you ?

https://powerautomate.microsoft.com/en-us/blog/automate-document-processing-with-power-automate/

This is document processing framework present in power automate

Thanks & Regards,

Nived N 🚀

LinkedIn: Nived N's LinkedIn
YouTube: Nived N's YouTube Channel

🔍 Found my answer helpful? Please consider marking it as the solution!
Your appreciation keeps me motivated. Thank you! 🙌

[takolota]

@Nived_Nambiar @rswayne 

I think that is a good example set up of the general workflow, but I think it may still require training models for different document formats

https://learn.microsoft.com/en-us/ai-builder/form-processing-model-overview

If you want something pretty general to any Word, PDF, or image text scenario then you can check over this template to see if it will work for your scenario… https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Extract-Data-From-PDFs-and-Images-With-G...

This set-up extracts text from any PDF or image, replicates the document text positioning in a text format, then passes the text formatted data to an Azure GPT prompt where you can specify & customize data extraction processing to whatever you need. If Microsoft sticks to charging something in the 1-3x the equivalent credits of $.003 per 4000 GPT prompt characters usually charged for GPT services, then it may be half as expensive as the other AI Builder data extraction methods too. And if they start to charge more than that, then with an Azure account there is still a way to set up a GPT service in Azure to do the same thing for the $.003 per 4000 characters price, it just will take extra work.

[rswayne]

Hi Nived.  Thank you for your reply.  I already have the link you sent me and have reviewed it (I may need to spend more time with it).  I found a video of a person actually showing some of this using Power Automate.  It looks like we must use expected document layouts to train models for extraction.  I understand models to be "templates" for known document layouts.  Like I said in my original post, we have given up on templates because there are so many variations of so many document that keeping up with all of them, and their changes, was too much.  I may be mis-understanding what a model is but from what I have been able to research, models are useful when you have a limited number of variations of a known document.  We are dealing with a staggering number of variations of many different document layouts for many different document types from many different clients. I am hoping that further research will show me that I am not looking at the problem the right way !!

Also, the example of a validation process (what we call verification) is a step in the right direction but does not  appear to meet our requirements for this feature:

• Navigate to the area on the displayed document from where the extracted value was retrieved.
• Highlight the text on the document that was extracted. 
• Allow the user to change the selected value by selecting a different value from the document

A video I have seen using this feature (also) shows a simple example and it  supports the lack of I just described.

We also must be concerned with protecting very personal data.  We really do not want to store out source documents outside of Azure.

As I get back to the business with research results some requirements could change.

Thank you, again, for your reply.  It is MUCH appreciated. I am back in the office on Tue and will be continuing the research.

[rswayne]

Hi @takolota.  Thank you for your reply.  I need to look at the "train your models" requirement.  Like I said earlier, we have so many variations of documents (by document type, by client ...) that maintaining models (I am looking at models being templates for known documents) will be very difficult.  Your suggestion of using Azure GPT is interesting and I will have to look into this more.

I have created a POC using C# to call the Forms Recognizer API, saving the extracted data to SQL Server, then executing stored procedures to process the extracted data -- described in my original post in the paragraph starting with "We are working on a proof of concept (POC)."

Thanks, again, for your reply.  I am grateful for all the help I can get with this.  I'll update this post as research progresses!