Form Processing Doesn't Recognize Fields

[jasonkhonlaw]

The AI Builder form processing doesn't work well enough to be usable.  I know it's in preview, however it really needs the ability to allow the user to create their own fields and then select an area of the document.  Even though my PDF invoices are originals (no scanned) and are very clear, it doesn't recognize many of the fields.  I can even select and copy and paste the text right out of the PDF, but the form processor doesn't recognize them.  It also doesn't recognize the table of line items as a table, even though some of the documents used in the analysis have many rows of data there.  It didn't even recognize the bill to address, which seems like a pretty common one.  Any suggestions? Please see attachment.

[NitramArbok99]

Microsoft please step up the game. I've been using doc-parser which is very powerful.  When using AI Builder it can not recognize many fields which are obvious like invoice number, net weight and so on. 

[JoeF-MSFT]

Hello everybody,

Yesterday we released an update to Form processing where you can now draw fields that were not automatically detected by AI Builder. If you had tried Form processing before and you were not able to extract all the fields you wanted, I invite you to give it a try now.

Do not hesitate to share your feedback as you try it out!

More information on our documentation page: https://docs.microsoft.com/en-us/ai-builder/create-form-processing-model

[markslosberg]

Will give it a try now.

[markslosberg]

Ok, that was a little better but still pretty far from the target.  I think I did better on the single page Packing Slips but the multiple page slips aren't working too well.  I don't really know whether the single-page packing slip is working either but it doesn't give an error.

This is a fairly complex form.  I am happy to work with someone if they want to go through it with me.SinglePagePackingSlipMultiPagePackingSlip

[JoeF-MSFT]

Thanks @markslosberg, for trying it out.

I've sent you a private message to discuss into more details your use case. 

[Jcrone]

I am also having this issue. Even with the option to draw around the values it does not want to recognize it. I think it is the fact the the are grey boxes with text inside throwing it off (Editable fields).. If I change them to Jpeg then the AI builder can read the values. It's weird that it can not recognize it in PDF format but can in Jpeg.... 

[JoeF-MSFT]

Hi @Jcrone,

The way we extract text from images and PDF are different, and that's why you are seeing different results. We are working on improving the extraction of data from PDFs with editable fields. 

[ehalsey]

We are facing similar challenges w/ Lab Test Results forms we receive from various labs.  It fails to recognize a table we have and there is no way to add it.  I've tested creating samples with different values to try to help it but it just can't detect this specific table no matter what I try.  A couple of questions:

1. How does the builder determine which form to use to "pick" the fields?  Can this be user specified? Seems a bit arbitrary in my limited testing?
2. Can you share the rules used to detect tables e.g. bold headers, column aligned data, vertical distance between fields etc.? 
3. If we can create forms for training with "hints" what is the likeliness forms provided by vendors will be recognized if the font is  different, the tables are in a slightly different vertical position, etc.?
4. Is there a documented roadmap w/ estimated dates?  We have real projects and having some idea on when the required features will be available will be helpful.  Like others, we need the ability to automatically ID the model based on the provided form e.g. Lab 1 Form vs. Lab 2 Form, etc.
5. Does AI Builder create MS Form Recognizer OCR layout files behind the scenes?  If so, will we be able to make manual adjustments when AI Builder Analyze process guesses incorrectly?

Attached example of one of the forms we are trying to recognize.  Blurred areas added to protect customer info.  Very promising stuff.  Hoping it gets to the finish line so it can be used in a real world environment.

Regards,

Eric

[JoeF-MSFT]

Hi @ehalsey,

Apologies for the delay in replying. Thanks for the questions, here are the answers:

1. How does the builder determine which form to use to "pick" the fields?  Can this be user specified? Seems a bit arbitrary in my limited testing?
   
   It's the first form that gets uploaded to AI Builder. 
   
   
2. Can you share the rules used to detect tables e.g. bold headers, column aligned data, vertical distance between fields etc.?
   
   For best results, tables should have line separators between columns and rows. Text should be aligned the same way across columns. 
    
3. If we can create forms for training with "hints" what is the likeliness forms provided by vendors will be recognized if the font is  different, the tables are in a slightly different vertical position, etc.?
   
   If the forms change slightly, the model can still pick up the values where you have provided hints (manually drawn fields). For better performance tag a minimum of 10 documents.
   
   
4. Is there a documented roadmap w/ estimated dates?  We have real projects and having some idea on when the required features will be available will be helpful.  Like others, we need the ability to automatically ID the model based on the provided form e.g. Lab 1 Form vs. Lab 2 Form, etc.
   
   The ability to train multiple form types within a single model is planned for the 2020 Wave 2 release --> October 2020
   
   
5. Does AI Builder create MS Form Recognizer OCR layout files behind the scenes?  If so, will we be able to make manual adjustments when AI Builder Analyze process guesses incorrectly?
   
   We use Azure Form Recognizer behind the scenes but we are not planning to expose the layout files. Can you give me an example of manual adjustment you would like to be able to do?

Regarding tables and looking at the screenshot you provided, if tables have always the same number of rows, you can manually tag each row as a separate field. 

Hope this clarifies. Feel free to reach back with any additional question!

[EricHalsey_HB]

Thanks for the reply Joe!  This does help.  See answers below.

How does the builder determine which form to use to "pick" the fields?  Can this be user specified? Seems a bit arbitrary in my limited testing?
It's the first form that gets uploaded to AI Builder. 
Great!  This helps

Can you share the rules used to detect tables e.g. bold headers, column aligned data, vertical distance between fields etc.?
For best results, tables should have line separators between columns and rows. Text should be aligned the same way across columns. 
Do you know if we trained with lines but the real forms do not have lines it would still pick up the tables?

If we can create forms for training with "hints" what is the likeliness forms provided by vendors will be recognized if the font is  different, the tables are in a slightly different vertical position, etc.?
If the forms change slightly, the model can still pick up the values where you have provided hints (manually drawn fields). For better performance tag a minimum of 10 documents.
Awesome!  Did not realize we could use more than 5 training documents

Is there a documented roadmap w/ estimated dates?  We have real projects and having some idea on when the required features will be available will be helpful.  Like others, we need the ability to automatically ID the model based on the provided form e.g. Lab 1 Form vs. Lab 2 Form, etc.
The ability to train multiple form types within a single model is planned for the 2020 Wave 2 release --> October 2020
Great.  This does help w/ planning.

Does AI Builder create MS Form Recognizer OCR layout files behind the scenes?  If so, will we be able to make manual adjustments when AI Builder Analyze process guesses incorrectly?
We use Azure Form Recognizer behind the scenes but we are not planning to expose the layout files. Can you give me an example of manual adjustment you would like to be able to do?

Sure.  b/c AI Builder is not picking up tables & fields properly it would be nice to have AI Builder do most of the heavy lifting and then be able to modify the layout file to make whatever adjustments, e.g. table locations & names

Regarding tables and looking at the screenshot you provided, if tables have always the same number of rows, you can manually tag each row as a separate field. 

Unfortunately that is not the case as there will be a variable number of rows.  I will work w/ the customer to see if they can provide more samples for training that include variable number of rows for that table.