Multi-page table spanning issues

[Wittig]

One of our suppliers sends multi-page invoices where the invoice summary information at the top of each page is repeated then a table begins with with 5 named columns  (Description, Order Qty, Shipped Qty, Unit Price, Net). The description column field can have multiple rows and is overloaded to include 3 pieces of information we wish to extract (Manufacturer, Model #, Actual description of the item).

I have been using the advanced tagging and identifying all 7 pieces of information we wish to extract (3 pieces in the description column and 1 piece each in the remaining 4 columns. I have also been using the multi-page support since the table can span multiple pages.

The issue I have now is a single item can span two pages i.e. the overloaded description column field had one line at the end of one page and the remainder on the next. Any ideas on how to properly tag this?

Here is a rough example of the issue. Item #3 is the issue and is split between pages:

------------------------------------------------------------

Supplier name                             order date

Address                                       order number

Phone

------------------------------------------------------------

  Description                    Order Qty         Ship Qty   Unit Price       Net

  MyMfg1  MyModel#1           1                      1            $10             $10

  MyDescription of #1

  MyMfg2  MyModel#2           1                      1            $10             $10

  MyDescription of #2

  MyMfg3  MyModel#3           1                      1            $10             $10

-------------------------- Page Break ------------------------------

Supplier name                             order date

Address                                       order number

Phone

------------------------------------------------------------

  Description                    Order Qty         Ship Qty   Unit Price       Net

  MyDescription of #3

  MyMfg4  MyModel#4           1                      1            $10             $10

  MyDescription of #4

  MyMfg5  MyModel#5           1                      1            $10             $10

  MyDescription of #5

[Antrod]

Hi @Wittig ,

Thanks for reaching out on that.

Indeed, there's no possibility for now to continue tagging a field that started in a page and spans over a second page.

Did you try not tagging documents in this case but only documents with descriptions that don't span over two pages? Then see if at prediction time, such documents are predicted correctly.

[Wittig]

I found 12 invoices which span multiple pages but do not split the overloaded description field. After tagging these files as well as 12 more single page invoices the accuracy score is now 87% across the entire model but the overloaded field is only 39%. Is there a way to improve this?

[Wittig]

Bad news. Once the model encounters one of these "page spanned" description fields the accuracy goes way down for every entry thereafter. Even when it goes to a new page with no "page spanned" description it can't recover. In one document I tested it even started to drop columns.

At this point it is unusable which is very disappointing.  

[Antrod]

Hi @Wittig ,

Sorry to hear about that.

Did you try to tag the Description as a unique field (instead of 3 fields) and see if you get better results?  If it's the case, then perhaps you would be able to do post processing logic in a flow to separate the values.

[Wittig]

I have not tried that but if the AI model can't handle the complex parts of the document, I could just write a non-AI parser for the entire document.  Our goal is to add more collections which contain invoices from other suppliers so we could have one process for all.  Also, I would have to start the entire document tagging process over again or lose all the existing tagging.

I did try yet again to add even more training documents to the model which contained many of the special cases/anomalies which show up in the invoices. Unfortunately, the accuracy score went down as a result and when I tested it with a document, I used to train it with it couldn't replicate the training data.

Are there any plans for improving the types of issues I am encountering? If so, is there a timeline?

[Wittig]

Antrod - I tried what you suggested, and I still get poor results. I trained a new model with 8 documents, and it has problems with both single and multi-page documents.

I have no idea how to make the tables any simpler.  

[Antrod]

Hi @Wittig ,

I'm really sorry to hear that latest tests weren't positive.

We unfortunately have no date to have this scenario covered.

Are you using the Structured version of the Document processing model? If it's the case, the very last thing I may suggest you is to test with the Unstructured version. I understand it could be tedious to recreate the model but perhaps you could create a very basic version of the model just to see if you have better result.

[Charanjit]

@Antrod @Wittig  I am facing the exact same issue. I need to extract information in tabular format from order confirmation pdfs received from suppliers. Each pdf has multiple items and each item will have a code, description, quantity and delivery date. So the table will have four columns: Code , Description, Quantity, Delivery Date with each row representing an item. 
The problem arises when some details for an item are present at the bottom of one page and the remaining details are on the next page. 

The document cannot be tagged correctly in such case and the accuracy of model is pretty bad. Suppose the document has 2 pages then the tagged tables look like this

Code	Description	Quantity	Delivery Date
101	this is first item	56	22.08.2023
102	this is second item	65	23.08.2023
103	this is third item

Code	Description	Quantity	Delivery Date
		72	24.08.2023
104	this is fourth item	80	23.08.2023
105	this is fifth item	60	21.08.2023

[takolota]

@Charanjit @Wittig 

You could also try this template that just uses text recognition to create a text replica of the given file & feeds it to GPT for data extraction:

https://powerusers.microsoft.com/t5/Power-Automate-Cookbook/Extract-Data-From-PDFs-and-Images-With-G...

[Titi1234]

Hi @Charanjit ,

Did you manage to resolve this issue?

If yes, could you please provide the solution?