AI Builder: Extract content from multipage tables in document processing is working for 3 pages afterwards it stops working

[BramVeldeman]

Hello,

I created a custom Document Processing AI Builder model. The layout we want to capture is an Order Confirmation from our supplier towards us.
This model includes some header fields (e.g. supplier, date, references...) and an Items table (e.g. item, quantity, delivery date, price ...)


We used the following feature (Extract content from multipage tables in document processing | Microsoft Learn)

However only 3 pages get correctly processed. (So the data from page 1 / 2 / 3 get processed correctly, page 4 and following aren't captured)

The following info is available on the page I posted:

Feature details

An experimental feature currently exists in AI Builder that allows you to extract content from multipage tables. The feature is limited and doesn't always detect tables accurately if they span more than two or three pages.

I think this is the problem I have. Is there a timespan for solving this issue?

(Extra: AI Builder document processing isn't extracting tables that span across multiple pages | Microsoft L...)

Regards,

Bram

[JoeF-MSFT]

Hi @BramVeldeman - thanks for letting us know about this.

How many documents did you use for training? Those training documents, how many documents there are where the table spans more than 3 pages?

Can you add more sample documents for training where the table spans more than 3 pages? This will help the model better learn.

Let us know if this helps. 

[BramVeldeman]

Hello, I used 7 documents, 3 document have > 3 pages. 4 documents only 1 page.

When I test only the first 3 pages get captured. AI Builder doesn't capture the other ones.

I will try to add some extra documents with > 3 pages. 

Regards,

Bram

[BramVeldeman]

Hello @JoeF-MSFT 

My colleague and I each trained an own collection with similar results.

Collection 1:

• 13 documents
  • 1x 1 Page
  • 3x 2 Pages
  • 3x 3 Pages
  • 3x 4 Pages
  • 1x 6 Pages
  • 2x 8 Pages

Collection 2:

• 16 documents
  • 4x 1 Page
  • 2x 4 Pages
  • 1x 5 Pages
  • 2x 6 Pages
  • 4x 7 Pages
  • 2x 8 Pages
  • 1x 9 Pages

Once a document passes 3 pages the performances / results of AI builder aren't trustworthy for the following pages. Page 1 - 3 gets captured almost perfect. After page 3 the models doesn't recognizes certain fields, or whole pages...

Could we contact you to have a look at this?

Thanks in advance and Kind regards,

Bram 

[JoeF-MSFT]

Hi Bram - could you do the following test?

1. Edit your existing model. 

2. Select Unstructured documents as document type.

3. Retrain the model. 

After the model is trained, do you get better results?

[BramVeldeman]

Hello @JoeF-MSFT 

our findings:

the performance of the model when testing has increased dramatically. Sometimes it makes mistakes with the pages at the end but it's a lot better than before.

I Don't get it to be honnest.. We have structured documents when I look at the examples so why did I have to choose for unstructured documents? Is this logic?

Thanks for the help and kind regards,

Bram

[BramVeldeman]

Hello another remark for the product team.

I have read that when exporting an AI model and importing into another environment only the Model is exported and not the training data. Therefore it isn't possible to edit the Model in the new environment. 

As we started in a test environment this is quite a pitty as we lose a couple of days by recreating our collections..
This could be improved.


Kind regards,
Bram

[JoeF-MSFT]

Hi @BramVeldeman - thanks for sharing the outcomes of using the unstructured document option. Under the hood it uses a newer AI technology that works very well for unstructured documents but also for structured. Currently it works best for documents in English and it's not available in all Power Automate regions. In a future AI Builder update, once we support more languages and regions, we will most certainty just use this new AI model for both structured and unstructured documents. 

Thanks for the feedback as well on moving training data across environments - this is something that it's also planned in an upcoming AI Builder update. 

[BramVeldeman]

Hello @JoeF-MSFT thanks for the feedback - now it makes more sense. We keep op testing and hope our Proof Of Concept works out the way we planned 👍

Another option that my colleague and I discussed today that would be a big improvement in our eyes: 
The possibility to train the model while testing the model.

ideal flow in our eyes. 

1) I use 5 documents to train the initial model

2) afterwards I upload a test document and notice a couple of mistakes (e.g. 20% is wrong) > I correct them and add the tested document + corrected data to the model

3) retrain the model based on 6 documents

Now it's quite time consuming for some of our vendors to train the tables with multi page documents (+ not so structured tables that require advanced tagging). It would be a great time saver if it would be possible to add a (corrected) test document to a collection so not every single field has to be re-tagged every single time. 

Kind regards,
Bram

[JoeF-MSFT]

Thanks @BramVeldeman - this is fantastic feedback! 🙂 It goes in the direction we're working towards: making it easier and more delightful to build document processing automation solutions. 

Have you tried the new feedback loop feature? Continuously improve your model (preview) - AI Builder | Microsoft Learn You can add condition in your flow where if some quality conditions are not met, the document is sent back to the training set.