different models for different vendors

[seetharaman]

Hello Experts ,

i have a question regarding AI model training. 

we are dealing with documents from different vendors (not invoices). each vendor has different layouts obviously , we are extracting 7 data points in total. some of these values are  appearing inside some table , but we don't want the table data actually  we are interested in only one value. 

we have decided to take an approach of creating AI model for each vendor. is this a good approach , rather than creating layouts for each vendor ?  by any chance creating layouts will reduce the accuracy comparing to having a separate model for the vendor ? i understand that there are challenges in choosing separate models , like 

how to route the document belong to a  vendor to the right model . 

Feed back loop becomes little complex and documents need to be routed back to the right model for retraining .

there may be difficulties in maintaining multiple models , not sure. 

but from the accuracy point of view will this approach of separate model will considerably make a difference ?

also maintaining separate models rather than one model will it  have any pricing / cost impacts? experts please suggest 

@jo 

[CedrickB]

We have created the collection mechanism to precisely address this kind of problem.
See https://learn.microsoft.com/ai-builder/create-form-processing-model#group-documents-by-collections 
You can create one collection per vendor and thus only have to maintain one model.
In addition, you don't have to worry about model routing as every document will be processed by your unique model which will automagically resolve the appropriate matching collection.

[seetharaman]

ok , the question is , maintaining a separate model per vendor will increase the accuracy as against having collections as solutions. 

also i have one more question regarding Feed back loop . 

https://learn.microsoft.com/en-us/ai-builder/feedback-loop?source=recommendations this link from microsoft documentation talks about feed back loop. does it happen automatically ? 

i mean lets say document 1234 has failed on the data point lets say "booking number ". my understanding is that , via feed back loop the document 1234 will be stored along with the fact that the model "A" failed to extract "bookin number" from this. 

weekly / monthly some human has to collect the data from the feed back loop and train the  model again. 

if this  under standing is wrong.  i wan to know how the training atuomatically happens ? how does the system know the model "A" is lacking in data extraction as far as "bookin number" feild. 

we need some one to teach the model again and improve right ? how  the retraining  happens automatically ? 

or feed back loop and retraining are two different concepts? 

if this sounds as a seperate question i will ask one more , but kindly reply for this. 

[CedrickB · Accepted solution]

You will get the same accuracy with separate models than using collections, so it is more convenient to use collections.
The way feedback loop is intended to work is the following:

1. Once a document is extracted, you will implement automated validation checks in Power Automate to decide wherever the data extract is correct and document must be manually review or if it can be directly sent to the target system.

2. If sent to manual review, you will provide your reviewers with an application (You can take inspiration or directly use the Document Automation kit) where they can fix missing/incorrect data and decide to send it to the feedback loop to improve the model

3. Adding to the feedback loop is performed by using the "Send to feedback loop" action in Power Automate (You can have a "Send to feedback loop" button in your manual review application which will trigger a flow calling this task)

4. When a document is sent to the feedback loop, the Maker will get a notification that the model can be improved and will have to tag it, then retrain the model.

We decided to keep manual gating in the above process to avoid any uncontrolled changes on the model which could harm accuracy.