Data failures with Multiple Collections in AI Model

[brianmayo]

  One of my vendors updated the format of their documents, so I added a second collection (5 samples) of the new format. The overall score of the AI model is still well over 90%, but when I ran a real document against it, the wrong collection was selected. The confidence score was only 53% (no score was provided for the second "appropriate" collection). Is there a way to determine the confidence score of the the second collection while running a flow to extract the AI data? 

Another post said that the first collection is always used by the flow, but I know this is not true. I have dozens of flows working with multiple collections and vendors and the flow normally can distinguish between the vendors. I am wondering if my two collections are so "close" that AI builder cannot really discern between them, and sometimes the wrong collection gets selected. I've viewed the formats of the documents, and my test document is right on the money compared to new format documents. However, the AI keep selecting the old format collection and getting the wrong data as a result.

Thanks for any suggestions!

[Antrod]

Hi @brianmayo ,

Indeed is format are very close, it's possible that the wrong collection is considered. Can you tell us (high level) which changes you have between your first and second collection in terms of document format?

Also are you using structured or unstructured document processing?

[brianmayo]

Hi, Antrod:

   At a high level:

1) I am using unstructured documents. We like the versatility of the unstructured format. 

2) The changes are minimal. In the header section of the document, one line (meaningless to us, and we don't even use this data), but it does shift certain fields down one row. So, spatial differences in the Header portion.

3) In the table, columns were shifted around spatially. All the same columns are used, in the same sequence spatially speaking. However, the width of each column changed, so the line items are shifted slightly horizontally. 

That's it...no other changes. 

Appreciate any thoughts out there,

Brian

[Antrod]

Hi @brianmayo ,

Thanks for the details explanations.

Based on your answer, I might think that documents could be considered the same AI-wise. You may want to test merging the 2 collections to check if you get better results.

If results aren't good after merging, you could also change the model to unstructured (if it's not the case already), it often provides better results.