Extract Hand Written Text and Train with AI Builder

[Runner55552]

I am starting to use the AI Builder within Power Automate, and have trained a model to recognize fields in hand-written forms.  The field recognition works fine, but some of the hand-written answers, such as the person's name, are not providing accurate answers. Is there a way to "train" the model to provide the correct answer? For example, I have a Name field, with only a small number of names that would be entered.  When the model sees John Smith in hand-written form, it incorrectly interprets it from form to form. Can I tell it during training that this answer in this form is "John Smith"? So far, although you can see the extracted text during training, I have not found an option to correct the answer. The values are shown during training, but cannot edit.

[JoeF-MSFT · Accepted solution]

Hi @Runner55552, thanks for the question! 

Unfortunately, today it's not possible to edit values that are detected during training. 

After you train the model and use it, do you get a low confidence score for handwritten text that is wrongly detected? If so, you could build a condition that in case of low confidence score for a field, trigger an alert for a human to review the extracted data and push the right data to the desired destination. 

Another option is to use expressions in Power Automate to format the data after extraction. For example, if collector name always come in the format K. yakes, the expression below will convert it to K. Yakes. 

concat(split({REPLACE-BY-YOUR-VALUE}, ' ')[0], toUpper(slice({REPLACE-BY-YOUR-VALUE}, indexOf({REPLACE-BY-YOUR-VALUE}, ' '), add(indexOf({REPLACE-BY-YOUR-VALUE}, ' '), 2))), slice({REPLACE-BY-YOUR-VALUE}, add(indexOf({REPLACE-BY-YOUR-VALUE}, ' '), 2)))

We're also constantly improving the quality of text detection technology, including for handwritten text, so over time you should see better and better text recognition. 

[Runner55552]

Thanks. The confidence scores from the training for that field were very high. It appears the confidence is mainly that it found the field in the right location, not so much whether it correctly interpreted the hand writing.

I was thinking along the same lines as you noted above, if there are repetitive errors, use replace and split functions to correct them. It had a harder time with one of the other names, spelling it differently multiple times, but I could test to see if the First name began with G then if so, replace with the full first name and last name. It seemed to recognize the first G quite well.

[Diallom]

Hello, 

How to add a step in case of low confidence score for a field, trigger an alert for a human to review the extracted data? CN you please provide a video on how to add the human review screen with the image form displayed ? 

[SalmanShaik]

Hi @JoeF-MSFT ,

I am extracting 30 fields from a single form can you share me the flow if any of the fields confidence score is less than 90% then trigger an email for manual verification it will be useful to everyone. I have to run this in a loop since i will be having 1000+ documents in a single shot...
This feature is already available by default in other RPA tools only we have to select the confidence score criteria it will be sent to manual verification automatically, human has to do the necessary changes.