AI builder handwriting recognition to table reference

[MRBCODC]

I want to make a flow that we can upload hand written time sheets to and store in SharePoint documents with a file name of name - date 

I have built the AI but the results are quite poor due the peoples handwriting.

is it possible to have the AI read name reference a table and if it is a close enough match use that name and also the date?

[JoeF-MSFT]

Hi @MRBCODC - thanks for sharing. 

When the results are poor, are you also getting back a low confidence score? One way could be to use that confidence score to decide whether you can use the extracted result or need human review.

The AI model by itself cannot be trained to match a list of names. Depending on the data source where you have the list of names, you might be able to query it from a Power Automate flow using fuzzy search. For example, Dataverse does support fuzzy search. In the flow, you would first run the AI model to extract the data, and then call the Dataverse search action to see if there is a match.

[MRBCODC]

Hi @JoeF-MSFT thanks for responding!

For the name field I am getting an 85% confidence score, but the actual output is not great at all.

The date is 99% confidence score and it is not too bad at recognizing the numbers but the date format is way off. (to be honest I would need a standard date format across the board)

Ok that does sound like what I am looking for. I am a bit of a newbie on all things power in O365 and have not used dataverse yet.

Would I need to create a table in power automate to use dataverse?

[JoeF-MSFT]

Hi @MRBCODC - welcome to Power Platform within O365! 

Yes, you will need to:

1. Create a table in Dataverse with the list of names. You can find more info here: Create a custom table (contains video) - Power Apps | Microsoft Docs
2. Enable Dataverse Search in your environment: https://docs.microsoft.com/en-us/power-platform/admin/configure-relevance-search-organization#enable...
3. Use the Search rows action in Dataverse. The following page explains how to configure the action correctly for fuzzy search: Use Dataverse search to retrieve rows - Power Automate | Microsoft Docs

Hope this helps. Keep us posted on the progress! Your learnings here can be interesting to share with others that want to build a similar use case. 🙂

[MRBCODC]

Hi @JoeF-MSFT 

I do not have access to be able to create tables in data verse and I am unlikely to get granted authority.

is there a way I can do this from excel or SharePoint?

another thought I had is if I was to upload a lot more time sheets will the AI get better at reading handwriting and is there a way to correct its mistakes? 

[JoeF-MSFT]

It sounds like your IT team is a hard one to convince. 🙂

One possibility could be using a SharePoint list and OData filtering. 

Here for example, I have this SharePoint list of people names:

I can use the substringof function in the Filter Query input to see if there is a name that contains a part of a name. In the example below, if I search for mar in the first parameter, I will get back Mario Rogers. 

It's not as powerful as the fuzzy search in Dataverse, but it might do the job. What do you think?

[MRBCODC]

Hi @JoeF-MSFT 

sorry for the long reply had another project on.

I don’t feel like that solution will work as the results from the hand writing  are too unpredictable. I’ll try it in my flow to see what kind of results I will get and let you know if it is suitable for me