Object detection and text recognition together

[jinivthakkar]

Hi All,

I want to do object detection and also extract text from the image, is there anyway to do both in one go ?

We want to detect objects like different grocery products, and then get the barcode details, weight etc.

Thanks in Advance.

[JoeF-MSFT]

Hi @jinivthakkar,

In a canvas Power App you will need to add the two AI Builder controls Object Detector and Text Recognizer.

One approach you can take is perform to object detection in a flow in Power Automate to identify the product, and then use the Power App to perform text recognition with the text recognizer control.

[jinivthakkar]

Hi Joe,

I understand that by using two AI builder controls it can be done.

But I want to do that using only one control, do you know a way it can be done ?

Detect object and text from the same object.

[JoeF-MSFT · Accepted solution]

Hi @jinivthakkar,

Unfortunately there is no AI Builder control today in Power Apps that can do both at the same time.  

[sossie07]

I have the same requirement, I want to detect an "Asset Tag" in an image, then read the text in the Asset Tag.

Has there been any advancement since 2020 where this functionality is possible with AI Builder and Flow?

[AVFC]

Hi @JoeF-MSFT 

I'm trying to create an meter reading app so i've trained my object detection model but now need to store those objects and perform text recognition on them. Can you elaborate on this or point me in the right direction please?

[sossie07]

Hi AVFC,

My requirements are similar in that I want to read asset tags in photos. The only way I found was to create an image analysis model, and a custom "document processing" model. The flow would analyse the image, and if it found an asset tag in it, I would send the image to the document processing model to read the text in the asset tag.

However I found this to be so unreliable it was not suitable for production use, and there is no way to "correct" or train incorrect reads of the text. So in my view its not really deserving of the name "Artifical Intelligence" as to my view Intelligence implies the ablity to learn over time. Microsoft should just rename it "Have A Guess" or "HAG" for short;-)

[AVFC]

Thanks @sossie07 for the reply, kinda sounds like what Joe suggested in message 2. Will have another play and see what i can do but as you say is kinda not messy...

[AVFC]

Would you mind elaborating on how you stored the results of your image analysis? I'm very new to Power platform and can't see which output i can use to store my captured meter readings. I've sent myself an email as a test from the flow but it just sends the coordinate details??

[sossie07]

If I understand your question correctly...

When document analysis is run in a flow, for each area of text that you tag in the AI Builder Model, there will be a range of items that it gets, in my case the tag is "Asset ID" and "Asset ID Value" is the one you want. "Asset ID Confidence" is useful for testing. e.g. Do if "Assett ID Confidence" > .8

I use Assett ID Value to rename the sharepoint file, and put details in a sharepoint list etc.