OCR to text account key - what is that?

[mahakala_]

Hi I am trying to use flow with powerapps to get OCR to text functionality.  While building out in flow, I added the Computer Vision API for the OCR to text, and one of the fields is an account key.  I am not sure what that is.  I am using a trial version of of powerapps, so I am guessing I need a license for the OCR?  Can someone please help.   My canvas app will take a picture I have taken with the phone, and it will convert the numbers in the image into text.  I am trying to go through flow to do it.  If you have other suggestions please advise. Thanks.

[mahakala_]

Sorry @PaulD1,

Please disregard the last reply to you.  The flow shows up now.  I realized i had two environments, and the flow i created initially was in an environment different from my app.  I will continue to play with it and let you know how it goes. Thanks again! 

[mahakala_]

Hi @PaulD1,

I got an error when hitting the button to convert to text.  Have you seen this before?  I am doing this from my phone.  I have a full bar signal.  

[mahakala_ · Accepted solution]

Hi @PaulD1,

Just wanted to let you know I was able to figure out what caused that error message in flow.  So turns in the flow history, it shows my failed flows and there is a generic detailed description explaining why it failed.  So error msg was Access denied. You do not have permission to perform this action or access this resource.  I thought I didn't have access to sharepoint, but I see in flow that the connection was made to sharepoint.  So it turned out that I had the Site address wrong.  I was able to find the correct address and it now works.  Hoping you can help me answer these two next questions:

1)  While i was trying to get this to work, I was trying to get it create file within one drive for business or google drive.  So I did get it to work with one drive for business and tried it for at least 5 times, but it started to fail afterwards.  I was getting the following msg, wondering if you know what it means (got the same msg for google drive as well):

Unable to process template language expressions in action 'Create_file_2' inputs at line '1' and column '2092': 'The template language expression 'json(decodeBase64(triggerOutputs().headers['X-MS-APIM-Tokens']))['$connections']['shared_onedriveforbusiness']['connectionId']' cannot be evaluated because property 'shared_onedriveforbusiness' doesn't exist, available properties are 'shared_sharepointonline, shared_cognitiveservicescomputervision'. Please see https://aka.ms/logicexpressions for usage details.'.

2)  Now since I got the ocr to text to work many thanks to you, sometimes it does not seem accurate.  Is there a way to make it more accurate?  and is there a way to use regular expressions or filter, to extract just the part of the text that I want?  

Many thanks again.  I will mark this as solved.  

[PaulD1]

Hi @mahakala_ 

Regarding the error message - please check your OD4B connection. Maybe try deleting and recreating the connection and making sure your Flow is pointing to the latest 'version' of you connection.

Accuracy will depend a fair bit on the quality of the image being processed. In the PowerApps player app for Android (and I think for iOS), under the hamburger menu there is a setting for camera image resolution, so you may want to check that. Beyond that I think all you can do is try to capture well lit, focused images of the text. I did experiment using OCR on car odometers and it was very flakey due to the fonts used, but for normal text I've had pretty good results.

If the issue appears to be the resolution (I believe the max setting equates to 640x480 when using the PowerApps Camera control) you could try an approach such as capturing the image using Office Lens set to save to OD4B and having a Flow that is triggered by the arrival of a new image in OD4B. This has the benefit of a higher resolution image with contrast etc, set to give more read-able text. The downside is that the processing will have a delay (you have to wait for the Flow to be triggered which could be a few minutes) and will not give a response direct to the PowerApp, so this may not work with your scenario.

How would you identify the part of the text that you want? The OCR to JSON action gives you back the location of your text within the image (though you'll want to work with consistent 'scans' of consistent documents to pick out text by location). Otherwise you can do some string manipulation to find, say, a consistent 'heading' and bring back all text between that and the next 'heading', but doing so in PowerApps formulas is tricky when the rules become complex. An alternative might be to look at using an Azure function for the text manipulation.

[mahakala_]

Hi @PaulD1,

Thanks for your help.  The camera image resolution is already enabled in the powerapps player app.  Maybe I can stick with what I have but just do string manipulation as you suggested, or just take a picture with the phone and use it as input to the app.  But I think i am in a good place now thanks to you.  I'll try to tinker with it some more.  

[mahakala_]

Hi @PaulD1,

Sorry to keep bugging you, but you seem to be the main OCR guy in this small community.  I am trying to implement your solution to OCR to text in a different way. Instead of using a camera control to take pictures, I want to use the Add picture control and grab a picture that I have already taken with the phone.  Then send it via flow to azure and back.  I would assume that the OCR to Text step in flow would need to be changed since I am grabbing an existing picture into powerapps, instead of powerapps actually taking the picture, and which from what I understand is a uri image.  In this case would I still use 'Image Content' as the Image Source, and what would I use for the Image Content?  Thanks 

[mahakala_]

Hi @JoeF-MSFT,

I was able to create the text recognition flow based on the article you provided.  The last step of that article was a 'Post a message' action.  It seems this sends the text results as a message.  But I am trying to send it back to powerapps.  How would you send the results back to powerapps?  And how would you invoke it from powerapps?  I was able to invoke it from powerapps, through flow, to azure and then the results are sent back to powerapps.   But I am trying to do it with the AI builder in flow.  Thanks.

[PaulD1]

I have found the easiest way of working with images from Add Picture (rather than the camera) control is to save the image to Azure Blob Storage (using the Azure Blob Storage connector which sadly is now premium). Saving the file to Blob storage is easily done from within PowerApps (no need for Flow for that part).

Having saved the image to Blob storage, you can then pass the container (folder) and file name to a Flow which can then easily use the Blob Storage connector to Get File Content and then pass this to the Azure Cognitive Services (using the action Optical Character Recognition (OCR) to Text action).

As the OCR to Text action returns a simple string, you can use the Respond to PowerApps step to return the 'Detected Text' as text.

The resulting formula looks like below - where TextFromBlob is a Flow that grabs the file content for the specified Blob (image with text), passes it to OCR to Text and then sends the text output from that step to PowerApps via the Respond to PowerApps step.

AzureBlobStorage.CreateBlockBlob(
    "imagetest",
    "mytest1.jpg",
    UploadedImageDetect.Image
);
Set(
    varOCRtext,
    TextFromBlob.Run("imagetest/mytest1.jpg").OCRText
)

[JoeF-MSFT]

Hi @mahakala_,

Great to hear that you are trying out AI Builder. To send the results back to Power Apps you can use the Respond to Power Apps action as rightly pointed out by @PaulD1 . More info about this here: https://flow.microsoft.com/en-us/blog/return-data-to-powerapps/

[mahakala_]

@PaulD1 ,

So with blob storage,  I would have to provide API key in order to start using that code in powerapps?  

AzureBlobStorage.CreateBlockBlob(
    "imagetest",
    "mytest1.jpg",
    UploadedImageDetect.Image
);
Set(
    varOCRtext,
    TextFromBlob.Run("imagetest/mytest1.jpg").OCRText
)

And would all this code be in the OnSelect for a Button?  So if I am understanding correctly, I would need the API key for the blob storage and the API key for the Computer Vision?

So when the Set function would send the image from blob storage to flow, I would use the same flow sequence as I would while using the camera, but eliminating the "create file" in sharepoint?  So the whole sequence would be :

1)PowerApps

2)Optical Character Recognition (OCR) to Text (Preview)

3)Respond to a PowerApp or flow.

Thanks.