Object Detection using images from a timeline in Dataverse.

[lowcodelewis]

Hi,

I'm attempting to use the object detection capabilities in AI builder in a Power Automate flow to take an image added to a timeline and then run a model against this to determine whether there is a specific object in that image.

I'm getting a record and then listing the notes on that record. In my solution there will only ever be one note at this point as this is triggering on the creation of the parent record. The note is then created straight away after (so yes a delay may be needed).

Eitherway, I am getting my file content in the 'documentbody' column of my note record.

I'm then using this as an input to my model but an additional amount of JSON is added to the content which I believe is causing things to fail.

If I do a simple flow with a file input and use the outputs from the trigger as the content, things work with the exact same file.

Could anyone suggest how to use file content from Dataverse and pass this through an AI builder object detection model?

I have also tried using the download a file action in Dataverse 

Thanks in advance

Lewis Baybutt

Microsoft MVP Business Applications

lowcodelewis.com

[lowcodelewis]

More info - I have tried to simplify this by just triggering my flow on the creation of a record in the notes table then using the document dynamic content which I believe is the file content to use as the image input to my AI builder step for object detection. This still isn't happy as a supported file type. 😞

[CedrickB]

You should use this to upload your file to Dataverse

and this to retrieve and use the file content  

[lowcodelewis]

Hi Cedrick,

Thanks for this. I have tried this, and have used loads of different methods in compose actions to see what sort of content things are expecting. I am managing to make things work when using a manual trigger with a file input and then referencing that input from the trigger body in my AI builder step. This works.

Using the download a file or an image step works but then the AI builder step doesn't like that dynamic content as inputs. The outputs from the dataverse download action look right and would be what I expect to use as input for the next step. But when you then download the inputs of the AI builder step there is additional JSON at the start not found in the previous step outputs resulting in the input not being just file content. 

[lowcodelewis]

Forgive me using the dynamic content from the dataverse step looks like it just results in inputting a string being the file name. If you pass it through a compose first you seem to get the expected binary file content, but then additional JSON is added when you look at the run history on the AI builder object detection step.

Thanks,

Lewis Baybutt

Microsoft MVP Business Applications

lowcodelewis.com

[CedrickB]

"Download a file or an image" should retrieve the binary properly.

How did you save you file to dataverse? The problem may be there.

Did you use the "Upload to a file or an image" action?

[lowcodelewis]

I have noticed that doing something like getting file content from OneDrive or SharePoint and then using this works.

Using the outputs from my get file content step here, which include an object including both properties $content-type and $content and this working, suggests to me that the input should be an image object like this. 

So going back to my flow where I'm trying to get my file content from Dataverse...

First I am downloading the image with the download a file or an image action, even though I do see all of the file content, and details i.e. documentbody, mimetype etc in my retrieved Dataverse record...

Then I'm trying to use the content out of this action. But given the outputs, I can understand in this case why not much is happening. These are the outputs which are nothing more than some text being the file name.

So... okay instead I could try to get the content directly from my record where I can actually see binary file content, a content-type under mimetype etc...

You can see I've got my file content and my file content-type now. When I try to put these together as an image object the AI model doesn't like that input and says the file type isnt supported. 

So maybe it doesn't want an object and maybe it just wants the binary file content?? So this is what the start of my file content looks like before it becomes an input to the AI builder step for object detection...

Looks right. Now lets see that output ^ as an input to the AI builder step, with suddenly extra content added!

Theres the additional bit at the front with the "item/requestv2/base64Encoded" property. Now comparing back to the OneDrive flow that worked, I would image we need the value of this property to be an image object with both $content-type and $content. That seemed to work before.

But using an image object as an input also doesn't seem to work and the property above seems to get generated itself. 

I'm now a bit lost 🙂 

[lowcodelewis · Accepted solution]

Finally got it to work! It seems the object did work. So I needed to use the 'documentbody' value as my file content and the 'mimetype' as the content type in an object like below...

Who knows what I was doing wrong with the object approach earlier but this works! Blog post upcoming! 🙂

Thanks for help 🙂

Thanks,

Lewis Baybutt

Microsoft MVP Business Applications

lowcodelewis.com

[lowcodelewis]

Yes I suppose this part or this method should have worked but didn't properly. I wasn't able to use the upload to a file or an image action. The image I'm trying to get to would be one that is created on a timeline of a new record submitted on a Power Pages basic form. So a bit odd that the download action wouldn't work on that note record but not to worry as I didn't need to use the action to get to the content and content-type.