AUto tagging images using Computer Vision AI (Sort on confidence)

[JimmyWork]

Hi

So i created i flow to auto tag images when uploaded to a SharePoint library. This all works very good.
What i can see is that some words i just dont want it to tag the image with. For example i upload a image of a car and it will add a tag called "cake" 

If i check th Analyze Image results in flow i can see it rates tags based on confidence score.

So i wonder is it possible to filter or sort out words if confidence score is below a certain number? Now i dont see the tag cake have any confidence rating so i wonder why is this even added?

{
  "categories": [
    {
      "name": "trans_car",
      "score": 0.9921875
    }
  ],
  "tags": [
    {
      "name": "sky",
      "confidence": 0.9996563196182251
    },
    {
      "name": "car",
      "confidence": 0.9995741248130798
    },
    {
      "name": "outdoor",
      "confidence": 0.99254310131073
    },
    {
      "name": "blue",
      "confidence": 0.9446885585784912
    },
    {
      "name": "ocean",
      "confidence": 0.9404265880584717
    },
    {
      "name": "beach",
      "confidence": 0.8443412184715271
    },
    {
      "name": "vehicle",
      "confidence": 0.7354080677032471
    },
    {
      "name": "automotive",
      "confidence": 0.6446248292922974
    },
    {
      "name": "wheel",
      "confidence": 0.6275951862335205
    },
    {
      "name": "land vehicle",
      "confidence": 0.5628407001495361
    },
    {
      "name": "sandy",
      "confidence": 0.181476891040802
    },
    {
      "name": "roof",
      "confidence": 0.10028418898582458
    }
  ],
  "description": {
    "tags": [
      "car",
      "outdoor",
      "blue",
      "ocean",
      "beach",
      "surfing",
      "water",
      "sitting",
      "cake",
      "sandy",
      "parked",
      "sand",
      "driving"
    ],
    "captions": [
      {
        "text": "a blue car on a sandy beach next to the ocean",
        "confidence": 0.9698794056125721
      }
    ]
  },
  "requestId": "deleted",
  "metadata": {
    "width": 2000,
    "height": 1500,
    "format": "Jpeg"
  }
}

[yashag2255 · Accepted solution]

Hi @JimmyWork 

You can use the filter array action for that: 

Here i am checking the confidence score greater than 0.5 and filtering the tags based on that.

Hope this Helps!

If this reply has answered your question or solved your issue, please mark this question as answered. Answered questions helps users in the future who may have the same issue or question quickly find a resolution via search. If you liked my response, please consider giving it a thumbs up. THANKS!

[RezaDorrani]

Hi @JimmyWork 

Use filter array 

for the From option point to the tags attribute in your JSON

In my case I hardcoded your JSON response in Compose action

Make sure to filter on the ['tags'] node in the response object

Next for the filter critera

Use item()?['confidence']

--------------------------------------------------------------------------------
If this post helps answer your question, please click on “Accept as Solution” to help other members find it more quickly. If you thought this post was helpful, please give it a Thumbs Up.

[JimmyWork]

So i tested to use the dynamic content from the Analyze Image. Not sure this is possible but im not getting it to filter. Can't we use the dynamic content available without the need to use JSON?

[yashag2255]

Hey @JimmyWork 

You can use the dynamic content too. I did not have the option to run the action using the analyze image so I just took the data you posted originally and then used it through parse json to get the appropriate values. 

Hope this Helps!

If this reply has answered your question or solved your issue, please mark this question as answered. Answered questions helps users in the future who may have the same issue or question quickly find a resolution via search. If you liked my response, please consider giving it a thumbs up. THANKS!

[JimmyWork]

@yashag2255 Thank you

I tested filtering using dynamic content and it works.

But im not able to filter Tag names im not really sure how to create the same output that i have today.

My output looks like this.

I filter using dynamic content but this is for Tags and not Tag names, so how can i remove tag names or filter them because i dont get a confidence score on them?

[JimmyWork]

@yashag2255 Basicly i dont know how to create a compose that looks like Tag Names result and in the same time sort on confidence score. SO i want the output to look like my compose so i can write the tags the same way.

[yashag2255]

Hey @JimmyWork 

You can filter that using the apply to each like this:

I have used the variables for reference purpose only. You can replicate using the dynamic content accordingly. 

Hope this Helps!

If this reply has answered your question or solved your issue, please mark this question as answered. Answered questions helps users in the future who may have the same issue or question quickly find a resolution via search. If you liked my response, please consider giving it a thumbs up. THANKS!

[JimmyWork]

@yashag2255 Thank you

DO you know why microsoft don't implement confidence score on Tag Names, if they do then this would be done really easy. Right now they only have Confidence score on Tags and not Tag Names, so i dont know how Tag Names are generate really?

Anyway thank you for all the help

[yashag2255]

Hey @JimmyWork 

The tags with the score has the tag names too right. So it is basically providing you with all the output and you can pick from those and use the ones that are required in the scenario you are building. I think it is generalised output and you can customize it the way you want (again depends on the scenario). 

Hope this Helps!

If this reply has answered your question or solved your issue, please mark this question as answered. Answered questions helps users in the future who may have the same issue or question quickly find a resolution via search. If you liked my response, please consider giving it a thumbs up. THANKS!

[JimmyWork]

@yashag2255 My Tag Names and my Tags is not the same well mostly but not the same. So i dont know how it generates the Tag Names and from dynamic content i can only Select Tag Name not Tag Names (Multiple) if i filter.

Result from Tags:

[
  {
    "name": "car",
    "confidence": 0.9996582269668579
  },
  {
    "name": "road",
    "confidence": 0.9992706775665283
  },
  {
    "name": "outdoor",
    "confidence": 0.9959216117858887
  },
  {
    "name": "vehicle",
    "confidence": 0.9325582981109619
  },
  {
    "name": "land vehicle",
    "confidence": 0.9007418751716614
  },
  {
    "name": "automotive",
    "confidence": 0.8912363052368164
  },
  {
    "name": "red",
    "confidence": 0.8797998428344727
  },
  {
    "name": "mountain",
    "confidence": 0.8575743436813354
  },
  {
    "name": "wheel",
    "confidence": 0.6563787460327148
  }
]

Results from Tag Names

[
  "car",
  "road",
  "outdoor",
  "red",
  "mountain",
  "driving",
  "grass",
  "small",
  "riding",
  "parked",
  "motorcycle",
  "street",
  "black",
  "side",
  "parking",
  "lot",
  "man",
  "traveling",
  "white",
  "highway",
  "surfing",
  "hill",
  "mirror"
]

[JimmyWork]

@yashag2255 

I know i accepted the solution so sorry for tagging you and asking again but i tried getting this right and i can't

There is a difference beetwen Tags and Tag Names. Tag Names don't come with a confidence score so i can't sort them.

Tags comes with a confidence score but i can't structure the data output to look like Tag Names. Im new to using JSON and so on.

What  i get from my filter looks like this.

[
  {
    "name": "bicycle",
    "confidence": 0.9999939799308777
  },
  {
    "name": "bike",
    "confidence": 0.9754800796508789
  },
  {
    "name": "bicycle wheel",
    "confidence": 0.9688829183578491
  },
  {
    "name": "outdoor",
    "confidence": 0.9576890468597412
  },
  {
    "name": "parked",
    "confidence": 0.9431015253067017
  }
]

I want the output to look like

[
  "bicycle",
  "outdoor",
  "parked",
  "leaning",
  "wheel",
  "front",
  "sitting",
  "side",
  "standing",
  "large",
  "black",
  "wooden",
  "man",
  "laying",
  "riding"
]

I tried with Initialize variable and so on as you showed but i fail to get this to work. I just want the restructure the data i get out

[JimmyWork]

@yashag2255 

I solved it by creating a Html table, not sure if this is a good way to do it?

[yashag2255]

Hey @JimmyWork 

No worries keep posting until you get the issue resolved. 

Now about the tag and tag names, if you see the screenshot, I am first extracting all the arrqay elements of a particular order (tagnames for example), then I already have a tags array that is prepopulated and so, I am using the apply to each and populating a fresh array that exist in both the arrays (makes sense?)

The solution you derived pretty much does it though!

Hope this Helps!

If this reply has answered your question or solved your issue, please mark this question as answered. Answered questions helps users in the future who may have the same issue or question quickly find a resolution via search. If you liked my response, please consider giving it a thumbs up. THANKS!