Build your very own LEGO Inventory app using PowerApps, Flow and Microsoft Cognitive Vision Services

[darogael]

Recently, I’ve been intrigued by Microsoft Cognitive Vision services. This Machine-Learning functionality can extract rich information

from images to categorize and process visual data – and machine-assisted moderation of images to help curate your services. Now, as

some of you may already know, I’m a huge LEGO fan and I come from a family whose been playing with LEGOs for three generations.

One of the struggles each LEGO fan faces is identifying a part number. I took on this struggle as a challenge and have come up with a

Proof Of Concept using Microsoft PowerApps and Flow’s integration with Microsoft Cognitive Vision Services’ API. I introduce to you

my LEGO Inventory App.

Before I started building the app I had to find an Application Programming Interface (API) which would let me access a LEGO inventory

to gather information I need. Fortunately one of my personal favorite websites, Rebrickable.com already provided this API.

Rebrickable.com provides a number of API/Web Services to assist developers build their own websites or apps which use the

Rebrickable database. To be able to use these, you must first Generate an API Key which is free but does require registration

My first iteration of this app involved me using Cognitive Vision's Optical character recognition (OCR) which would detect the LEGO

part numbers extract the recognized numbers into a machine-readable character stream. This gave me the flexibility of taking photos

of text instead of copying to save time and effort. However, I did run into a road-block. You see the part numbers in the LEGO blocks

are so tiny that it was challenging to take a zoomed-in picture. Hence I added an option to type in a part number as well but I did create

some modified blocks to test the OCR feature.

This video demonstrates how the first iteration of this app works

LEGO Inventory App using PowerApps and Flow

The first iteration works well, however, there were already two apps available which could do the exact same. Rebrickable Shopper

by Rebrickable is a mobile app where you simply scan the bar code on a LEGO® box and see a wealth of information to help you decide

if you should buy it. The Brick Data by Jean-Michel Agresti is an iPhone app perfect to browse LEGO Sets on your mobile device, old and

new. With these two existing apps already available any solution that I build would have to that major WOW factor, something that

makes it easier. This is when Cognitive Custom Vision Services had my complete and undivided attention. The simple concept of upload,

train and evaluate is brilliant. In addition, through it's active learning images evaluated through your custom vision model become part

of a feedback loop you can use to keep improving your classifier. Hence I decided to pursue that path and came up with my LEGO

Inventory App 2.0

This video demonstrates how the 2.0 iteration of the app works

Microsoft PowerApps with Cognitive Custom Vision

From a design and build standpoint the difference between OCR and Custom Vision was the change in the FLOW.

The below screenshot displays how the OCR and Custom Vision action is called and how it's values are us

Currently Custom Vision is in preview hence there are some limitations but the overall P.O.C. was very successful.

URL:

https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/

https://azure.microsoft.com/en-us/services/cognitive-services/custom-vision-service/