How to connect Google Dialogflow with Microsoft Flow (without coding)

[Elowy]

Everyone who has experimented with Google voice dialogs might experience that there is no out of the box way to connect the Google dialogs with automation software. There are a lot of programming opportunities in Dialogflow, but building a automation solution without coding seems to be difficult.

I took the challenge to connect Dialogflow with Microsoft Flow. From there, it is easy to build a solution that interacts with data or other functionalities. For example; make through your Google Home a reservation in a restaurant, get information about the latest support call or send a white paper that is requested in the voice dialog. That could all be possible if the voice dialog could be connected to Microsoft Flow automation software.

In this blog I dive into connecting both worlds. The blog does not explain how to make a Dialogflow dialog, nor how to retrieve (for example) data from Powerapps within Microsoft Flow.

Disclaimer; I’m not a Dialogflow expert, just curious.

Fulfillment webhook in Dialogflow

Dialogflow provides a “fulfillment” functionality that enables the interaction with external systems. That can be performed with the use of code, or by using a default webhook. For our (no coding) solution we will use this default webhook.

Dialogflow uses “Intents” to define the response on certain questions. The logic of this intent -what should be the response to a question-, is defined within that same intent. When Fulfillment is enabled for the intent, the logic will be processed by an external system. Dialogflow sends the request (and variable that are collected during the dialog, called “Entities”) to the external system by using the webhook. The webhook processes the question and variables so it can respond back to Dialogflow.

Only one webhook can be defined in Dialogflow, so requests for all fulfillment enabled webhooks are send to the same interface.

In the following diagram the above explained:

And this is what the Fulfillment request and response look like:

It is easy to find out what JSON call is (or will be) send to the webhook. In Dialogflow you can try the dialog you created and during the intent where fulfillment will be enabled, select “Diagnostic info” in the bottom right of the screen.

Enable Fulfillment in Dialogflow

Enabling Fulfillment in general can be done using the fulfillment menu entry:

The URL that should be provided over here is defined by Microsoft Flow, later on in this tutorial.

Per Intent for which fulfillment should be enabled:

The option “slot filling” is being used when fulfillment is also used for processing the filling of variables in the intent that are not filled instantly. So when you have for example a mandatory variable “Time of reservation” in your Intent, and that variable is not filled directly. The question to the user can be defined by Dialogflow, or by the external system (Enable webhook for slot filling).

Logic in Microsoft Flow

Luckily Microsoft flow has the ability the create an API webhook in Azure that remains online and send all requests to the defined flow. This is done by the trigger “When a HTTP request is received”, in combination with the “Response” action to send back the response.

Because there will be only one flow (and one webhook) for all fulfillment enabled Intents, Microsoft flow has to determine from which Intent the request is coming first. From there it can define a response to the request and send that back.

So, the flow will be like:

Step 1: When a HTTP request is received

To enable the webhook API, this trigger may be used. Upon saving the trigger, a HTTP post URL in Azure is created that is connected to this trigger. This is the URL that has to be copied to the Dialogflow general fulfillment settings.

The JSON schema in “When  a HTTP request is received” can be generated with a sample payload. This payload can be generated by testing the Dialogflow in Dialogflow and at the end of the fulfillment intent select the button “Diagnostic info” in the bottom right. From there the sample payload can be copied.

But be careful; if you want Microsoft Flow to handle more than one fulfillment action, the JSONs can be different with various variables. In that case you should merge both fulfillment schemas so all variables are included.

In my use case the merged variables in the parameters section of the JSON schema look like:

Phonenumber in this example is used in another dialog than NumberPersons.

Step 2: Switch based on the Intent

To determine what Intent is requesting the flow, I use the JSON property “displayName”:

From there is easy to define the specific automation of an Intent in one of the branches of the Switch.

Step 3: Do some magic

Based on the Intent and the needed automation, the magic varies ofcourse. This is an example of my testcase where opening hours of a restaurant are requested from Powerapps (the CDS):

Step 4: Send a response back to Dialogflow

In the last step Microsoft Flow sends back a response to Dialogflow so the response from the logic can be given back to the Dialogflow user.

In this example only text is give back (saved in the variable “Response”). There are more responses possible in Dialogflow like images (not in a voice dialog of course) or cards.

Thanks for reading!