Dataflows missing from list (but still exist and run)

[LOanaSI]

Hi!

I have a few dataflows that are set to run automatically. After changing the refresh settings for a couple of them, I have noticed that they are no longer visible in my Power Apps Dataflows listing. I have not made any other changes to them, and this hasn't happened before.

The flow is V1 (currently unable to move them to V2) and it still exists and is running. I have accessed the Data Integration link https://admin.powerplatform.microsoft.com/ext/dataintegration/project/{dataflow_id} and I can view the dataflow information and run history.

Thanks in advance for any tips or ideas on how to fix this!

[AndrewElans]

One possible way to review which dataflows you have is go to Solutions => Standard Solution (default) => type in the search field of objects "dataflow" => select table "Dataflow".

You will typically have 2 copies of each dataflow (see my dataflows on the attached pic). So I guess when a dataflow is created, Copy 1 is created first, and Copy 2 is created some minutes later. The latter is used for actual updates as Refresh history shows.

In the blue box, I have only one copy. This dataflow "...DEPREC..." was deprecated and deleted from my environment. It is still sitting in my custom solution, which is why it is present here... 

The Dataflow table also has a column called "Mashup Setting" which has actual code running in the dataflow, so this can enlighten on which copy is doing what.

[tfraser]

Yes I also noticed the duplication in the Dataflows table.

In my scenario I'm working with dataflows not yet published, but just imported, into a new environment.

When you first import the dataflows to an environment, or create it in an environment without publishing, they only appear once per dataflow in the dataflow table, but when they are first published and run they appear a second time with refresh history.

My hunch is that the copy without refresh history could be holding the draft version, for example if you start editing a dataflow but don't publish it, since you are able to return to editing that flow without losing your changes, and you can run the previously published version without your edits having an effect. I haven't confirmed this by checking for corresponding differences in the mashup value in the table rowr though.

It's just odd that we can 'delete' the dataflow using the normal user interface list, which effectively makes them invisible, but there's nothing in the dataflow table which indicates that any individual one should be hidden (although, as you say, if one is published/run and therefore duplicated in the table, you can see that a deleted one is not duplicated).

So I wondered if there's another table or set of tables that controls the visibility of objects in the PowerApps / Dataverse user interface which is separate from what is available in the Dataflow table?

[AndrewElans]

That's an interesting question. I think since the whole concept of these apps is "low-no-code", much is hidden behind the curtains and not explained anywhere in the documentation. So we have to dig ourselves to get answers...

Offtopic: I have ditched all the standard views/forms in the Power Pages portal (don't like the "no code" thing),  connected the MSAL library, build custom UI and now use the Dataverse web api functionality at full scale (batch requests, return=representation and much more...) which gives much more flexibility

[tfraser]

I agree, the low-code thing is very attractive at first but it quickly becomes a hindrance when it obscures the underlying mechanism of something you need to understand. And often you need to fall back on code or code-like development which probably takes longer on Power Apps than it would using other platforms.

I haven't come across the MSAL library. I'm sure it would be very useful however we don't have the resources right now to create a UI for this purpose. Have you published any of your work by any chance?

[AndrewElans]

I'm developing an internal procurement portal with a company's supplier base, and the only allowed solution was to build something with power pages. But I have plans to make a public copy for further reference with some dummy data. 

@tfraser, please update here if you found answers regarding the dataflow question discussed above. 

[calechko]

@AndrewElans I did the same thing. I came up with a web template that allows me to hook into a Node.js application with a React front end and surface my own react components (with fluent UI) on any page that uses my web template using a simple routing protocol. I can still securely execute web API requests on behalf of the user, or in an elevated manner using my node back-end (delegated calls to web APIs secured with Azure app registration) and all of the code can be managed and deployed with Azure DevOps pipelines. I found this to be way more beneficial than trying to navigate through the low-code options which were not sufficient for developing large custom portals. Sounds like you found a similar approach.

[tfraser]

That sounds really useful - are you able to share any of the work you've done? I appreciate it's probably taken a lot of effort for you to just give away but it would save a load of time for others to be able to use it instead of developing their own from scratch!

[AndrewElans]

FYI I have started to share some notes on implementation of Power Pages Single Page Application in vanilla js here https://dev.to/andrewelans/login-redirect-in-power-pages-spa-with-hash-and-query-params-4o5

[calechko]

@AndrewElans Thanks for sharing. I'm curious to see if you were able to use the msal library to make Web API calls (or even Graph calls) using the token of the currently logged in user without using the standard Portal API endpoints. I've found that the Portal API endpoints are a bit limiting and slower than standard Dataverse API calls, but I'd like to maintain the security by using the currently logged-in user's token to make the calls. Can't wait to see the rest of the posts about this!

[AndrewElans]

@calechko , I use MSAL which I connected to Power Pages to send Web API queries to Dataverse and Azure Storage.  For each service MSAL is obtaining a separate token with one of the MSAL methods (acquireTokenSilent, ssoSilent, acquireTokenPopup or acquireTokenRedirect). Could you please advise what do you mean under token of the currently logged in user? Which token is that?

ID token provided by the identity provider in Power Pages cannot be used elsewhere as they write here. However, there is one interesting thing I noticed if you have Dynamics CRM -> user_impersonation added as API Permission in app registration in portal.azure.com. When you are logged in, you may find in Application Cookies in Dev Tools a cookie called "MakerToken" and this is exatly the one to query Dataverse (I tested), so this is already available without MSAL. This cookie cannot be taken with javascript, but you can get it with liquid using {{ request.params }}. However, this cookie is out of control (for example it's not set in inkognito mode) so I couldn't rely on it and therefore had to connect MSAL separately.

P.S. We can connect on teams by the way to share our experiences.