Custom Connector - OAuth2 - Old refresh token used

[Knots]

We've created a custom connector for Exact Online, but are having issues with the access tokens. Everything works, but at some point the connection states it "Can't sign in" and you'll need to fix the connection before you can use it again. When checking the detail of that connection, it states:

Failed to refresh access token for service: oauth2. Correlation Id=..., UTC TimeStamp=..., Error: OAuth 2 access token refresh failed. Client ID and secret sent in form body.. Response status code=Unauthorized. Response body: {"error":"unauthorized_client","error_description":"Old refresh token used."}

The authentication type of the custom component is set to OAuth2. The identity provider is the Generic OAuth2 one. It uses the "Authorization Code" flow. We've tried the "Implicit" one as well to no avail.

Since the connection works initially, the authorization and token URL seem to be correct. We can authenticate and use the connection in an, e.g. Canvas App.
According to the documentation, the refresh URL is the same as the token URL:

../api/oauth2/token

If the token would expire once a month or so, I could live with it, but the token expires after just 10 minutes...

Can anyone tell me why an old refresh token is used? Is the token simply not updated after refreshing it? Is it refreshing multiple times simultaneously, in which case the second call probably results in the error?

What am I missing?

edit: added additional information regarding the authentication method.

[HEATFreight]

Here is a wayback machine link to the QBO API custom connector tutorial that I linked to above. This might take a while to load, but it has all the screenshot photos and instructions that I used. "Print to PDF" this archive page, and then you can easily reference it later.

[JOAS_Niels]

Hi @HEATFreight,

Thanks for your reply. But the link seems to refer to your custom connection for QuickBooks. I was more interested in this part of your comment:

"The other 3rd-party API does not follow the OAuth2 spec. At this point I forget exactly what the issue was, but their implementation is incorrect. And they can't easily fix it because it's a widely used API in production with thousands, possibly millions, of users. We had to deploy it via Power Automate flows. We have Parent flows and Child flows, and the Child flows can't use any "run-only user" connections. In other words, all connectors in the Child flows must be in maker context.

Our PowerApps canvas app sends requests to Parent flows which then run the Child flows to perform API requests using the HTTP action in Power Automate. We even have backend flows triggered on SharePoint edits which run the Child flows to refresh tokens and make API requests on behalf of the user for automated backend tasks.
"

I would like to create something similar for the Exact Online API, but I don't know if I'm skilled enough. I hope you can give some more details in the form of screenshots or something, so it would be easier to recreate it for our environment. Did you manage to do it without code, only with flows? 

Hope you can give me a bit of a head start. Thnx!

---

[HEATFreight]

You can do it with flows only, but I do not recommend.

Power Automate is atrocious. I have had it with their unprofessional service and downright abysmal support (I'm looking at you Tata Consultancy Services).

Slightly unrelated, but their GitHub integration within PowerApps is a joke. And the entire Power suite's ridiculously slow page loads when editing apps or flows has derailed my progress and killed my development pace time and time again. I am constantly plagued by bugs that further degrade the experience.

For instance, if you have several flows open for which you commonly need to edit the code or investigate run data, occasionally Power Automate will decide that your session is expired (presumably) and force some sort of reauthentication that fails with an "Out of storage" error (no matter whether you use Chrome or Edge, fully updated), thus causing you to lose the actual URL of that flow from your browser history within that tab (and all other focused Power Automate tabs within each browser window). And if the "Out of storage" error is still ongoing at the moment you view that browser tab, then upon navigating back to try to get back to the flow URL, your history in that tab disappears and you completely lose context to the flow which is supposed to live in that browser tab. Hope you bookmarked all your flow tabs in a folder or your screwed!

Microsoft: "Torturing users with attention deficits since 1975".

I am halfway through the process of pulling the plug on Power Automate completely. My Power Apps canvas app will call no flows when I'm done with this!

All flow logic moves to Azure Function Apps. Currently have at least three console apps running in production on Azure, although I had no idea how to write console apps before this. Basically, just converted each flow's functionality to pseudocode and asked ChatGPT to write a console app to implement that logic. It takes quite a bit of customization of the ChatGPT-written code and it's a lengthy iterative process to feed the current refactoring of the code back into ChatGPT to refactor it again until it meets your needs, but it's worth it to learn something new. Azure is great. Power Automate blows. Stay far away. Do not under any circumstances trust mission critical code to Power Automate. I mean it will probably kind a work and it's easy for simple stuff, but with ChatGPT now lowering the bar for entry-level console app development, there is simply no point in screwing around with Power Automate. I wish ChatGPT had come out three years ago and then I wouldn't be stuck with this Power Apps bull💩 too. Power Apps is only marginally more usable that Power Automate for professional, mission critical infrastructure. I fully intend to replace my entire canvas app with a Next.js webapp.

Low code was a mistake. Anybody got a time machine leaving for 2020? Planning to steal the gpt4 weights from OpenAI, as well as COVID vaccine efficacy data from Pfizer and Moderna, and travel back to February 1st, 2020. We can fix the last three years! Who's with me?

EDIT: Let me just say that the real impetus for switching away from Power Automate to Azure Function Apps was that there was little to no Power Automate training in OpenAI's models (gpt3 and gpt4). And Power Apps for that matter. ChatGPT just does not natively understand powerFx syntax (which itself is a disjointed amalgamation of syntaxes: you have the JSON data that describes a flow or canvas app in the context of, for instance, the same format used in the files within an exported Power Automate solution vs. the powerFx syntax within a canvas app vs. the powerFx syntax of the visual coding elements in a flow vs. the powerFx syntax of flow expressions).

If you try to copy the visual coding blocks (actions and other coding elements like data compose/select/filter, etc.) in a Power Automate flow, which includes any expressions embedded within, you get JSON data that looks totally different than what you see on the page, and it's quite hard to parse because it encodes all the connection data and whatnot. So, it's quite hard to just copy your logic for an iterative AI-based approach to refactoring. You can get ChatGPT to work with powerFx syntax (whether canvas app syntax or flow expression syntax or, presumably, the JSON syntax too). The problem with the full JSON syntax is you hit the limit of the context window rather quickly. But you can get ChatGPT to respect the syntax. You just have to specifically prompt it that the syntax is not: javascript, C#, C++, C, etc. Example prompt: "Do not change the syntax. Do not remove code unnecessarily. Do not make any unnecessary changes. Refactor the code to do abc123." It will handle basic stuff ok, and it would probably do pretty well if you boosted the prompt with documentation for all the Power Automate expression functions, but it does not know them last I tried. And even if it did "know" the powerFx syntax and available expressions, it would not be easy to copy over new code. You would essentially need to delete entire "scope" blocks and paste in the JSON of the refactored "scope" blocks from clipboard. It would be clumsy and slow. Not integrated with GitHub. Pretty much a joke. (Granted, they now have some kind of AI editor assistance for Power Automate, but totally useless for me because it only works for new flows, so Microsoft just says "🖕 you, have some technical debt!")

Anyway, by moving away from Power Automate to Azure Function Apps using other languages well-supported by ChatGPT training data I have massively increased the pace of my development. Power Automate will do nothing but slow you down and hinder your future progress. It was helpful to me three years ago because ChatGPT did not exist. Today I would say No Code / Low Code is dead.

Iteratively prompting ChatGPT to fix your code will quickly teach you how to code.

[JOAS_Niels]

Thanks for your answer @HEATFreight. I'm with you for the most part, but I'm not really a developer. I can handle flows for most part, but programming is another level. Learning Azure Function Apps would take far more time. Time that I don't have at the moment. We are a small company with simple integration needs, so Flows were kind of ideal. But this simple integration with Exact Online is a pain, because of the strict policies of their API.

I will dig a little bit deeper in Azure Function Apps, but I will have to outsource it then, I think. That's not what I wanted.

Can you show me your setup with the flows, non the less? Then I can make a proper evaluation for our situation.

[GrootCRM]

I build a custom connector to Exact Online that I use in flows and only expires when it is inactive for a month. So, it seems possible :-). Let's identify the variances so we can document them here. Below, you'll find a screenshot of my Security tab:

Please feel free to contact me directly if you have further questions.

Kind regards,

Elowy.

[HEATFreight]

I use Custom Connectors or Azure Function Apps to make API requests.

I can't recommend flow because it requires the same complexity as building a context app (if you want it to work right). If you do not build out support for handling of exceptions and edge cases, what's the point?

My flows to handle API requests ended up being so complicated that I don't think you would benefit from me trying to show you how I did it. And they still fail to account for the variety of edge cases that my Azure Function Apps handle.

There's no catch-all solution. You just have to read the documentation for the API you're using and attempt to match the structure of your request to the examples in the documentation.

Don't use flow. Pay for ChatGPT and ask it for help. That's literally all I did.

[JOAS_Niels]

Hi @GrootCRM,

My security tab is identical and my connection also works for about a month and then loses it's connection. Only that is not because of inactivity? Are you sure it is because of inactivity at your end? Not because of the reason mentioned in this thread?

Do you use only one endpoint of Exact Online? When I added a second endpoint for another integration, the connection failed every few hours. That is not workable in a production environment anymore. I have tried to create a second custom connector for the second endpoint, but that didn't make a difference. I will soon try to run the second connector with a different user to see if that works better, but a user for every endpoint is also not really a solution (pricey). 

As I understand it, reading through all the info, the policies that Exact Online enforces on it's API just doesn't work flawless with the custom connector.

[GrootCRM]

Hi @JOAS_Niels ,

Actually I'm not sure if the connection will remain after a month. I can tell you in a few days, when a month is expired since the last connection update 😉

I don't know what your definition of endpoint is, but we use several services from Exactonline, like  PurchaseInvoices, -lines, Contacts, Projects and more...

Cheers!

Elowy.

[JOAS_Niels]

Hi @GrootCRM,

Yes, that's what I meant with endpoints. Do you use the same custom connector and user for each endpoint? When I added the second endpoint the failures became a lot more frequent.

I'm curious how your connector will function the coming weeks.

[Piebe_Post]

How did this work out, one year later?

[JOAS_Niels]

I have not found a solution to the problem. The connector fails approximately once per month. That is, when you use the same connection for each flow. If you create multiple custom Exact connectors or use multiple connections with the same connector, then it fails within a couple of hours.

That's my experience.

The alternatives are developing a custom proxy yourself (I have not tried this yet, probably above my pay grade) or a paid service that has a connection to Exact.

[Piebe_Post]

Hi Joas, thanks for your reply.

Until a few weeks ago I had it failing approximately once a week or so. But now it fails about twice a day. Only got one connection, don't really know what happened.

This kind of s****. If I knew this beforehand, we'd probably pick another ERP, but it's too late to switch. I think I'll build an Azure Service as a proxy, which I'll authenticate with an implicit grant type.

[JOAS_Niels]

I know what you mean, it's so frustrating!

Just in case, be sure to set the connection in your flow step to the same connection everywhere.

It should also be the same connection as your test connection if you have tested it in the edit view of the custom connector itself:

This was an eye opener for me. These connections also interfere with each other.

But it is still not production environment worthy when it fails once per month.

Love to see your solution if you go the custom proxy path!

[Piebe_Post]

Thanks for your extensive reply. I only got one connection present in my environment, so I can never use different ones around multiple flows.

I suspect that the refresh chain keeps breaking because the refresh request is made "too late" from time to time, so after 10:00 minutes (say 10:02 minutes, which is 2 seconds too late). So another thing I think about is building a flow that creates a key (maybe implicit), stores the key in an Azure Key Vault and repeats this process every 9:30 minutes. Then I could get the key for each flow and put it inside the header of the Exact actions present in that flow. Might be easier. But not sure if this makes sense, didn't really overthink this scenario yet.

[GrootCRM]

Same here. My customer has to refresh the connection each month. Unfortunately...

[Piebe_Post]

Each month, at set times (so known beforehand)? Or when it randomly fails?

It's not really a solution for me. I'll try some solutions in the coming weeks.

[JOAS_Niels]

---

@Piebe_Post wrote:

Thanks for your extensive reply. I only got one connection present in my environment, so I can never use different ones around multiple flows.

I suspect that the refresh chain keeps breaking because the refresh request is made "too late" from time to time, so after 10:00 minutes (say 10:02 minutes, which is 2 seconds too late). So another thing I think about is building a flow that creates a key (maybe implicit), stores the key in an Azure Key Vault and repeats this process every 9:30 minutes. Then I could get the key for each flow and put it inside the header of the Exact actions present in that flow. Might be easier. But not sure if this makes sense, didn't really overthink this scenario yet.

---

Maybe a flow that does a dummy request every 8 minutes with the same connection would be a workaround for now?

[JOAS_Niels]

---

@Piebe_Post wrote:

Each month, at set times (so known beforehand)? Or when it randomly fails?

It's not really a solution for me. I'll try some solutions in the coming weeks.

---

I don't think it's at set times.

[Piebe_Post]

Something like that. But just making a dummy request doesn't fit the bill I'd think, since the dummy request should be able to update the refresh token and the access token that is stored somewhere in the connection settings (or whatever that's called) of Power Platform.

So I'm thinking of a solution, probably an Azure Service since a flow cannot present a login screen, that creates and maintains a connection and stores the keys inside a key Vault. I'd update the custom connector to include a header field for the access token, which will be retrieved using the Azure Key Vault connector.

Or even better, I'd like to be able to manipulate the custom connector in a way that it doesn't refresh after 10 minutes but after 9:30 minutes instead. But as far as I know, there isn't such a possiblity, but haven't looked into that yet.

[JOAS_Niels]

I don't believe the last thing is possible. 

If I'm not mistaken, with a dummy flow, the token will stay valid and you would not need to refresh it. Or am I missing something? At least for a month or so....