Custom Connector - OAuth2 - Old refresh token used

[Knots]

We've created a custom connector for Exact Online, but are having issues with the access tokens. Everything works, but at some point the connection states it "Can't sign in" and you'll need to fix the connection before you can use it again. When checking the detail of that connection, it states:

Failed to refresh access token for service: oauth2. Correlation Id=..., UTC TimeStamp=..., Error: OAuth 2 access token refresh failed. Client ID and secret sent in form body.. Response status code=Unauthorized. Response body: {"error":"unauthorized_client","error_description":"Old refresh token used."}

The authentication type of the custom component is set to OAuth2. The identity provider is the Generic OAuth2 one. It uses the "Authorization Code" flow. We've tried the "Implicit" one as well to no avail.

Since the connection works initially, the authorization and token URL seem to be correct. We can authenticate and use the connection in an, e.g. Canvas App.
According to the documentation, the refresh URL is the same as the token URL:

../api/oauth2/token

If the token would expire once a month or so, I could live with it, but the token expires after just 10 minutes...

Can anyone tell me why an old refresh token is used? Is the token simply not updated after refreshing it? Is it refreshing multiple times simultaneously, in which case the second call probably results in the error?

What am I missing?

edit: added additional information regarding the authentication method.

[Anonymous]

I would either reach out and see if they have a configurable token lifetime, or rework the custom connector to not use the preconfigured OAuth security tab.  You will still do OAuth 2, you will just handle the refresh calls yourself like the example below.  Flow would need to store the last refresh token, clientID and clientSecret in a secure location like Azure Key Vault.  This is what is recommended for "Backend Applications" on their site.

POST ../api/oauth2/token
content-type x-www-form-urlencoded

{
refresh_token: “Gcp7!IAAAABh4eI8DgkxRyGGyHPLLOz3y9Ss …”,
grant_type: “refresh_token”,
client_id: “b81cc4de-d192-400e-bcb4-09254394c52a”,
client_secret: “n3G7KAhcv8OH”,
}

Don't forget to save the new refresh token back to Key Vault.  Subsequent calls to your custom connector for data should include the bearer token in an authorization header.   Flow will pass the token to the connector.

Key: authorization  Value: Bearer AAEAAGxWulSxg7ZT-MPQMWOqQmssMzGa…

[Knots]

Thanks for the response.

Regarding the flow to refresh access tokens:

1. As the connection for this API no longer has any security defined, how would the user initially authenticate?
2. Isn't this simply a bug in the Power Apps connections? It feels wrong to "write custom code", i.e. a flow, to refresh an authentication token, right?

[Anonymous]

1.  If this is intended as a backend service, the goal is to keep it running without user intervention.  Since you've already authenticated, you should be able to run indefinitely.  If you do need an initial authentication, you could compose that in another custom action.  You could put it into a try/catch where you try the refresh token, and reauthenticate on failure.

2.  This is a 3rd party API, so getting Microsoft to troubleshoot it for you may be sketchy.  It MAY be an issue with the custom connector, but more likely Exact Online is doing something noncompliant. 

You may have better success with asking Exact Online to troubleshoot it.  It is to their benefit to play well with Microsoft tools.  I created a custom connector for another 3rd party API, who later added their own Flow connector.  Long story short, theirs broke last week because Microsoft tightened up their specs on the JSON object, so I'm very grateful I'm still using my custom one.  It may be "wrong", but necessary in order to keep production services running.

[Anonymous]

1.  If this is intended as a backend service, the goal is to keep it running without user intervention.  Since you've already authenticated, you should be able to run indefinitely.  If you do need an initial authentication, you could compose that in another custom action.  You could put it into a try/catch where you try the refresh token, and reauthenticate on failure.

2.  This is a 3rd party API, so getting Microsoft to troubleshoot it for you may be sketchy.  It MAY be an issue with the custom connector, but more likely Exact Online is doing something noncompliant. 

You may have better success with asking Exact Online to troubleshoot it.  It is to their benefit to play well with Microsoft tools.  I created a custom connector for another 3rd party API, who later added their own Flow connector.  Long story short, the 3rd party Flow connector broke last week because Microsoft tightened up their specs on the JSON object, so I'm very grateful I'm still using my custom one.  It may be "wrong", but necessary in order to keep production services running.

[JOAS_Niels]

Hi @Knots ,

Did you find a solution? I'm also trying to make a custom connector for the Exact Online API.

Thanks!

[guido]

It is quite challenging to work around the old refresh token used issue of Exact Online. It was introduced in 2019, but only for some apps. Most apps received an exemption, but that is no longer possible. In Dutch you can find some background on https://forums.invantive.com/t/exact-online-foutmelding-old-refresh-token-used/1427.

A recent change includes that the refresh token handed out is not yet activated; it is activated on the first call. Otherwise, the previous valid refresh token remains the starting point for the single instance chain.

Things are getting worse since the implementation has some technical issues breaking the chain. A change is currently rolling out (already live in UK, DE, FR, ES and BE) that requires minimum interval between two access token and associated refresh token refreshes. Also, the lifetime of an unused refresh token is being reduced to 30 days.

I recommend writing a little proxy that does the heavy lifting and not relying on the apps stack. Exact is the sole vendor I know of that has such challenging security requirements in place using Code Grant Flow. Atlassian and some others are reducing lifetime slowly, but a single instance validity is nowhere else to be found. Even with a proxy (we offer Invantive Cloud as a proxy) it can be quite challenging.

Another alternative is to use a separate hosted environment to load data into a data platform such as an Elastic Pool (check license conditions).

Final alternative is to use Implicit Grant Flow only and automatically calculate the verification code from the TOTP-secret.

[JOAS_Niels]

Hi @guido ,

Thanks for the information. Do you mean it is possible to use Invantive Cloud as proxy between Exact Online and Power Automate?

[guido]

Maybe, never tested it. It is mainly used for Power BI, Power Query and ADF. Some use it with Google Functions.

I meant to state that it is maybe wiser to write and introduce a separate component between Microsoft and Exact which does the heavy lifting. The recent changes of Exact and deviations from common standards make it quite hard and expensive with most connecting platforms to directly connect and keep it running at a serious scale. A proxy can handle these issues such as semaphores more gracefully, whether written in some serverless-code or plain old website / service.

[JOAS_Niels]

Thanks for your answer. Writing a proxy is probably a bit over my head. I would have to dig into that. Quite annoying what Exact is doing...

[guido]

Yes, creating a proxy from scratch can be quite time-consuming, but maybe some library is available like picqer for the outgoing part to exact.

[HEATFreight]

I'm having the same issue as @Knots and none of the responses here seem to really address the issue.

I believe Microsoft either has bugs or missing OAuth2 features in its implementation of Custom Connectors.

Simply put, the OAuth2 "authorization code flow" is not possible in a Custom Connector. It honestly may have something to do with the fact that I imported a Postman (ver. 2.1) collection to define the Custom Connector, and Postman does not yet automatically refresh OAuth2 access tokens. Postman does provide the functionality to perform the initial part of the OAuth2 authorization code flow, which returns an access token and refresh token, but it does not have any capability to refresh the access token on its own. With all the OAuth2 APIs I have worked with in Postman, I had to manually build a POST request for the grant_type = refresh_token part of the OAuth2 spec. Custom Connectors built from imported Postman collections are able to perform that auth code flow (two or three GET requests and a POST request) pretty seamlessly. The redirect URI launches a login window and the session is initiated just fine. It's subsequent refreshes that I can't figure out. Part of the problem is that Custom Connectors don't like it when you try to edit the Authorization header manually. The other thing is that, as far as I can tell, Custom Connectors will only respect one type of Authorization header for every request within the Custom Connector. The refresh token request uses a different authorization type than the Authorization Code flow or normal API requests ("Bearer access_token" vs. "Basic base64_encoded_clientID_&_secret"). Thus there does not seem to be a way to make a refresh token request within a Custom Connector, not even as a standalone request. I have even tried manually updating the authorization header using Triggers, References, and Policies in the Custom Connector, but that is either not allowed or ignored.

If there is a way to get Microsoft's Custom Connector builder to perform the refresh token request, I can't figure it out. I have even tried a simple Postman collection with only one request, the refresh token request. That request works in Postman (assuming you've already done the authorization code flow and have its outputs saved as variables), but when I import that single request into the Custom Connector, the body of the request mysteriously becomes {"key": "", "type": "", "value": ""} instead of the Content-Type=application/x-www-form-urlencoded grant_type and refresh_token parameters from the Postman request.

So to sum it all up, I believe there must be some way to use the Swagger Editor within the Custom Connector to get the desired functionality. Similarly, there must be some way to build the requests within a Postman collection that will work in Custom Connectors. But I have tried and tried to no avail.

What's the point of using a Custom Connector if you have to reauthenticate every time the access token expires?

Seems totally useless.

Edit: Scratch that, I've figured it out! I exported a Postman v2.1 collection with all requests using the OAuth2 authorization method. Crucially, I left out the manual refresh token request from the collection. Postman requires you to build a manual request to keep the token refreshed because it will not do that for you even though it has that convenient "Get New Access Token" button: 

You must leave out any manual authorization or refresh requests that you might use in Postman. These are not necessary (nor will they even work) in the custom connector. So you export a v2.1 Postman collection of just the non-authentication-related requests that you want to include in your connector. Each of these needs to have a type of "OAuth 2.0" in the Authorization tab. I followed this guide.

You import the Postman collection into the Power Automate custom connector and proceed through the >General and >Security settings. Crucially, you must save the >Security settings before moving on. I never see this mentioned in tutorials, but if you don't do it, the client ID and secret as well as the Refresh URL will not be saved and you'll have to return to finish the >Security tab before you can actually "Update connector".

The interesting thing here is that no matter what you do, the Refresh URl field will wipe itself as if it did not accept your input, but rest assured this is the expected behavior.

Next tab is >Definition. The thing to remember about >Definition is that the conversion between the Swagger Editor and the Custom Connector user interface is broken. YOU CAN NOT TRUST IT TO CONVERT THE SWAGGER CODE PROPERLY! So here is what you do, follow the guide I linked above to edit the code in the Swagger Editor, and then click "Update connector" WHILE YOU STILL HAVE SWAGGER EDITOR TOGGLED ON! I never see any guides mention this part. If you switch back to the regular interface by toggling off the Swagger Editor, there is a chance something will break. You should be safe if you don't open any of the request parameters (like Path, Query, Headers, Body or any of their sub-fields), and clicking "Update connector" toggles off the Swagger Editor anyway, but definitely if you start opening any of those request parameter fields it will start breaking your Swagger code! I do think you can toggle off Swagger Editor before clicking "Update connector" and it will still work. It's mainly just opening the request parameters that begins to break the Swagger code.

One example is if you try using integers as default values for string types. The normal Custom Connector interface will error out if you give a string parameter an integer for a default value (say if your API requires a company ID # parameter). If you get this "integer as string" error, you can simply open the Swagger Editor and put single quotes around the affected default value integers. If you click "Update connector" while Swagger Editor is still toggled on, or at least without navigating back to the string parameter in question, the single quotes will remain intact and your default value will populate in the >Test tab and wherever you use the connector, like Power Automate. But the second you open the string parameter in the normal Custom Connector interface (i.e. Swagger Editor toggled off), **BAM** no more single quotes. After opening the string parameter which has an integer default value, toggle Swagger Editor back on again and you will see that the single quotes have disappeared from around the default value integer.

Off the top of my head, I'm not sure what other parts of the Swagger code breaks when you try editing the Custom Connector parmeters outside of the Swagger Editor, but suffice it to say something ain't right. Stick to the Swagger Editor and you should be fine.

I'm at T+1 day of QuickBooks Online API Custom Connector keeping itself refreshed. The OAuth2 authorization code flow and refresh tokens work great in Custom Connectors if you do it right. 

[HEATFreight]

Ah crap, I'm back again. I got the QuickBooks Online API apparently working perfectly in a custom connector, but I have another API that is returning the following error:

Failed to refresh access token for service: oauth2. Correlation Id=..., UTC TimeStamp=..., Error: OAuth 2 access token refresh failed. Client ID and secret sent in form body.. Response status code=BadRequest. Response body: {"error":"invalid_grant"}

It seems that something is happening with the refresh token request that is causing it to fail, but that happens in the background. We have no visibility into that request or the response, the OAuth2 custom connector handles that for you and it just works. Or so it did with the QBO API. This other API is more stubborn and I suspect it has something to do with the way they want their requests formatted. I'm not sure what Postman settings would affect this, but for instance these three settings in Postman's >Authorization tab seem like possible culprits:

Add authorization data to [Request URL] vs. [Request Headers]

or

Authorize using browser [y/n]

or

Client Authentication [Send client credentials in body] vs. [Send as Basic Auth header]

 

I am using the exact settings in the broken custom connector as what I used for the working QuickBooks Online custom connector, except with the base URL, auth URL, token URL, refresh URL, request URLs, client ID, and client secret all being unique to each API obviously. Other than those, everything else is exactly the same. API #2 works great for 20 minutes and then its access token expires and I get that error above^.

Any Power Platform wizards out there? @guido 

[DerekH_AU]

Hi @HEATFreight 

Wondering if your integration is still working, I seem to be having a similar issue with the refresh token not being requested/granted (unfortunately this API has a 60 day refresh validity, so it takes 2 months to test as we can't invalidate the token because we don't know what it is).

I tried to look at the article you mentioned but it was no longer on the link. I found another article with a similar name (and all the screen shots are broken which is annoying), but it seems like I won't be able to build the connector in the UI, only via raw Swagger to have it work?

Is this a bug we should log with Microsoft given it should be doing the refresh as part of the o-auth spec?

[HEATFreight]

Based on the conference we had with a Microsoft Support Engineer and the developers of a 3rd-party API, Microsoft expects 3rd-party API OAuth2 requests to behave a certain way in order for them to work in custom connectors. For instance, QuickBooks Online (QBO) follows the spec perfectly, and our QBO API custom connector has faithfully kept itself authorized for probably over a year now with no hiccups. I believe it refreshes tokens hourly.

The other 3rd-party API does not follow the OAuth2 spec. At this point I forget exactly what the issue was, but their implementation is incorrect. And they can't easily fix it because it's a widely used API in production with thousands, possibly millions, of users. We had to deploy it via Power Automate flows. We have Parent flows and Child flows, and the Child flows can't use any "run-only user" connections. In other words, all connectors in the Child flows must be in maker context.

Our PowerApps canvas app sends requests to Parent flows which then run the Child flows to perform API requests using the HTTP action in Power Automate. We even have backend flows triggered on SharePoint edits which run the Child flows to refresh tokens and make API requests on behalf of the user for automated backend tasks.

It was a bit of a nightmare getting the Power Automate implementation of the 3rd-party API working well, but now it works perfectly. This 3rd-party API needs tokens refreshed every 20 minutes, and our integration has faithfully kept our tokens refreshed (on demand rather than running every 20 minutes) for almost as long as our QBO custom connector.

It's probably a little higher latency than a custom connector, but it works.

TL;DR: If your 3rd-party API does not perfectly follow the OAuth2 spec, there seems to be no way to configure a custom connector to keep the tokens refreshed.

In the case of our 3rd-party API that was having issues keeping tokens refreshed within a custom connector, I believe it had something to do with the 3rd-party API not allowing clientId and secret to be included in the body of the refresh token requests. In other words, Microsoft's (correct) implementation of the OAuth2 spec in custom connectors includes the clientId and secret in the body of refresh token requests, but the 3rd-party does not expect that and actually fails if you do it. The 3rd-party API fails when the refresh token request is done properly according to the OAuth2 spec!

I tried getting around this using the limited capabilities of Actions/Triggers/References/Policies in the custom connector Definition, but I was unable to make it work after significant efforts. Additionally, due to the opaque nature of the authorization and refresh token requests made by the custom connector backend, it's very hard to understand how any given changes affect the structure of the requests. It's basically spray and pray. But with an OAuth2 API that requires tokens to be refreshed every 2 months, you essentially have no ability to iterate and try different things, so it's unlikely you'll be able to figure out how to make it just work. My understanding is that it's simply not possible. There is of course the Code section in the custom connector editor, but again my understanding is that neither Actions/Triggers/References/Policies or Code are allowed to mess with the OAuth2 token requests. That leaves the possibility of using "no authentication" in the security tab, but again, I don't think you can "roll your own" OAuth2 token requests.

One would think you could make a custom connector whose only requests are to authorize and refresh tokens, but that's not allowed as far as I know.

As a workaround, you can, however, follow these steps:

• Use Azure API Management to stand up an API (called a Frontend API) that will act as a proxy between your custom connector and the 3rd-party IDP.
• Have that Frontend API accept our token refresh format as an input for an Operation.
• Have that Operation grab the refresh_token and grant_type from our request.
• Issue a request to the 3rd-party IDP using the refresh_token and grant_type to get a new access token.
• Return that access token along with all of the other returns from 3rd-party IDP as the response on the Operation.
• Tell the custom connector to use your APIM endpoint as the token refresh URL.

This workaround was the solution that the Microsoft support engineers came up with for me. I declined to implement it myself because I did not think it would be any faster in terms of request latency than doing the token requests in Power Automate.

¯\▁(◉◡◉)▁/¯

[DerekH_AU]

Hi @HEATFreight,

Many thanks for the comprehensive reply. I think I might be in the same category as your quickbooks connector, where the API is compliant, but the UI of the power automate connector configuration screws it somehow so that it can't do token refreshes, so I may need to go down the same path you did in generating/using swagger only for it.

The API I am talking to does have an OpenAPI v3.0.0 spec which annoyingly has to be converted to Swagger v2 to be digested by power automate. I think I might press on... even if I have to have a connector to help generate the swagger for new end points, then keep the existing connector only edited by swagger editor from now on.

[HEATFreight]

Paste your swagger code and I can tell you if it's right. Try to limit it to one request. If you can make one request work and stay authorized, then you can add others later. Or follow the same structure to modify it for the other requests.

The guide I linked to (whose link is broken now) showed how you have to modify the swagger code that you get after importing from Postman. I can compare yours to mine and probably show you how to fix it.

I exported a Postman collection and imported that into the custom connector, then edited the swagger to fix the parameters and that was it. Real super easy.

[GrootCRM]

Hi @Knots ,

I'm curious to know if you managed to establish a successful connection to Exact Online using a custom connector. I find myself in the same situation, as refreshing the token at Exact Online isn't functioning as expected.

Thanks!

Elowy 

[JOAS_Niels]

I've got a working connection, but it breaks down often. Then I have to fix the connection. I first had only one request and then it failed maybe once a week. But now I have added a second request and now it breaks down every half an hour or so.

I have now created a second custom creator for the second request, but that does not fix the problem, unfortunately. It's not really production worthy like this.

I don't know how to solve it either.

EDIT: BTW my error is somewhat different:

Failed to refresh access token for service: oauth2. Correlation Id=1f4a927c-3596-4086-b252-0a2e3a9daf49, UTC TimeStamp=8/31/2023 12:14:50 PM, Error: OAuth 2 access token refresh failed. Client ID and secret sent in form body.. Response status code=BadRequest. Response body: { "error":"invalid_grant","error_description":"Token is not allowed, because of invalid or empty chainId" }

[JOAS_Niels]

Hi @HEATFreight ,

Can you maybe show us the flows you used to do the refresh manually through Power automate. I'm thinking about building this too, because it is not workable with the custom connector.

Would be greatly appreciated!