Custom Connector successfully performs OAuth2 authorization code flow but fails refresh token request... possible response body schema or parsing issue?

[HEATFreight]

It's a story of two APIs, one that works and one that doesn't.

Step 1: Export v2.1 Postman collection of some API requests with all requests configured to use the OAuth2 authorization code flow (grant type: authorization code). The request in the collection are merely normal interactions with the API, but NOT authorization requests or refresh token requests, as there should be no need to manually build those requests because the custom connector takes care of the authorization code flow and refresh token requests on its own in the background, invisible to the user. If you do it right...

Step 2: Import to Custom Connector and follow this guide to fix the code errors in Swagger Editor, which mostly consists of just replacing the URI encoded Postman variables, like {{CompanyID}} for instance, which get replaced with % notation in the Swagger Editor. You also have to fix the double curly brace % notation and convert to single curly brace notation, then add a parameter definition line for each Custom Connector variable that you just fixed, like {CompanyID} needs 

- {name: CompanyID, default: '1234567890', in: path, type: string, required: true}

One nuance here is that integers as strings, as in the above default value for CompanyID, require single quotes in Swagger Editor, and if you try editing in the regular custom connector editor, that notation will break and you will get an "expected string, but got integer" error. So whatever you do, "Update connector" or "Create connector" while you are still in the Swagger Editor. It will return you to the regular custom connector editor interface. DO NOT OPEN ANY OF THE REQUEST PARAMETERS FROM THE REGULAR EDITOR. Also, it helps to save the connector from the >Security tab before moving on to the >Definition tab, because if you don't you'll have to go back and redo it. Do not fear if refresh token URL is blank, that is the expected behavior when you return to the >Security tab after successfully saving the >Security settings. The client ID, client secret, and refresh token URL fields get wiped if you return to the >Security tab, but if you previously saved them, rest assured they are saved.

Step 3: With the collection imported and all settings configured, go to the >Test tab and create a connection to this custom connector. If you did it right, in the background the custom connector makes a series of GET requests and a POST request that together are called the "authorization code flow" in the OAuth2 spec. If those requests worked, you will get redirected to a login for the API service, where you will enter your user credentials and submit, and those credentials will get sent back to the redirect URI for the custom connector, which is provided for you on the >Security tab once you've created and saved the connector. The custom connector then uses the authorization code returned by the previous request and makes a POST request for the access token and refresh token. That POST request also has an expiration time and token type in its response body. Keep in mind, all of these requests so far were performed by the custom connector in the background without any user visibility except for the login popup, and also keep in mind that none of these request have to be built or should be built in that Postman collection. These requests are defined by the OAuth 2.0 configuration in the >Authorization tab in Postman, plus the credentials that the custom connector makes you re-enter in the custom connector >Security tab (since it won't pull those credentials from the Postman collection). So, that is to say the user does not write the authorization requests or the refresh token requests. The entire auth code flow and the subsequent refresh token request are all handled in the background by the custom connector itself, assuming you provide the correct settings and inputs in Postman>Authorization and connector>Security. It's magic! More on that later...

Step 4: If Step 3 worked, you will now have an authenticated session with a valid access token, so run one of your requests that you imported in your Postman collection, say GET Invoice-ReadByID or POST Invoice-Create for instance, and make sure it succeeds. Run more requests if you need to test them too, or save your connector and add it to a Power Automate flow if you like. Run the flow on a schedule such that you know it will be running just before and just after the refresh token from Step 3 is set to expire.

Step 5: If your custom connector is functioning properly, there was some hidden magic that happened at the end of Step 3. The custom connector parses the last auth code flow request, which is that POST request which returns the access token and refresh token remember, and it stores those for later use. It should store four parameters from that request:

"refresh_token":"...",
"access_token":"...",
"token_type":"bearer",
"expires_in":3600

If the access_token parameter was parsed and stored successfully as well as the token_type parameter, your test requests or Power Automate requests through this connector will succeed. If the refresh_token parameter was parsed and stored successfully as well as the expires_in parameter, your connector will keep on working indefinitely. It will internally keep its own tokens refreshed and the user will be none the wiser.

Here's where our problem rears its head! For our QuickBooks Online API custom connector, everything works perfectly. It has kept itself refreshed for at least a week now, and the QuickBooks access tokens expire every hour, so we would know within one hour if the refresh token request failed. Since the first time we learned how to configure this properly (by following that guide I linked to earlier) we have not seen the QBO connector fail a single time. Not one time. It just stays refreshed like magic.

We suspect that our custom connector which is failing to refresh its access token, which is under nondisclosure agreement (NDA), is breaking down at the end of Step 3 when that hidden magic parses the access token response body. If that magic succeeds in parsing the access token itself but fails at parsing the refresh token, then Step 5 will return an error. Step 4 will work fine, and the user won't notice any problem until the access token expires. Here is the error that our NDA API, let's call it the Example.com API, returns:

Failed to refresh access token for service: oauth2. Correlation Id=..., UTC TimeStamp=12/13/2021 11:26:20 PM, Error: OAuth 2 access token refresh failed. Client ID and secret sent in form body.. Response status code=BadRequest. Response body: {"error":"invalid_grant"}

The service we are referring to as "Example.com" is not at fault here. We are able to refresh our tokens just fine in Postman or with HTTP requests in Power Automate. It's the custom connector that is breaking.

It works with QuickBooks but fails with Example.com, but not until the refresh token request. Example.com is a very large and popular service, and their API is used by thousands if not millions of people. Their API follows the OAuth2 spec. Again, IT WORKS IN POSTMAN! Of course, we have to manually write the HTTP request in Postman that refreshes the token, because Postman does not have any functionality to automatically keep the token refreshed. But the request we make in Postman to refresh the token should be exactly the same as the request that the custom connector makes in the background, and yet only the custom connector fails, so it must not be the same request.

Thus the magic is breaking. Either it's parsing the wrong value or not parsing the refresh token at all, or otherwise generating an invalid refresh token request somehow.

The only other relevant information I can think to provide would be example response bodies from the two APIs for the auth code flow POST request that generates the access token and refresh token after the user is prompted to login. We believe that something about the response body schema is breaking the custom connector.

First, the response from the fully functional QuickBooks Online API (private data replaced with ellipses and newlines added to response body for readability:

"Response Headers": {
    "date": "Tue, 14 Dec 2021 15:04:21 GMT",
    "content-type": "application/json;charset=utf-8",
    "content-length": "1028",
    "connection": "keep-alive",
    "intuit_tid": "...",
    "x-spanid": "...",
    "x-amzn-trace-id": "...",
    "x-content-type-options": "nosniff",
    "server": "envoy",
    "cache-control": "no-cache, no-store",
    "pragma": "no-cache",
    "x-frame-options": "SAMEORIGIN",
    "x-xss-protection": "1; mode=block",
    "x-envoy-upstream-service-time": "60",
    "strict-transport-security": "max-age=31536000"
  },
  "Response Body": "{
      \"x_refresh_token_expires_in\":8726337,
      \"refresh_token\":\"AB116...\",
      \"access_token\":\"eyJlbmM...\",
      \"token_type\":\"bearer\",
      \"expires_in\":3600}"

And second, here's the response from the failing custom connector for Example.com (not really Example.com...):

"Response Headers": {
    "date": "Tue, 14 Dec 2021 15:00:16 GMT",
    "content-type": "application/json;charset=UTF-8",
    "content-length": "2185",
    "connection": "keep-alive",
    "cache-control": "private",
    "pragma": "no-cache",
    "expires": "-1",
    "server": "Microsoft-IIS/10.0",
    "set-cookie": [
      "AWSALB=...x; Expires=Tue, 21 Dec 2021 15:00:16 GMT; Path=/",
      "AWSALBCORS=...x; Expires=Tue, 21 Dec 2021 15:00:16 GMT; Path=/; SameSite=None; Secure"
    ],
    "x-powered-by": [
      "ASP.NET",
      "ARR/3.0",
      "ASP.NET"
    ]
  },
  "Response Body": "{
      \"access_token\":\"78840e...\",
      \"token_type\":\"bearer\",
      \"expires_in\":1199,
      \"refresh_token\":\"62f50...\",
      \"claims\":\"[{\\\"Name\\\":\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name\\\",\\\"Value\\\":\\\"username@example.com\\\"},{\\\"Name\\\":\\\"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"client_id\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"AccountUserId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"AccountId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"ContactSfId\\\",\\\"Value\\\":\\\"\\\"},{\\\"Name\\\":\\\"V5AccountId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"V5MainAccountId\\\",\\\"Value\\\":\\\"\\\"},{\\\"Name\\\":\\\"V5AccountUserId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"userId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"userName\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"truckCompanyId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"IsProUser\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"TruckCOId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"MAgent\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"IsMasterAccount\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"MaxSearchRadius\\\",\\\"Value\\\":\\\"500\\\"},{\\\"Name\\\":\\\"MaxLoadSearchResults\\\",\\\"Value\\\":\\\"0\\\"},{\\\"Name\\\":\\\"MaxTruckSearchResults\\\",\\\"Value\\\":\\\"1000\\\"},{\\\"Name\\\":\\\"IntegrationId\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\"LOADPOST\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"TRUCKSEARCH\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"BIDPOST\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"LANESEARCH\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"HOTPROSPECTSEARCH\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"CREDITPLUS\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"CARRIERREF\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"RATEMATE\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"PushTruckSearching\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"EnhancedNegotiations\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"LoadPostingBIN\\\",\\\"Value\\\":\\\"true\\\"},{\\\"Name\\\":\\\"urn:oauth:client_id\\\",\\\"Value\\\":\\\"...\\\"},{\\\"Name\\\":\\\".issued\\\",\\\"Value\\\":\\\"Tue, 14 Dec 2021 15:00:16 GMT\\\"},{\\\"Name\\\":\\\".expires\\\",\\\"Value\\\":\\\"Tue, 14 Dec 2021 15:20:16 GMT\\\"}]\"}"

Notice the "claims" parameter is a string formatted JSON object with backslash escape characters. QuickBooks Online also has a strange parameter called "x_refresh_token_expires_in", but I do not believe that has anything to do with OAuth2. I believe that has something to do with AWS since we see references to "x-amzn-trace-id" in the response header. QuickBooks probably has parts of their API deployed in AWS buckets.

So anyway, both response bodies contain the four parameters necessary to define a valid auth code flow POST response schema: "refresh_token", "access_token", "token_type", and "expires_in".

The only differences are the value of the timeout parameter and the unique extraneous parameters that the APIs are using, "claims" and "x_refresh_token_expires_in".

Since I do not believe "x_refresh_token_expires_in" matters in terms of OAuth2 spec, I want to focus on that "claims" parameter. Does anyone know if that long JSON string in "claims" could be breaking the custom connector? I think the "claims" parameter follows the spec, like it's not wrong to use that AFAIK, but it seems significant that the API which returns the "claims" parameter is the one that's failing.

My assumption is that the custom connector fails to parse this response correctly because of some incompatibility with the long JSON string in "claims". Notably, "refresh_token" is the last parameter in the response body before "claims". It seems extremely likely that the custom connector is failing to parse "refresh_token" due to some inability to deal with that "claims" parameter.

Any custom connector wizards out there? @RezaDorrani 

[Syndicate_Admin]

@Syndicate_Admin any updates on this?

Just to reiterate, I believe the issue is that custom connectors can't deal with triple-slash in response body!

This is probably because the Swagger Editor regards triple-slash as a comment line.

For the 3rd party API which is having issues, the response body of the background POST request that should be parsed to store the refresh token contains a "claims" property which is a JSON formatted string with triple backslash escape characters preceding each double quote inside the main string.

Does anyone know if \\\ in response body causes issues with custom connector Swagger Editor?

I'm almost 100% sure that this is the problem, but what do I know? I'm no expert, just figured out how to use APIs fairly recently. Would appreciate any words of wisdom here!

[markyboy]

I wish I could help you, Reza. As a PA newbie, I have been greatly helped by your YouTube videos and really appreciate you doing them. I am building an inventory app that collects info, snaps a picture, and then stores the data in Sql Server and creates a For Sale listing on eBay. Thanks to your videos, the storage on Sql Server works fine. However I'm struggling to build a custom connector to the eBay Trading API (the older API). I'm getting closer. At first I tried doing it from a Postman Collection, but now I'm working from scratch using the Swagger editor. When I try to create the connection, it successfully goes through the authentication process, I get the eBay successfully authenticated message, plus a message that I can safely close the authentication success screen, but the connection is not made and I get the annoying 'fix connection' link. It never actually makes the connection. Any ideas?

mark

marky@lucideq.com

[markyboy]

More correctly stated, my Custom Connector successfully performs authorization but authentication fails