GET Restful API - Loop through offset requests

[TesDA]

I have a Restful API request using the HTTP Step:

Method - Get

Authentication - Raw

Value - API key

It picks up everything on the first call correctly, but I dont know how to create the loop so that it continues until there are no more records.

Below is the JSON returned from the initial request (from the first set of records). How do I create a variable and loop through so that it uses the "next" in the JSON until "next" returns null?

{"count":7816,
"next":"https://blackedoutname.endpointname.com/api/v1/something/?_limit=4000&_offset=4000&include_custom_fi...",
"previous":null,
"results":[
{
"id":"9172391872397a",
"jurisdiction":{
"id":"kajhdkjhasdkjhb",
"name":"ABCDEF",
"slug":"ABCDEFno"},
"organization_person":{
"id":"kahdkjahdh",
"custom_fields":{},
"person":{
"id":"kajshdkhd",
"username":"65432","first_name":"Nate",
"last_name":"Smith",
"email":"Nate.Smith@something.com"}},
"is_primary":false,
"status":"active",
"in_compliance":false,
"start_date":"1995-01-01",
"end_date":"1996-12-31",
"extended_date":"1997-12-31",
"earned":5765.0,
"applied":5740.0,
"overall_deficit":1000.0,
"overall_applied":5740.0,
"carry_over":0.0,
"license_id":"",
"latest_annual_period":{
"id":"kasdkjahsdh",
"in_compliance":false,
"start_date":"1997-01-01",
"end_date":"1998-12-31",
"extended_date":"1999-12-31",
"earned":0.0,
"applied":0.0,
"overall_applied":0,
"overall_deficit":1000.0,
"carry_over":0.0}}]}

[TesDA]

Additional information added to the summary

[yeamans]

'm having a similar issue. I have 100s of records with a system, however I can only call 100 at a time. I need to figure out how to loop through the pages. 

I do get presented with s string "record": 857

Would i need to create a variable based upon dividing the total records by 100 then rounding up the next whole number? 

[TesDA · Accepted solution]

I finally figured this one out. In the returned JSON is a next, with the next http statement for the next download of json. I created a variable to pick up the "next" from the parsed JSON and then did a do until loop until next = null:

Step 1 - Initialize Variable

"next" is initially set to the initial http address.

Step 2 - Do Until loop

I set the count to 4 (it was defaulted to 60), since I know the API should only need to loop 2-3 times.

[Automate123]

Hi @TesDA 

I have a similar issue when trying to call Jiras REST API. I am only getting back a maximum of 100 results. I am trying to apply your logic into my current flow.

Could you share what is inside the append to array variable connector please?

Thanks

[yeamans]

@Automate123,

Here's what I ended up doing to get it to work. 

Larger Screen Shots: 

Condition expression is: 

length(body('Get_Items')?['items'])

Let me know if you want help. Happy to jump on a screen share to walk through it. 

[Automate123]

Hey @yeamans 

Apologies for the late reply I went out and just got back.

Thank you very much for sharing your solution. From first glance it is beginning to make more sense as to how I can achieve this.

I am going to try and apply the logic you used into my flow now.

In regards to your offer of a screen share that would be much appreciated, If I struggle to get it working I will send you a DM to try and set that up with you.

Thanks again, I have been trying to find a solution for weeks now