SQL Get Rows is producing duplicates

[chriscanavan58]

Hi Everyone I hope someone would be able to help. I have a flow which connects to a SQL database to extract the data from a table/view which has just short of 15000 records using Get Rows (V2) with pagination set to 20000 rows. It then creates a CSV table and then saves it to a SharePoint. 

The issue I am having is that it is returning the correct amount of rows which are in the table but it is duplicating a number of the rows sometimes up to 3 times. 

[v-litu-msft]

Hi @chriscanavan58,

Could you please share screenshots of your Flow configuration?

Please make sure there is no additional Apply to each action.

"It is returning the correct amount of rows which are in the table but it is duplicating a number of the rows sometimes up to 3 times." Is it means some rows will be replaced by duplicated rows if the amount of rows is correct, so there will be some rows loss? 

Best Regards,
Community Support Team _ Lin Tu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[chriscanavan58]

Hi @v-litu-msft  thanks for the response

here is the screen grab, as you can see there are no apply to each actions in there

[v-litu-msft]

Hi @chriscanavan58,

There seems to be nothing wrong with Flow.

Is it possible that these duplicate rows exist in the SQL table before getting via Flow?

Best Regards,
Community Support Team _ Lin Tu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[chriscanavan58]

Hi @v-litu-msft 

I have checked the source data and there are definitely no duplicates as they all have a primary key assigned to them.

[chriscanavan58]

@v-litu-msft  I have noticed on another branch where it is extracting 650 rows of the data the issue isnt present. it only seems to happen when it is pulling in a large number of rows. 

[v-litu-msft]

Hi @chriscanavan58,

OK, this issue has never met before. You can temporarily set the number of Get rows to a more secure range. If you need to deal with these records, you can use filter query in Get rows to minimize rows number. If the same situation still happens to other users in the future, and we can reproduce it on our side, we will report this issue, in time and let you know.

Best Regards,
Community Support Team _ Lin Tu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[chriscanavan58]

@v-litu-msft  limiting the rows isnt going to be possible as I require all the rows to be extracted unless you can think of another way of getting all this data?

[v-litu-msft]

Hi @chriscanavan58,

Is it an on-premise SQL instance? If it is, there is no way to do it.

If not, you could use Execute a SQL query action with SQL query to get all rows.

Best Regards,
Community Support Team _ Lin Tu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[chriscanavan58]

@v-litu-msft 

yes it is an on premise SQL and we are not allowed to use the execute SQL query 😞

[chriscanavan58 · Accepted solution]

@v-litu-msft 

I found a solution. The tables required indexing. once this was completed it resolved the issue 

[patrik_exopen]

@v-litu-msft We're hitting the same issue, Get Rows V2 via data gateway against on-prem SQL-server, pagination 100000, creates identical rows but with different ItemInternalId.

The query is also against an SQL view, what kind of indexes did you create @chriscanavan58 ?

[JatinSanghvi]

Please take this as a personal comment only as I am not representing Microsoft here.

It seems that pagination in SQL actions do not return all pages from the same point-in-time. It would be easier to explain what I mean through an example.

Let us say that a SQL table has 5000 rows. The first request to fetch SQL table rows will returns the first page with 2048 rows. The second request will simply skip the top 2048 rows, so if in the meantime, one or more rows were inserted in the first page, the rows situated in the bottom part of the first page will get shifted and reappear in the second page response as well, which will lead to row duplication.