Get Items for SharePoint list is returning no data

[Thelearner11]

I just had a question about an issue I am facing:

I am attempting to return some items from a SharePoint list for a flow on Power Automate. Currently, I'm attempting to retrieve the data for a specific date, and the data amount is about 4000 items for that date. The full list has maybe about 30-40 thousand items in the list. I am using an ODATA filter query in the Get items connection that filters by date, so for this specific day, there are about 4000 items to retrieve.

However, the get items returns no data in its output. I have checked the SharePoint list to confirm data exists for this date, and it does. So I was wondering what the issue may be here. Thanks.

The ODATA filter query works with other dates I input, just some dates it doesn't work with. Another observation is that the ODATA filter query seems to work when the date has around 100 items, but not when it has about 4000 items. Does that play a role? And if it does, what is a possible workaround for this? Thanks.

[v-liwei-msft · Accepted solution]

Hi @Thelearner11 

This issue occurs because you have too many items(By default Get items only returns 100 items).

You can enable pagination and set it to 100000 (maximum items returned).

I think this link will help you:

How to get more than 5000 items from SharePoint Online in Flow - Alex Tofan's blog

Best Regards,

Levi

[Thelearner11]

Hello Levi thanks for the response. In this case, I am only retrieving about 3500-4000 records. Is there a limit on how many records one can retrieve without pagination? Or is the limit based on how many records one has in their sharepoint list as a whole?

[Thelearner11]

This worked marvelously! Thank you so much. Now I am able to set it to about 4500 items in the filter, and it retrieves the items for the date. You're a genius 🙂 Are there any limitations to Pagination that I should know about?

[Thelearner11]

Also just one more question: What happens when the SharePoint list exceeds 100k rows, does the pagination first use the ODATA filter query on the SharePoint list, and then return a maximum of 100k rows on the filtered list? Or does it get the first 100k rows, and then filter those records and return the filtered list?

[v-liwei-msft]

Hi @Thelearner11 

Pagination is limited to an upper limit of 100k.

When it exceeds 100k, flow will only get the first 100k and skip the subsequent items.

Best Regards,

Levi

[Thelearner11]

Hello Levi, thanks for the explanation. I assume pagination retrieves the first 100k in the entire sharepoint list regardless of what the filter query is, right? Also, I am attempting to use a column name in the odata filter query for one of the sharepoint lists with whitespace using _x0020_ however it is still mentioning that the column doesn't exist. Is this feature supported in Get Items for sharepoint lists?

[v-liwei-msft]

Hi @Thelearner11 

In the filter query, flow will only filter the first 100k items returned.

If you have spaces in your column names, please ignore the spaces in the filter query.

Best Regards,

Levi

[Thelearner11]

Sorry, I may have phrased the question incorrectly. Currently:

It limits me to 5000 items it seems as the pagination threshold for it. But in that case, it has returned the data in the list still and applied the date filter query. Would that indicate that it is first filtering the SharePoint list, and then the pagination returns the items only in that date? Because I have about 30-40 thousand records in the SharePoint list and it returned the exact 3000-4000 items I need specifically for that date, unless I got lucky and the first 5000 rows that the pagination retrieved from the SharePoint list happened to be the items for that date by chance.

Edit: I'm asking this because if the list were to increase to over 100,000 items, then I would like to know if it would still retrieve all the correct 3000-4000 items for a specific date in the filter query and face no issues doing so no matter how large the list is; otherwise I guess Power Automate cannot handle lists that big and it would have to be done manually.

[v-liwei-msft]

Hi @Thelearner11 

I think I get what you want.

I want to explain to you that the Get Items action will always capture all the records that fits the filter query but return only 100 of them at max.

For example, you have 200 records that Title equals 1, but the final result will always return 100 of them if you set the threshold as 100 no matter you enable the Pagination or not.

Best Regards,

Levi

[Thelearner11]

I see. So if I understand correctly:

Get Items will apply the filter query to the entire sharepoint list no matter how large the size, but will only return the number of those filtered items stated in the Pagination threshold, which in my case I have set the threshold to 5000, so it will return up to 5000 of those filtered items. --- or is the threshold option and the pagination option in the settings separate from each other?

I just want to say that I apologize if these are too many questions, I am just trying to wrap my head around this. I very much appreciate you taking the time and effort to answer my questions and help me understand :).

[v-liwei-msft]

Hi @Thelearner11 

Yes, your understanding is absolutely correct.

Best Regards,

Levi

[Thelearner11]

Thanks so much Levi, I appreciate it. I was just asking all of these questions as I was worried that if the size of my SharePoint list reached above 100k items that the flow would fail, but I am relieved to hear that the filter query will run on all the SharePoint items no matter the size, but it only matters how many items the filter query will return, and in my case it should never really return more than 3000-4000 or so after being filtered for a date, which is well within the threshold for amount of items to return.

I have faced another issue however, and that is that some of the columns are returning null values even though I have specifically checked that they contain a value. Could this be due to naming conventions used for columns, or the actual value itself? The column is a single line of text column.

[v-liwei-msft]

Hi @Thelearner11 

We really understand that you have more than one issue need to be solved today and we are surely happy to help you with those, but to better focus on the new problem and ensure the service quality, we would advise you to create another ticket on the issue.

It can also help us to get the case better recorded and let Microsoft know what you have been through clearly. Thank you so much for your understanding. 

Best Regards,

Levi

[Thelearner11]

Sounds good, will do 🙂 Thanks for all your help.