I just had a question about an issue I am facing:
I am attempting to return some items from a SharePoint list for a flow on Power Automate. Currently, I'm attempting to retrieve the data for a specific date, and the data amount is about 4000 items for that date. The full list has maybe about 30-40 thousand items in the list. I am using an ODATA filter query in the Get items connection that filters by date, so for this specific day, there are about 4000 items to retrieve.
However, the get items returns no data in its output. I have checked the SharePoint list to confirm data exists for this date, and it does. So I was wondering what the issue may be here. Thanks.
The ODATA filter query works with other dates I input, just some dates it doesn't work with. Another observation is that the ODATA filter query seems to work when the date has around 100 items, but not when it has about 4000 items. Does that play a role? And if it does, what is a possible workaround for this? Thanks.
Solved! Go to Solution.
This issue occurs because you have too many items(By default Get items only returns 100 items).
You can enable pagination and set it to 100000 (maximum items returned).
I think this link will help you:
How to get more than 5000 items from SharePoint Online in Flow - Alex Tofan's blog
Best Regards,
Levi
This issue occurs because you have too many items(By default Get items only returns 100 items).
You can enable pagination and set it to 100000 (maximum items returned).
I think this link will help you:
How to get more than 5000 items from SharePoint Online in Flow - Alex Tofan's blog
Best Regards,
Levi
Hello Levi thanks for the response. In this case, I am only retrieving about 3500-4000 records. Is there a limit on how many records one can retrieve without pagination? Or is the limit based on how many records one has in their sharepoint list as a whole?
This worked marvelously! Thank you so much. Now I am able to set it to about 4500 items in the filter, and it retrieves the items for the date. You're a genius 🙂 Are there any limitations to Pagination that I should know about?
Also just one more question: What happens when the SharePoint list exceeds 100k rows, does the pagination first use the ODATA filter query on the SharePoint list, and then return a maximum of 100k rows on the filtered list? Or does it get the first 100k rows, and then filter those records and return the filtered list?
Pagination is limited to an upper limit of 100k.
When it exceeds 100k, flow will only get the first 100k and skip the subsequent items.
Best Regards,
Levi
Hello Levi, thanks for the explanation. I assume pagination retrieves the first 100k in the entire sharepoint list regardless of what the filter query is, right? Also, I am attempting to use a column name in the odata filter query for one of the sharepoint lists with whitespace using _x0020_ however it is still mentioning that the column doesn't exist. Is this feature supported in Get Items for sharepoint lists?
In the filter query, flow will only filter the first 100k items returned.
If you have spaces in your column names, please ignore the spaces in the filter query.
Best Regards,
Levi
Sorry, I may have phrased the question incorrectly. Currently:
It limits me to 5000 items it seems as the pagination threshold for it. But in that case, it has returned the data in the list still and applied the date filter query. Would that indicate that it is first filtering the SharePoint list, and then the pagination returns the items only in that date? Because I have about 30-40 thousand records in the SharePoint list and it returned the exact 3000-4000 items I need specifically for that date, unless I got lucky and the first 5000 rows that the pagination retrieved from the SharePoint list happened to be the items for that date by chance.
Edit: I'm asking this because if the list were to increase to over 100,000 items, then I would like to know if it would still retrieve all the correct 3000-4000 items for a specific date in the filter query and face no issues doing so no matter how large the list is; otherwise I guess Power Automate cannot handle lists that big and it would have to be done manually.
I think I get what you want.
I want to explain to you that the Get Items action will always capture all the records that fits the filter query but return only 100 of them at max.
For example, you have 200 records that Title equals 1, but the final result will always return 100 of them if you set the threshold as 100 no matter you enable the Pagination or not.
Best Regards,
Levi
I see. So if I understand correctly:
Get Items will apply the filter query to the entire sharepoint list no matter how large the size, but will only return the number of those filtered items stated in the Pagination threshold, which in my case I have set the threshold to 5000, so it will return up to 5000 of those filtered items. --- or is the threshold option and the pagination option in the settings separate from each other?
I just want to say that I apologize if these are too many questions, I am just trying to wrap my head around this. I very much appreciate you taking the time and effort to answer my questions and help me understand :).
Thanks so much Levi, I appreciate it. I was just asking all of these questions as I was worried that if the size of my SharePoint list reached above 100k items that the flow would fail, but I am relieved to hear that the filter query will run on all the SharePoint items no matter the size, but it only matters how many items the filter query will return, and in my case it should never really return more than 3000-4000 or so after being filtered for a date, which is well within the threshold for amount of items to return.
I have faced another issue however, and that is that some of the columns are returning null values even though I have specifically checked that they contain a value. Could this be due to naming conventions used for columns, or the actual value itself? The column is a single line of text column.
We really understand that you have more than one issue need to be solved today and we are surely happy to help you with those, but to better focus on the new problem and ensure the service quality, we would advise you to create another ticket on the issue.
It can also help us to get the case better recorded and let Microsoft know what you have been through clearly. Thank you so much for your understanding.
Best Regards,
Levi
Sounds good, will do 🙂 Thanks for all your help.
Dear Community Members, We'd like to let you know of an upcoming change to the community platform: starting July 16th, the platform will transition to a READ ONLY mode until July 22nd. During this period, members will not be able to Kudo, Comment, or Reply to any posts. On July 22nd, please be on the lookout for a message sent to the email address registered on your community profile. This email is crucial as it will contain your unique code and link to register for the new platform encompassing all of the communities. What to Expect in the New Community: A more unified experience where all products, including Power Apps, Power Automate, Copilot Studio, and Power Pages, will be accessible from one community.Community Blogs that you can syndicate and link to for automatic updates. We appreciate your understanding and cooperation during this transition. Stay tuned for the exciting new features and a seamless community experience ahead!
We are excited to announce the Summer of Solutions Challenge! This challenge is kicking off on Monday, June 17th and will run for (4) weeks. The challenge is open to all Power Platform (Power Apps, Power Automate, Copilot Studio & Power Pages) community members. We invite you to participate in a quest to provide solutions to as many questions as you can. Answers can be provided in all the communities. Entry Period: This Challenge will consist of four weekly Entry Periods as follows (each an “Entry Period”) - 12:00 a.m. PT on June 17, 2024 – 11:59 p.m. PT on June 23, 2024 - 12:00 a.m. PT on June 24, 2024 – 11:59 p.m. PT on June 30, 2024 - 12:00 a.m. PT on July 1, 2024 – 11:59 p.m. PT on July 7, 2024 - 12:00 a.m. PT on July 8, 2024 – 11:59 p.m. PT on July 14, 2024 Entries will be eligible for the Entry Period in which they are received and will not carryover to subsequent weekly entry periods. You must enter into each weekly Entry Period separately. How to Enter: We invite you to participate in a quest to provide "Accepted Solutions" to as many questions as you can. Answers can be provided in all the communities. Users must provide a solution which can be an “Accepted Solution” in the Forums in all of the communities and there are no limits to the number of “Accepted Solutions” that a member can provide for entries in this challenge, but each entry must be substantially unique and different. Winner Selection and Prizes: At the end of each week, we will list the top ten (10) Community users which will consist of: 5 Community Members & 5 Super Users and they will advance to the final drawing. We will post each week in the News & Announcements the top 10 Solution providers. At the end of the challenge, we will add all of the top 10 weekly names and enter them into a random drawing. Then we will randomly select ten (10) winners (5 Community Members & 5 Super Users) from among all eligible entrants received across all weekly Entry Periods to receive the prize listed below. If a winner declines, we will draw again at random for the next winner. A user will only be able to win once overall. If they are drawn multiple times, another user will be drawn at random. Individuals will be contacted before the announcement with the opportunity to claim or deny the prize. Once all of the winners have been notified, we will post in the News & Announcements of each community with the list of winners. Each winner will receive one (1) Pass to the Power Platform Conference in Las Vegas, Sep. 18-20, 2024 ($1800 value). NOTE: Prize is for conference attendance only and any other costs such as airfare, lodging, transportation, and food are the sole responsibility of the winner. Tickets are not transferable to any other party or to next year’s event. ** PLEASE SEE THE ATTACHED RULES for this CHALLENGE** Week 1 Results: Congratulations to the Week 1 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Community MembersNumber SolutionsSuper UsersNumber Solutions Deenuji 9 @NathanAlvares24 17 @Anil_g 7 @ManishSolanki 13 @eetuRobo 5 @David_MA 10 @VishnuReddy1997 5 @SpongYe 9JhonatanOB19932 (tie) @Nived_Nambiar 8 @maltie 2 (tie) @PA-Noob 2 (tie) @LukeMcG 2 (tie) @tgut03 2 (tie) Week 2 Results: Congratulations to the Week 2 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Week 2: Community MembersSolutionsSuper UsersSolutionsPower Automate @Deenuji 12@ManishSolanki 19 @Anil_g 10 @NathanAlvares24 17 @VishnuReddy1997 6 @Expiscornovus 10 @Tjan 5 @Nived_Nambiar 10 @eetuRobo 3 @SudeepGhatakNZ 8 Week 3 Results: Congratulations to the Week 3 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Week 3:Community MembersSolutionsSuper UsersSolutionsPower Automate Deenuji32ManishSolanki55VishnuReddy199724NathanAlvares2444Anil_g22SudeepGhatakNZ40eetuRobo18Nived_Nambiar28Tjan8David_MA22 Week 4 Results: Congratulations to the Week 4 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Week 4:Community MembersSolutionsSuper UsersSolutionsPower Automate Deenuji11FLMike31Sayan11ManishSolanki16VishnuReddy199710creativeopinion14Akshansh-Sharma3SudeepGhatakNZ7claudiovc2CFernandes5 misc2Nived_Nambiar5 Usernametwice232rzaneti5 eetuRobo2 Anil_g2 SharonS2
On July 16, 2024, we published the 2024 release wave 2 plans for Microsoft Dynamics 365 and Microsoft Power Platform. These plans are a compilation of the new capabilities planned to be released between October 2024 to March 2025. This release introduces a wealth of new features designed to enhance customer understanding and improve overall user experience, showcasing our dedication to driving digital transformation for our customers and partners. The upcoming wave is centered around utilizing advanced AI and Microsoft Copilot technologies to enhance user productivity and streamline operations across diverse business applications. These enhancements include intelligent automation, AI-powered insights, and immersive user experiences that are designed to break down barriers between data, insights, and individuals. Watch a summary of the release highlights. Discover the latest features that empower organizations to operate more efficiently and adaptively. From AI-driven sales insights and customer service enhancements to predictive analytics in supply chain management and autonomous financial processes, the new capabilities enable businesses to proactively address challenges and capitalize on opportunities.
We're embarking on a journey to enhance your experience by transitioning to a new community platform. Our team has been diligently working to create a fresh community site, leveraging the very Dynamics 365 and Power Platform tools our community advocates for. We started this journey with transitioning Copilot Studio forums and blogs in June. The move marks the beginning of a new chapter, and we're eager for you to be a part of it. The rest of the Power Platform product sites will be moving over this summer. Stay tuned for more updates as we get closer to the launch. We can't wait to welcome you to our new community space, designed with you in mind. Let's connect, learn, and grow together. Here's to new beginnings and endless possibilities! If you have any questions, observations or concerns throughout this process please go to https://aka.ms/PPCommSupport. To stay up to date on the latest details of this migration and other important Community updates subscribe to our News and Announcements forums: Copilot Studio, Power Apps, Power Automate, Power Pages