Error loading 500000 rows into entity

[v-shimud]

Hi ,

Earlier when i was trying to load data >  3 million was reciving an error limit only allows 500000 rows. 

But when i am trying to load data by reducing the row size limit to 500000 receiving a new error. Please advise.

Our scenario is we are currently using MDS for managing our metadata and would like to move to powerapps. 

Load to Common Data Service failed. Details: ([Internal Server Error], [{"error":{"code":"InternalServerError","message":"An internal error occurred (tracking Id: 2f7693b8-17fb-4798-a14f-449f7d5576c0)."}}], The remote server returned an error: (500) Internal Server Error.), Failed to retrieve project schedule from https://management.azure.com/subscriptions/6989f3ba-b425-45dc-affb-047d53242a54/resourceGroups/d365-.... (Job ID: e9f03dda-d86e-4a9a-9a7d-192e3adf1828)

Thanks,

Shiva M

[RandyHayes]

@v-shimud 

So the question would be - how functional would an App be with > 3 million records in it?

What I'm getting to is that the App should be an interface to information and subsets of information and not an "application" that is going to perform complex actions on large datasets.

So, there are certainly limits and some are able to be worked around - but, in many cases, it comes back to "why do you need so many records in a user interface?"

PowerApps is highly focused around lists and galleries of data - so the question would be, would you want your users scrolling through >3 million records in a list?

This is seen many times for large data sources where the concept is - need to search it all, but PowerApps does not do well with searching large sources.  

So, the first think to look at is the DataSource...how much can you offset onto the server before it ever comes back to PowerApps.  

For example - a large SQL database.  We could pull down (with some effort) every row in the database in order to then successfuly search the entire database in PowerApps.  But, perhaps the better method is to create a View or perform a function on the database and only have it return what is necessary/desired.

So...not sure if this helps in your design process, but it is something to consider.

[v-shimud]

Thanks Randy,

I am just replicating one of our MDS model into powerapps, requirement is load distinct productid's (>3 million) into one entity and then create a lookup relationship with other entity. What i am trying to achieve here is when users create a new record of productid column they should not enter a different id other than that available in the lookup list. Basically they wont have to select anything or scroll through the list but just if they could paste the productid it should throw an error if that id is isnt in the list and succeed if it is.

The actual metadata table that holds data is very small, however we have to create some lookup columns which could reference large tables.

Could you let me know if there is any such limit that we cannot load data more than 500000 into powerapps or if above scenario dosent work with powerapps.

[RandyHayes]

@v-shimud 

So in this case you're really only trying to load that list in order to check a productid at time of submit for a new item.

If that is the case, can't you just perform a lookup for that productid prior to executing the submit of the new item?  That way you could determine if it is already there.

You could also do it during the entry of the item - and disable the submit button and make an error message visible if it exists.

Just a thought...perhaps it will fuel some thought for you as well.

[v-shimud]

@RandyHayes 

Our scenario is we setup the environment and share the app with business users, who are then responsible for loading data into it. So we cannot validate it prior loading.

[RandyHayes]

@v-shimud 

Yes, but what I am suggesting is that in your App, when you go to create a new record as you said, at that point have the logic in the formulas to do a lookup on the data and validate the productid as valid or not.

[v-shimud]

Thanks Randy

Part of my issue is solved by using lookup function. However my initial issue will still exist as it dosent allow more than 500000 rows in the entity. Please suggest any workaround for this as the lookup function has to lookup values in table that has > 3 million rows.

[RandyHayes]

@v-shimud 

That one I'm not sure on...I yield to anyone else that might have a response.

[v-shimud]

@RandyHayes  i am creating model-driven apps, i cant seem to find a way to write functions in model driven apps. So to get the option for writing lookup functions should i be using canvas apps. 

Thanks,

Shiva M

[RandyHayes]

@v-shimud 

Not entirely sure the differences you might be experiencing.  My world is around canvas apps at the moment.  Model driven is on the agenda for this month.