Uploading offline records issue

[AlCado]

Hi,

I need your help to solve an issue concerning if function.
I've created an app which allows an auditor to add compressed air leaks (location, description) of the factory to a collection being offline. Then, when he grab an internet connection, a button permit him to upload those leaks to an Excel database stored on the factory's buisiness drive. There are hundreds of records to upload so the process is quite long but it doesn't matter.
The problem is that if during the upload, the internet connection crashes, all the leaks not uploaded yet are lost !
To avoid this to happen I putted the remove function in an if(Connection.Connected, remove(collection, last record uploaded). But the condition is only checked once when the upload button is actionned so when connection cut off the collect function stops while the remove function keep running.
How, without using a timer, can I force the condition to be checked each time a leak is uploaded ?

Here's my upload button code : 

If(Connection.Connected && CountRows(LeaksToBeAdded)>0; 
        ClearCollect(TemporaryCollection; LeaksToBeAdded);;
        ForAll(TemporaryCollection; Collect(DB_LEAKS; First(LeaksToBeAdded));; 
            If(Connection.Connected; Remove(LeaksToBeAdded;; 
                First(LeaksToBeAdded)));; 
        SaveData(LeaksToBeAdded; "NewLeaksInLocalStorage"));;
        Refresh(DB_LEAKS);;
        ClearCollect(LeaksCollection; DB_LEAKS);;
        SaveData(LeaksCollection; "LeaksInLocalStorage");;
        Back(); 
    Connection.Connected && CountRows(LeaksToBeAdded)=0;
        Notify("No leaks to be added"; NotificationType.Error);;
            Back();
        Notify("You are offline, find internet connection to proceed"; 
            NotificationType.Error))

LeaksToBeAdded corresponds to the collection which store the leaks recorded when offline.

DB_LEAKS is the database storing all the leaks in compressed air network, filled by the app.

LeaksCollection is a collection used as a cache for DB_LEAKS database.

Attached a flowchart of the upload button OnSelect actions.

[wyotim]

I think a decent solution for your situation would be to set the remove condition after you refresh your dataset and check against the refreshed data for what to remove and what to keep. Basically, if you refresh the dataset and the temporary item is in there, remove it. If the temporary item is not in the dataset (either because it wasn't able to be refreshed or something else), don't remove it.

One issue this brings, however, is that if some items are uploaded but the user loses connection (and thus the dataset doesn't refresh) you may end up submitting records twice. So, you would probably need to check before uploading, skipping any duplicates, and then check again after uploading. 

As far as the actual code goes, I don't have the time just now to work that out but would be willing to come back to it this weekend if you need some help. 

[AlCado]

Thank you for your response,

I tried to apply your proposal (remove after collecting the dataset by comparing the two collection's primary key). However, the problem remains: the remove condition is only checked once at the beginning.

Here is my new code

If(
Connection.Connected && CountRows(LeaksToBeAdded)>0; 
        ClearCollect(TemporaryCollection; LeaksToBeAdded);;
        ForAll(TemporaryCollection; 
            Collect(DB_LEAKS; First(LeaksToBeAdded));; 
            Collect(LeaksCollection; First(LeaksToBeAdded));; 
            SaveData(LeaksCollection; "LeaksInLocalStorage");; 
            If(First(LeaksToBeAdded).NumLeak in LeaksCollection.NumLeak; 
                Remove(LeaksToBeAdded; First(LeaksToBeAdded));; 
                SaveData(LeaksToBeAdded; "NewLeaksInLocalStorage")
            )
        );;
        Refresh(DB_LEAKS);;
        Back(); 
Connection.Connected && CountRows(LeaksToBeAdded)=0;
        Notify("No leaks to be added"; NotificationType.Error);;
            Back();
        Notify("You are offline, find internet connection to proceed"; 
            NotificationType.Error)
)

It would be very nice of you to help me find a solution, it's the last unsolved issue of my app before start using it.

You will find attached a docx file to help you understand the origin of names and the purpose of the app if necessary.

[wyotim]

So first off, awesome documentation. Just great stuff there. There is a typo on the flow diagram on the Leaks to Upload? step where both options are No, just so you know. Easy to see what is going on there though. 

In looking at your code a bit more, what about doing a Collect of the LeaksToBeAdded to the DB_LEAKS data source, refreshing DB_LEAKS, and then running the check/removal portion on the LeaksToBeAdded? This could have the secondary effect of limiting the "exposure" time where a disconnection could happen by getting the submission done first and then handling the check/removal on the device when it won't matter if the device is connected or not. 

Something like this (though keep in mind, this is obviously untested so don't use it!):

If(
Connection.Connected && CountRows(LeaksToBeAdded)>0; 
        ClearCollect(TemporaryCollection; LeaksToBeAdded);;
        Collect(DB_LEAKS; LeaksToBeAdded);; 
        Collect(LeaksCollection; LeaksToBeAdded);; 
        SaveData(LeaksCollection; "LeaksInLocalStorage");;
        Refresh(DB_LEAKS);; 
        ForAll(TemporaryCollection,
            If(First(LeaksToBeAdded).NumLeak in LeaksCollection.NumLeak; 
                Remove(LeaksToBeAdded; First(LeaksToBeAdded));; 
                SaveData(LeaksToBeAdded; "NewLeaksInLocalStorage")
            )
        );;
        Back(); 
Connection.Connected && CountRows(LeaksToBeAdded)=0;
        Notify("No leaks to be added"; NotificationType.Error);;
            Back();
        Notify("You are offline, find internet connection to proceed"; 
            NotificationType.Error)
)

This also helps to isolate the check/removal section as that seems to be the issue. To help debug it, you could comment that section out and make a temporary button with that code then testing and refining the code until you reach the desired result. 

As it stands, I am not seeing the issue directly. I made a simple test app to try out the remove code and it worked as expected, but again I had the collecting and removal as separate functions similar to what I put above but with two buttons instead. Maybe try separating them out and see what you get? I will think on this a bit more and see if anything comes to mind as well. And maybe someone else will comment with a great answer too. 

[wyotim]

Another thought I had on this: where is NumLeak created? I assume that is your primary key, yes? Is it possible that the collections in the check/remove section don't have that? And further, is it possible to use another field (or group of fields) to identify each record to achieve the same goal? That could be the whole issue.

[AlCado · Accepted solution]

After a long time thinking and experimenting I've finally reached my goal !

By following your advice I realized that I misevaluated the problem: the if condition is well checked for each record, the issue was in the order and the conditions themselves !

Big thanks for your help, your kindness and for the typo in the chart.

And to answer your question, the NumLeak is created in an unvisible and uneditable field which auto increment in the NewLeakForm.

Here is the final solution (I prefered put the collect also in the forall loop to be able to see the progression of the upload):

If(Connection.Connected && CountRows(LeaksToBeAdded)>0;                                                                  
    Notify("Upload has started"; NotificationType.Information);; 
    Refresh(DB_LEAKS);; 
    ClearCollect(LeaksCollection; DB_LEAKS);;
    SaveData(LeaksCollection; "LeaksInLocalStorage");;                                                                   
    ClearCollect(TemporaryCollection; LeaksToBeAdded);; 
    ForAll(TemporaryCollection;                                                                                              
        If(Not(First(LeaksToBeAdded).NumLeak in LeaksCollection.NumLeak);
            Collect(DB_LEAKS; First(LeaksToBeAdded));;                                                                           
            Collect(LeaksCollection; Last(DB_LEAKS));;                                                                           
            SaveData(LeaksCollection; "LeaksInLocalStorage")                                                                 
        );;
        If(First(LeaksToBeAdded).NumLeak in LeaksCollection.NumLeak; 
            Remove(LeaksToBeAdded; First(LeaksToBeAdded));; 
            SaveData(LeaksToBeAdded; "NewLeaksInLocalStorage")
        )
    );; 
    Refresh(DB_LEAKS);; 
    ClearCollect(LeaksCollection; DB_LEAKS);;
    SaveData(LeaksCollection; "LeaksInLocalStorage");;                                                                   
    If(CountRows(LeaksToBeAdded)=0;                                                                                          
        Notify("Upload is over !"; NotificationType.Success);                                                                
        Notify("Upload is not over, find internet connection and relaunch to complete"; 
        NotificationType.Information)   
    );;                                                                                                                  
    Back();                                                                                            
Connection.Connected && CountRows(LeaksToBeAdded)=0;                                                                     
    Notify("No leaks to be added"; NotificationType.Error);;                                                             
    Back();                                                           
Notify("You are offline, find internet connection to proceed"; NotificationType.Error) 
)

[wyotim]

Happy to hear you got it worked out! Looks like a great solution too.