Reset count of ID once they've all been seen, count how many days it took then reset until next time

[Syndicate_Admin]

Hi,

I have 1500 unique IDs, everyday many are seen and some multiple times a day.

I would like to do a cumulative count and know how many days does it takes to see them all 1500, once it reaches that amount, the cumulative sum needs to reset for another count until the next time the goal is met.

I've done a running total after a grouped count by date, but my issue aside from the needed reset is that the duplicates needs to be removed based on the time period needed to attain the goal. If i'm grouping my IDs by date then i'm losing the ability to sort out duplicates...

So i'm guessing I need to first find how much time is needed, then remove the duplicates from a custom column spanning over its own run tower the goal, and then, calculate the running...

Here's an ugly paint mockup of what i'd like my final result to look like (I realize I need my date format to be date.time otherwise i'll never get a correct goal and it'll be tricky)

I've found this solution here from Lind25 but the syntax is wrong, maybe from a previous PQ version:

https://community.powerbi.com/t5/Desktop/Running-total-that-resets-when-equal-to-or-greater-than-a-s...

let
    Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
    AddedIndex = Table.AddIndexColumn(Source, "Index", 1, 1),
    #"Added Custom" = Table.AddColumn(AddedIndex, "Running total reset", each List.Accumulate(List.FirstN(AddedIndex[Difference],[Index]),0,(state,current)=>if state+current > 60 then 0 else state+current))
in
    #"Added Custom"

Any takers ? I'm completely lost on that one...

Thanks

[Syndicate_Admin]

Hi @Kiwizqt 

I don't quite understand.

Can you show

1. the original data you are working on (an excerpt that shows all the relevant fields)?

2. the result you expect from the data on 1 and how you get there

Please mark the question solved when done and consider giving kudos if posts are helpful.

Contact me privately for support with any larger-scale BI needs, tutoring, etc.

Cheers 

[Syndicate_Admin]

Hi @AlB thank you for your time and sorry not to have explained it better.

My answer got removed for suspission of spam 😞 here's another go at it.

Here's what i'm working with:

And here's what I'd like to know about it:

Where Completion Cycles refer to the moment I choose to start counting ID toward the 1500 IDs needed to be seen,

Where Days to Achieve refer to the time it took to reach those 1500 unique IDs,

And ideally, a Time Span column telling me the time period those completion cycles took place in.

The trick is that I'm registering duplicates IDs on that road to 1500 uniques and that the next completion cycle needs to start as soon as the previous one is done toward a new 1500 unique IDs goal.

I do have another table where my 1500 total IDs are stored if a join is needed.

Note: I don't have much knowledge in DAX/PBI but if you think it's easier or you don't think it's doable in PQ, I could also upload my source data and make it work there.

[Syndicate_Admin]

This is easier done in DAX in my opinion.  Here is a measure expression you can try in a card visual to give the first (min) datetime where all the IDs have been seen.  To see the 2nd and 3rd time, just change the 1 to 2 or 3.  I made some sample data and called that table Seen, so change that throughout to your actual table name.  Note there is a lot of calculation going on here, so it may not be performant at large scale.  If you try it, please let me know how it performs.

First All IDs Seen =
VAR ids =
    ALLSELECTED ( Seen[ID] )
VAR idcount =
    COUNTROWS ( ids )
VAR summary =
    ADDCOLUMNS (
        VALUES ( Seen[Date & Time Seen] ),
        "IDsSoFar",
            VAR thistime = Seen[Date & Time Seen]
            RETURN
                COUNTROWS (
                    FILTER (
                        ids,
                        CALCULATE (
                            COUNT ( Seen[ID] ),
                            Seen[Date & Time Seen] <= thistime
                        ) >= 1
                    )
                )
    )
RETURN
    MINX (
        FILTER (
            summary,
            [IDsSoFar] >= idcount
        ),
        Seen[Date & Time Seen]
    )

Regards,

Pat

[Syndicate_Admin]

= let

// {1..6} is the set of IDs
    Source = List.Accumulate(Tabl[Index], 
        [dcompl={[when=Tabl[Date]{0},atPos=Tabl[Index]{0}]},ck={1..6}],
        (s,c)=>  if List.IsEmpty(s[ck]) then [dcompl=s[dcompl]&{[when=Tabl[Date]{c},atPos=c]},ck={1..6}] else  [dcompl=s[dcompl],ck=List.Difference(s[ck],{Tabl[id]{c}})])
in
    Table.FromRecords(Source[dcompl])

the main difficulty seems to be the presence of duplicate IDs before filling up.
This can be overcome by using the concept of union of sets, with the List.Union function.
It can be done in many ways,
If you place some tables that can be copied easily I can try it.

if not try to adapt this script or the equivalent made with list generate.

########

basically with the same logic, but this implementation is much more efficient, as it avoids continuous list sorting and comparisons are simpler (IsEmpty I think is much faster than list1 = list2)

let
    Source = List.Accumulate(Tabl[Index], 
        [dcompl={[when=Tabl[Date]{0},atPos=Tabl[Index]{0}]},ck={1..6}],
        (s,c)=>  if List.IsEmpty(s[ck]) then [dcompl=s[dcompl]&{[when=Tabl[Date]{c},atPos=c]},ck={1..6}] else  [dcompl=s[dcompl],ck=List.Difference(s[ck],{Tabl[id]{c}})])
in
    Table.FromRecords(Source[dcompl])

[Syndicate_Admin]

@mahoneypatthanks for the try but the card only returns the date of completion (correctly I might add, I checked it), but it doesn't count the numbers of days needed.

Ideally, I would have liked to plot it in on a line graph with the numbers of IDs on (y) and raw days on (x) so that I could superimpose them throughout various periods.

Hi @Rocco_sprmnt21 ! Thank you for your time, i'll be honest I can't decypher much of your code but i'm eager to learn from it, i've hosted an excel on googledrive below, is that enough/available for you ?

https://drive.google.com/file/d/1gWHYl3FxX3t2Dt6jQ8MCBWV1WF4d80V0/view?usp=sharing

[Syndicate_Admin]

I thought the finish time was one of your requests, but you can easily adapt the expression I suggested to get the time it took as follows.  You can choose another time dimension instead of MINUTE (DAY, HOUR, etc.).  Also, for cycles 2 and higher you would need to set the starttime variable to the same expression but with cycle minus 1. 

Time for First Seen =
VAR ids =
    ALLSELECTED ( Seen[ID] )
VAR idcount =
    COUNTROWS ( ids )
VAR summary =
    ADDCOLUMNS (
        VALUES ( Seen[Date & Time Seen] ),
        "IDsSoFar",
            VAR thistime = Seen[Date & Time Seen]
            RETURN
                COUNTROWS (
                    FILTER (
                        ids,
                        CALCULATE (
                            COUNT ( Seen[ID] ),
                            Seen[Date & Time Seen] <= thistime
                        ) >= 1
                    )
                )
    )
VAR completiontime =
    MINX (
        FILTER (
            summary,
            [IDsSoFar] >= idcount
        ),
        Seen[Date & Time Seen]
    )
VAR startime =
    MIN ( Seen[Date & Time Seen] )
RETURN
    DATEDIFF (
        startime,
        completiontime,
        MINUTE
    )

Regards,

Pat

[Syndicate_Admin]

Hi @Kiwizqt 

Given the size of your data, the List.Accumulate function is not good.

You could try List.Generate (but try incrementally (*) ) or try the path indicated by @mahoneypat .

try list.generate starting from small lists and gradually increasing the size to evaluate the times.

I propose you a draft, which calculates only the essential. Returns a vector with all nulls except the values of the position where the breaks occur.
You can use this list (filtered by null values) to find the affected rows.

let
    //idsV is a partial list of all ids
    // Seen is a partial list (20k-items, f.i.) of overall ids into Seen table
    ids=List.Buffer(idsV),
    s=List.Buffer(Seen),
    nID=List.Count(Seen),
    Source = List.Generate(
        ()=>[atPos = null,ck=ids,idx=0],
        each [idx] < nID,
        each  if List.IsEmpty([ck]) then [atPos= idx,ck=ids,idx=[idx]+1] else  [atPos=null,ck=List.Difference([ck],{s{[idx]}}),idx=[idx]+1],
        each [atPos])
in
    Source

(*)

at worst, you have 1500 ids and 100K rows, then for each of the 100k ids in the table it is necessary to "traverse" the whole vector of distinct ids 1,5M check.

If trying a list of 20k and 100 distinct ids it takes 1 ', then for your table it takes 100k / 20k * 1500/100 = about 75' (1H, as an order of magnitude of course)

[Syndicate_Admin]

Hi, @Kiwizqt 

In the two worksheets you provided, the number of non-duplicate IDs in the seen worksheet is 1698, but the number of IDs in the Total ID worksheet is 1374.

Can you re-provide a correct form file?

[Syndicate_Admin]

Hi @ziying35 sorry for the inconvenience, I cleared wrong datas by applying an inner merge. The 324 ID excess came from wrong inputs from users and I thought it didn't matter for the solution as I would've done that merge before any solution given here, or that I could manually edit the m-code.

Bellow the finite data:

https://drive.google.com/file/d/1gWHYl3FxX3t2Dt6jQ8MCBWV1WF4d80V0/view?usp=sharing