Filter excel file, create a html table and send it by email

[Lucas_Goncalves]

Hello, good morning from Brazil 🙂

I´m suffering on the last 3 weeks on this flow demmand. I checked google, Reddit, Medium, even here... can´t solve... I can´t sleep, can´t eat, cant be happy.... (Ok, Im joking...)

What I´m trying to do:

Trigger: When an item is created in a sharepoint list ( This list has a form where I will imput vendor code, say, 12345)(OK!)

1) List rows present in a table (Properly formated as a table) that has been saved on Sharepoint (file name is FBL1N.xlsx)(OK!)

2) Filter this excel with the trigger information (Vendor colum = form imput)(not OK!)

3) Create an html table(OK!)

4) Style Html table (OK!)

[replace(body('Create_HTML_table'), '<table>', '<table border="3" bgcolor="ffffff">')]

5) Send e-mail with the Styled HTML on it (OK!)

My actual flow:

What I´m stuck:

I cant create the filtered html table. I mean, the e-mail that has been sent on the above flow, is not filtered with the result from the trigger form information.

On step 2 (List row in a table) I know that there are a "Filter Query". But when I add the form information, it immediately changes to an "apply to each" and then I cant create html table anymore.

Are you there, Obi-wan Kenobi? Can you help me? What I am doing wrong? How can I escape the dark side and go to the correct Jedi Flow Pathway?

[fchopo]

Hello @Lucas_Goncalves 

Before creating the html table, you could use a "filter array" action, filtering the results by vendor and then creating the table. Have a look at this example:

 In this case, I'm filtering by the "expense" value column (greater than 10).

Hope it helps!

Ferran

[Lucas_Goncalves]

Hello @fchopo , thanks for your support!

When I apply the filter as per your suggestion:

Filter array from value of the Lists Rows present in a table, using colum name "Vendor" is equal to the information present on the form "Title".

It forces the "apply to each" and then the same problem persists:

It don´t shows colums informations on Dynamic content to allow me to create the html table:

[fchopo]

Hello @Lucas_Goncalves 

Just realized that it creates an "apply to each" action when you use the "title" field that you get from the "Get Items" SharePoint action. One question: Why do you use the "Get Items" action? Or what is the purpose of it? As I understand, you want to:

1) When an item is created in SharePoint.

2) Open an excel file and filter the rows according to the vendor value entered in step 1.

3) Export the excel to the HTML Table.

Am I right?

Ferran

[Lucas_Goncalves]

@fchopo Amazing, Apply each solved!

But...

Filter seems to not finding information as it sends me empty email:

This is how it is:

This is how it appears in the result: Filter output is empty

[fchopo]

Hello @Lucas_Goncalves 

Looking at the screenshots you shared, the vendor code is stored in a column name "Code" and not in the "Title" one. Could you check it? Therefore, you should filter by this column, and not by Title.

Hope it helps!

Ferran

[Lucas_Goncalves]

SP did not updated the Renaming of the standart column name "title" as "Codigo" on Flow... they are the same.

I figured out that there might be empty row problem, as I got to the excel file, selected the vendor code that appears on first line and the filter worked as it should be.

When I check on the second, after the empty rows, it broke:

 

Did a workaround to a condition: if empty, do nothing. If no, do that what you teached me. Appeared to work! Will do a few more test before giving you the big winning solution:

[Lucas_Goncalves]

@fchopo upon testing here i just noticed that HTML table is comming with dates in wrong format ("Data de Pagamento" and "Emissão" are date columns:

How can I fix that?

I´m still testing the filter.

[Lucas_Goncalves]

@fchopo  Good night!

Still with filter error 😞 made 20 different attemps.

It sends me an empty e-mail with no HTML table rows (only the columns name). It seems that the error is on the filter array that can´t output the information.

Can you check if the below picture can help you to help me? 

[fchopo]

Hello @Lucas_Goncalves 

Let's try to narrow your problem. Could you change your filter array expression and try it with simple values? For example, try to use an expression like: 

@equals(item()?['vendor'],'127476)

 You should try with different values or expressions until you find out what's happening.

On the other hand, if you want to format the dates, in the create html table you could you use an expression like:

formatDatetime(output('data de pagamento','dd/MM/yyyy')

Hope it helps!

Ferran

[Lucas_Goncalves]

Hi @fchopo , good morning!

About the problem: It seems that the filter array is limited to 256 lines on the excel file (it has more than 10k lines).

I found this as I checked the test result and found last line it could narrow. That is why all vendor code that is above this line number is suceeded.

How can I overpass this limitation? I know it is not pagination as it is for "Sharepoint´s get itens" and it is limited to 5k.

Will test your suggestion to specify the vendor code number as you suggested: @equals(item()?['vendor'],'127476) 

[Lucas_Goncalves]

@fchopo  figured out the problem. Excel Conector (Get a row) is limited.

Its native limit is 256 rows, but I can increase up to 5k rows manually with pagination.

I´ve tested and I could get information from rows 2561 and could not get from rows 5819 (my table has 17k rows).

Tested the vendor wich has rows between 4950 and 5010 and it prints me the html table up to row 5k.

Now... how I can bypass or arrange to get all the information automatically analyzed without changing the excel file?  

[Lucas_Goncalves]

@fchopo  I found a workarround on the excel rows limit, and confirmed that the entire rows (17k) information arrives on the Filter array (I checked the imput raw information and it has an information that is only stored on line 17718, per exemple), but still output is empty:

Here you can see my workaround that has been done with the link I provided before. It starts with 4000 rows, and skips 4000 rows each get rows until it founds that there is less than 4000 on the total row (4000, then 4000 then 4000 and then 4000. Since the file have 17728 rows, it finishes the "do until" because it gets the remaining 1728 rows, doing a 5 times check):

If you can, lets find out together on why the Filter array is comming with empty information...

[fchopo]

Hello @Lucas_Goncalves 

I've been doing some tests on my side and they are working fine. So I don't really know how I can help you with this. Let's see if more experiencied guys like @efialttes @abm @RobElliott @ChristianAbata can give you some support.

Regards,

Ferran

[efialttes]

OMG... what a challenging thread!

My suggestion is to split the problem in two:

1-First build an Excel table for testing purposes with less than 5k (or even less than 256) rows in order to make your filter strategy work as expected

2-Once you got a proper Filter strategy, test it with your 17k Excel by adding whatever strategy you chose to read the whole table (pagination/do until/etc)

I would focus this thread to the first step/problem only

If you agree on this approach, please let us know your progress with first step

Ánimo!

[Lucas_Goncalves]

EDITED

Hello @efialttes ! Good morning and welcome to this headache!

Ok, as I stated before I did a test with less than 256 rows. WORKED (Sent me the html table as spected!)!

Test with less than 5k rows. WORKED (Sent me the html table as spected!)!

Tested with more than 5k rows. FAILED (Sent me an email with empty) HTML table.

So the problem was on Get a row in excel. He could not, due to limitation on this item, get more than 5k rows.

As I mentioned before, I found a workaround and, by now, I can read entire 17k rows!

BUT

Filter array is still not working, as It receives all 17 k rows (I confirmed this by finding an information that is specific on the 17718 row.

But the output is going empty, for the 1st row, 256 row and 17th row. It sends me empty e-mail. 

What can we do?

Help me Master Yoda!

[efialttes]

Hi again!

"So the problem was on Get a row in excel. He could not, due to limitation on this item, get more than 5k rows."

Did you activate pagination in yor Excel 'List rows present in a table'? I just managed to read more than 20,000 rows in one shot by activating it and setting a huge Threshold: 100,000.

I am testing with 20,642 rows, and as you can see in the Compose output screenshot, 20,642 is the number displayed, so 'List rows present in a table' is prodiding as output more than 5k. No need to 'Do Until' logic

Hope this helps

[Lucas_Goncalves]

Hello @efialttes  made several tests here:

Paginations are limited to 5k.

But my workaround I mentioned before solved this part of "get rows":

The problem is within Filter Array.

Why I confirm that? Because the filter array had this command: @and(equals(triggerOutputs()?['body/Title'], item()?['Vendor']),equals(item()?['Document Type'], 'CA'))

And with this, it gaves me an empty table.

So I removed the @and and placed a simple filter:

Sucess! It sent me an table with information from row 17728 & 17729 (Both related to vendor 205044):

Now I need to figure out how to properly adjust the filter array to filter both information: Title equal to vendor AND Document Type equal to CA.

My code is: @and(equals(triggerOutputs()?['body/Title'], item()?['Vendor']),equals(item()?['Document Type'], 'CA'))

Can you find out what I made wrong?

[efialttes]

@Lucas_Goncalves 

Sorry to insist, but pagination is not limited to 5K, as I have shown you by sharing a screenshot of my test flow design and its results when executing it .

Just guessing, can you doublecheck you do not have 5000 assigned as 'Top Count' ? If so, it explains why you are experiencing a 5,000 limit

Once we get rid of this 5,000 limit in you current flow design, we will simplify the overall picture

Thanx!

[Lucas_Goncalves]

Hello @efialttes  no problem in insisting!

Please find error below. No top count and pagination placed in 50k:

That said, I believe the workaround I made is the only one avaiable... It works! Now the filter is the headache!