PAD - Extract from single table on single web page only returns data from the first 22 rows and not from all 257 rows?

[DanHu]

I have a simple extract (as a first test) from a website which contains a HTML table with 257 rows. All rows are in a scrollable HTML table within the first page (not an iFrame). There are about 15 rows visible in the window. For all other rows, one has to scroll down the table (not the browser window). That the data is in a scrollable table within the web page is the only difference to other tables I have successfully extracted.

As usual, I right-click in the first couple of rows the elements I need the text from.  After a few rows, the preview shows the first 20 elements of that table. Now I finish the extraction setup, save the flow and run.

Problem: Only the first 22 values from the first 22 rows are extracted. What could be the problem?

I tried all extended settings like single value, list, whole table, etc... no change.. always only the first 22 values are extracted.

What can I do to have all 257 rows being extracted?

[Pavel_NaNoi]

Try the following;

- Use Extract data from web page instead of the web recorder (or vise-versa if you're not using the web recorder)

- If you do use the Extract data from web page, play around with it in the advanced options, perhaps you're 1 CSS layer too deep, drop it in chat here if you want I can take a look at it.
- If that doesn't work, try selecting the 2nd selector that selects all similar values as the 23'rd row instead of the 2nd one
Let me know if neither of those two ideas worked, I'll take a stronger look at it, though I'll need the actual webpage or at least a very similar one to take a look at it and try it myself.

[mbenyishay]

Hi, did you ever figure this out? I have a bunch of automations using extraction from an HTML table that has a couple thousand rows, and it's only ever the first 100 rows that are extracted. Also a scrollbar, also not an iframe (I don't think) and it's far fewer than 100 rows that are visible at a time. Any luck? Thanks!

[Pavel_NaNoi]

If the website you're on is sharable, I can take a look to see what can be done.

[mbenyishay]

Hi @Pavel thanks so much for your response. Unfortunately, the site has PHI and is behind a login, so not shareable. Are there any generic instructions for how to use a scrollbar as a paging element? Is that even the proper way to do this? When I hover over the scrollbar, I do not get the red square of an element selector, it just jumps to the whole div (table).

[Pavel_NaNoi]

Don't think you can sadly, you need to use your keyboard instead via "Send keys" action, try either:

-Page Down

or

-Arrow Down (or any other arrow keys / Page keys to move around the table)

These two should scroll down for you.

If those don't work then you might need to get a bit creative, let me know if it doesn't work, and send me a screenshot with the table (blur the information in it if necessary, I just need to see what it looks like)

[mbenyishay]

Hi, here is a screenshot. I have been having a little trouble focusing the table in order to send a "page down" key. 

[mbenyishay]

Following up, here's another screenshot with focusing on a cell of the table and sending ten page-down keys ("{next}"). The page down works. This particular table has ~600-700 rows. However, the table still seems to be extracting a subset. It's usually around 130 rows but not totally consistent, strangely.

[Pavel_NaNoi]

Ah I think I've been through this problem before, its similar to this post https://powerusers.microsoft.com/t5/Power-Automate-Desktop/Power-Automate-Webscraping-help/m-p/15721...

I think you need to make your extraction selector more generic, the one you use is probably very specific to a "part" of the table. Play around with the extraction selector via the advanced options in "Extract data from web page" action until it highlights all the rows even when you scroll down, let me know how that goes.

[mbenyishay]

I was having trouble making the selector more generic. However, what I was able to do was to take this in parts. There is a bit of text that appears at the top of the page that identifies how many rows are in the result. I can retrieve this element's text, split it at the space, and convert it to # of rows. This tells us how many records we need to retrieve.

What I was able to do was retrieve a partial datatable, then go row by row to check if it's already in the master data table (row number from the results > max row number in the master data table). If it's not, insert it into the master data table. Then, send a page down (or 2 or 3, hard to tell how much it should be) and extract a new partial data table again. 

This currently appears very time consuming, but I think that's because I'm running it from editing the flow and it has to visualize the looping. We seem to be getting to the right number of rows, ultimately, possibly with a few duplicates. Since ultimately I am inserting these records into a SQL server table, I can check whether the record ID  (primary key) already exists before inserting it.

So this seems to be working without trying to mess around with the web page more, which is what I know least about...?

[ryule]

Yes, I have quite a few bots that take forever to run while I'm editing the flow, but it runs exponentially faster running it after saving and closing and clicking the play button, and then like another 10x faster running it from a cloud flow unattended on our VM!