Power automate web extraction

[Squeaky]

after following this tutorial below, I would like to extract from a website which has multiple pages. So page 1, 2,3

https://youtu.be/_O9eEotCT0U 

After trying the page selector just flashed yellow and doesn’t export the excel. 

[fraenK]

Sounds like that the next page selector is not correct. Try to test it without the extraction through a click action

[burque505]

@Squeaky, I'm having very much the same problem. I have a site with tables, 20 rows per page. There are 73 rows total, so four pages.  At the bottom of the page are the numbers 1 through 4 for the pages, then a 'Next' button, then a 'Last' button. I get two different sets of problems, depending on the run.

Works fine:

Why on earth is there an option for max web pages to process if it's set to 'Only the first'? Anyway, for a single page it works.

In this screenshot I use 'Next' for the pager.

Now I have to set the extraction differently. Here's what I get - why is there no field for max pages here, where it belongs?

Shouldn't this be something like 'all available' with another field for 'up to a maximum of', or similar? I'm stumped. But it is what it is.

Depending on the run, two things happen.

First, it usually just fails to extract the table. Second, less frequently, it extracts everything but keeps banging the 'Next' button, and will not stop, regardless of what error settings I try. The page was created with Angular, which may be part of the problem, I don't know. Frustrating though, and sounds like something similar to what you're dealing with.

EDIT: I have partial success now, but only partial, with the extraction set to 'only the first page' and the max number of pages set to 4 (why does that even work?). I also discovered the pager element had disappeared from the 'Advanced' settings, which was part of my problem. I've been unable to get the 'live web helper' to re-select the pager element from the context menu, so I've added it as a CSS selector manually in advanced settings. I'd be interested to know if anyone else has that problem.

So that gets rid of the non-extraction problem.

Perhaps my 'all available' problem is due not to anything in PAD, but on the web page. The Next button is disabled when I reach the last page, but PAD still sees it and keeps banging on it like a cop trying to serve an arrest warrant.😁

For this I may have to first find the 'Next' element, move to the first button to its left, extract the text on the button, put that in a variable, and then set the max number of web pages to that variable.

@Squeaky, I get the sinking feeling I'm not helping you much here. Sorry about that.

Regards,

burque505

[fraenK]

PAD is not really good at web data extraction so you need to manually check the selectors for data and next page. They might change per session and per page so you would need to make them more generic or even set the selector manually based on own tests in your browsers console.

[burque505]

@fraenK, that sounds like the plan. I automated extraction of the same site using AutoHotkey before (both with SeleniumBasic and Chrome.ahk) and had to use JS for almost everything. (At least that way I can check whether the 'Next' button is disabled.) So what you're saying makes perfect sense.

[Squeaky]

@burque505 No still not sure what to do. Please put this in simpleton terms ha. I assume it is relatively  simple to do? Do I just right click on the "next page" button and set to pager? or do I right click on the whole selector box? 

This way is ideal as I can track the price changes over time. I've done Web Scrapes using power Bi but there's no way to automatically track the price changes daily. 

[burque505 · Accepted solution]

@Squeaky, yes, right-click on the 'Next' button and set it as pager. If it doesn't show up in the 'Advanced' settings you'll have to add a CSS selector by using DevTools (Chrome, Edge) or Inspector (Firefox).

Regards,

burque505

[Squeaky]

Thanks, I ran it on another website and it works. Just right clicked and set the next page button as the pager. I think it must be dependant on the website which is frustrating but at least I have a working report now. 

[Brenainn]

What if there isn't a button or link for next page--only hyperlinks for each page individually? For example, 1 2 3 4 5 etc. What should the process be then? Thanks in advance!

[DylanEvans]

bump on Brenainn's question: how do you paginate if there's no next button, only numbers for each available page?

[MV_NK]

How to get count of pages while extraction ? to fill max web page to process dynamically

[Anonymous]

I had this problem on the linkedin jobs page where you dont have a previous/next button and just the page numbers. 

In this case I noticed that the query string of the URL had a parameter for &start that increments the count by 25 for each page.  So I created a counter variable that stores the total results, and then run a loop reloading the webpage with the updated &start parameter in the query string incremented by 25, and was able to extract data from multiple pages.

[YeuHarng]

hi is that any possible to do based on the left title and get right hand side details? This is from web 

[Agnius]

Nope. You would need to extract the entire table and then use Find or replace in data table to find your match and retrieve the value on the other column via an index.

[YeuHarng]

Hi, so now i already extract the table in excel is that correct? So about the Find or replace in data table function is that in excel or in power automate? I have no idea how do it

[Agnius]

Instead of extracting it to Excel, extract it to a variable. Find or replace in data table is a Power Automate Desktop action.