Scrape a website that has hidden data

[uok2me]

TLDR: cant scrape site with multiple pages for phone number be cause it's behind a click

I built a few flows pulling data from other sites like the Better Business Bureau and yellow pages but now I am stumped at the silliest thing.

I want to pull the NAME of the company, CITY/ADDRESS and phone number. The problem that you have to click on the phone number to see it so it can be captured

launch new browser: https://www.houzz.com/professionals/kitchen-and-bath/c/Concord--CA

click on View Phone number for first company

scrape NAME of the company, CITY/ADDRESS and phone number

go to next listing 

click on View Phone number for 2nd company

scrape NAME of the company, CITY/ADDRESS and phone number

and 3rd and 4th and so on

click next page and do it all again

[Tjan]

Maybe I don't understand your problem well enough, but can't you just program the bot to click on 'view phone number' ? And afterwards scrape the phone number?

[uok2me]

yes I can. I can tell it to:

"click link on web page" 

then:

"extract data from web page"

But that will just get me the data from that one company that I told the bot to click on the phone number

What I am having trouble is to say:

1. click "phone number" 

2. download data 

3. go to next contact and do 1 and 2 again

4. at end of data for this page click next 

5. repeat until no pages left

[Mitch6024]

Use the recorder to select the link you have to click to view the phone number for a few different companies. 

Then compare them with each other. Most likely, somewhere in the selector there will be a number that increases as you go the list clicking on the link to view the phone number. 

Add a loop and place the loop index in that spot so it increases which would move to the next link. 

[uok2me]

I don't understand what you mean?

1. If I use the recorder...that just records. It will NOT predictively guess that I want ALL the "View Phone Number" clicked on. I can record me clicking on each "view phone..." then run an "extract data" but that will just do it for page one and if i have to record each click it defeats the purpose of automating it 

2. "compare them with each other." Compare who? The "view number.." that i have clicked on and the ones i have not? ummmm....the ones i clicked on show a number and the ones i did not click dont. There is no selection to tell it to click on every chosen item. I even tried "if web poage contains" hoping that i could say "if web contains (click to view number) click on it" but that commands only finds txt and ends flow.

3. there is no point on making a loop if it will just click the "view..." on the same page

[Tjan]

The first option you mention, is actually the only way to do it. Simply for the reason that the phone numbers is behind the click (so it is a must). You can build this into another loop, so that it does it for all the pages. 

A tip: instead of clicking the pages one by one, you can simply scrape the AHREF of the different listings and use this list in a 'for each' to visit each page. Afterwards u can program it to click on 'view phone number' to scrape it (for each page visit ofcourse).

Be aware that there's no way to get the phone numbers without clicking the 'view phone number'. U need to program the click.

[uok2me]

"the phone numbers is behind the click (so it is a must)."

-----I know that's what I've been saying.

"You can build this into another loop, so that it does it for all the pages."

-----no you cant. If I RECORD me clicking on each "view number" and save it in my flow. Then use EXTRACT DATA FROM WEB PAGE then use CLICK LINK ON WEB PAGE  for the "next" button page turn at the end and LOOP it all. It should click the phone, get the  data and go to the next page and do again....since there slight differences on each page the recorded clicks breaks.

"you can simply scrape the AHREF"

-----nope. Im on the marketing/sales team. I dont program so dont know what or know how to use AHREF.

I need IF WEB PAGE CONTAINS to let me say: If web page contains "view phone number" click on it. Then run the extract then loop it.

or

if CLICK LINK ON WEB PAGE was as predictive most of the other tools that could work too. This is a really cool application, I am shocked that a "click to view" call to action on a page is such a roadblock

[PAG]

Hello @uok2me 

You *could* build the flow with recorder, but it will require a decent amount of time and knowledge. In addition, you would probably need to build the flow from scratch each time the web page changes, so I would not recommend to use that approach.

A much better way is to use JavaScript. You can open all hidden phone fields with it and you don't need to open web pages separately to see the info. JavaScript will loop and fetch all info and return a JSON string. You can use FOR EACH in PAD to use those values, here is the complete solution to fetch information from page 1 (just insert your URL to line 1):

https://power.automate.gallery/cSGH0VIlmO0a