Power automate - Webscraping

[Michael061297]

I'm building an automation to extract data from a webpage, I have an on-block error to validate the data that are being scraped. When it can't search for anything it will return as "Unable to search in Excel". The problem is that the website sometimes reloads on its own and the value that will return in Excel is "Unable to search" instead of "Active". I'm new in Power Automate, please help. 

[Srini007]

Hi @Michael061297 

So, do you know how much time it will wait and reload, so you need to place a timeout for the action you are placing, Let suppose you are using Extract data from webpage then you have the timeout option as below

by defining maximum timeout so that It will wait the webpage to load before throwing any error

If you find this reply helpful, please consider giving it a LIKE AND

If this reply answers your question or solves your issue, please ACCEPT AS SOLUTION

Cheers,

Srini

[Michael061297]

Hi @Srini007,

Thank you for your response, mine is set to 60. However, I need to power automated to detect if the website I'm scraping data from accidentally reloads by itself, if that happens I need to place a "decrease variable" in powerautomate to search for that data again instead of returning "Unable to search"

[Srini007]

HI @Michael061297 

So, let's say you searched with some text and it returns you "Unable to search in Excel" Then this is element you can keep the logic, you can place a IF windows contains "Unable to search in Excel" UI element then wait action this you can configure how much time you need to wait and then you can place your logic in that IF block

If you find this reply helpful, please consider giving it a LIKE AND

If this reply answers your question or solves your issue, please ACCEPT AS SOLUTION

Cheers,

Srini

[Michael061297]

Hi @Srini007 

I have a subflow that automatically types "Unable to search" if there's literally nothing to search. But the problem is that given that the webpage sometimes reloads by itself even if there's something that my automation can search when the webpage automatically reloads my subflow will run inside the on block error and it will return "Unable to search" instead of getting a result. So maybe I need my automation to be self-aware that the webpage accidentally reloads and It will need to backward one row to search for that text again.

[Deenuji]

@Michael061297 

If the website you're targeting keeps reloading, it could potentially disrupt the web scraping process in PAD. In such cases, it's advisable to explore alternative methods for scraping your data. You might consider using JavaScript or parsing the webpage's source HTML directly, possibly utilizing regular expressions, among other strategies.

Thanks,
Deenuji Loganathan 👩‍💻
Automation Evangelist 🤖
Follow me on LinkedIn 👥

-------------------------------------------------------------------------------------------------------------
If I've helped solve your query, kindly mark my response as the solution ✔ and give it a thumbs up!👍 Your feedback supports future seekers 🚀

[Michael061297]

Hi @Deenuji,

Thanks, I have a very minimal knowledge of javascript but I can learn. How would you suggest I do that using Javascript. Thanks!

[Deenuji]

@Michael061297 

I'm unsure about the specific data you're attempting to scrape from your website. If you provide screenshots about your data with your use case details along with the website source code, I can offer further assistance and suggest potential solutions.


Thanks,
Deenuji Loganathan 👩‍💻
Automation Evangelist 🤖
Follow me on LinkedIn 👥

-------------------------------------------------------------------------------------------------------------
If I've helped solve your query, kindly mark my response as the solution ✔ and give it a thumbs up!👍 Your feedback supports future seekers 🚀

[Michael061297]

Hi @Deenuji 

Apologies for the delayed response, the website that I'm using is "https://mycpa.cpa.state.tx.us/coa/" and as you can see on the screenshot there's an entity field for me to enter an example. One example is "123 MAIN ST LLC" If I click search a result will return, however, sometimes if I click the search button it just refreshes the page and it will not search for 123 MAIN ST LLC thus returning a value of "Unable to search". Thanks!

[Deenuji]

@Michael061297 

For me it's not throwing any error like "Unable to search" instead sometimes it's not returning any results and sometimes it is returning the below results.

Do you want to extract the above results?

Thanks,
Deenuji Loganathan 👩‍💻
Automation Evangelist 🤖
Follow me on LinkedIn 👥

-------------------------------------------------------------------------------------------------------------
If I've helped solve your query, kindly mark my response as the solution ✔ and give it a thumbs up!👍 Your feedback supports future seekers 🚀

[Michael061297]

Hi @Deenuji 

I have a long list of entities that sometimes don't really exist for example if I type test entity in the entity name textbox it will say "Business Name TestEntity was not found" That's where my subflow will come up and will return an unable to search and there are times that If I type "123 MAIN ST LLC" instead of returning that details on the screenshot that you sent it will just refresh and the 123 MAIN ST LLC that I typed in the textbox will disappear making it seem like the wholepage did reload after I click search. That's why I want my automation to be aware that the webpage has reloaded instead of searching for 123 MAIN ST LLC that way I can create a logic to decrease the variable by 1 so it will search for that entity again and have the correct result instead of "Unable to search"