Load More Button is Not Working

[CPee]

I've created a flow to allow continuous webscraping when the button "Load More" is available however, it is not working as intended and does not proceed to the next page. Is there a workaround for this? Thank you.

[Agnius]

The reason why it only returns the original 48 items is because the items in the page are provided in sections that correspond to the result pages. When you click on "Load more", a new <div> element appears that contains further results:

The first <div> contains the first 48 items, and the second one contains what was loaded. And you might get more of those, depending on the total number of results.

The correct way to work around this is to remove the index of the <div> element from your base selector in Extract data from web page.

The correct base selector should be as follows:

html > body > div:eq(0) > div > main > div:eq(0) > div:eq(2) > div:eq(3) > div:eq(1) > div:eq(1) > div > div

And it goes here:

By default, it has a div:eq(0) as the second-to-last element in the string. This will result in it only capturing the first page of the results. As soon as you remove the :eq(0) part from there, it will parse all sibling divs. So, once you have clicked through the "Load more" button enough to load all results, using this as a base selector in your extraction will return all results.

Make sure you don't actually use the extraction within a loop, as you only need to extract the data once. You just need to click the "Load more" button in a loop until it's no longer there. Then you extract all results at once and write them to Excel.

[CPee]

This looks great! I don't have Premium license though and it may be the reason why I can't copy and paste your code. I don't know how to place the code on the Run .NET script.

[Agnius]

Not having Premium does not prevent from pasting code, unless the code includes Premium actions (which in this case it doesn't). The reason you cannot paste it is likely because you're using an older version of PAD than the one that the code originates from, and the syntax has changed. Try pasting individual actions instead of the entire blob.

[Deenuji]

@CPee 

Please find the below screenshot and code what i have used in .net scripts:

Click on the Edit and create parameters like mentioned in the screenshot

.Net Code

// Merge data into the same column of dataTable1
        foreach (DataRow row in MergeTable.Rows)
        {
            DataTable.Rows.Add(row[0], row[1]);
        }

I am wondering why copy/paste is not working for you. As @Agnius  ,mentioned it won't require any premium license. In case while you create your PAD if you enabled power fx, pls disable that and recreate and just do normal copy/paste.

Thanks,
Deenuji Loganathan 👩‍💻
Automation Evangelist 🤖
Follow me on LinkedIn 👥

-------------------------------------------------------------------------------------------------------------
If I've helped solve your query, kindly mark my response as the solution ✔ and give it a thumbs up!👍 Your feedback supports future seekers 🚀

[Deenuji · Accepted solution]

@CPee 

This is how my copy and paste works in my machine. Just for your reference, please make sure you are doing the same stuffs.

Thanks,
Deenuji Loganathan 👩‍💻
Automation Evangelist 🤖
Follow me on LinkedIn 👥

-------------------------------------------------------------------------------------------------------------
If I've helped solve your query, kindly mark my response as the solution ✔ and give it a thumbs up!👍 Your feedback supports future seekers 🚀

[CPee]

@Deenuji thanks for this, I have copied it successfully on my PAD. It is now running successfully. I also replaced Base Selector CSS as recommended by @Agnius. However, when I tried saving the DataFromWebPage to Excel, it only shows a total of 32 items out of 85.

[Deenuji]

@CPee 

I believe there might be some confusion with both suggestions, as each solution would work in a different manner.

Please connect with me through LinkedIn or private message.

Thanks,
Deenuji Loganathan 👩‍💻
Automation Evangelist 🤖
Follow me on LinkedIn 👥

-------------------------------------------------------------------------------------------------------------
If I've helped solve your query, kindly mark my response as the solution ✔ and give it a thumbs up!👍 Your feedback supports future seekers 🚀

[CPee]

It is working now! It created two different Excel file with 48 items from page 1 and the other with 37 items from pag 2!

[Deenuji]

@CPee 

Your use case posed one of the toughest challenges, but I'm very glad and proud that we were able to figure it out together! Happy Automation!!

Thanks,
Deenuji Loganathan 👩‍💻
Automation Evangelist 🤖
Follow me on LinkedIn 👥

-------------------------------------------------------------------------------------------------------------
If I've helped solve your query, kindly mark my response as the solution ✔ and give it a thumbs up!👍 Your feedback supports future seekers 🚀