Unable to extract datas more that 30 records from webpage because of infinite scrolling component or scroll down page

[PAuserFromFranc]

Dear All,

On this website https://global-industrie.com/fr/liste-des-exposants i'm unable to grab the datas i want because it is happening up to 30 records and then stops.

May someone help to see how can i extract the URLs of all records instead of the 30?

Previous topics was opened dealing with extraction but each time there is a small issue like this one which stop the flow.

@VJRmay we have your knowledge on that?

thanks

Frederic

[VJR]

This is how the infinite scrolling works manually.

Scroll the page and then it loads a certain number of records (about 30).

So I tried to mimic the same using the PAD flow, with the only difference of scrolling to the very end of the page.

In doing that it was found that after scrolling for 57 times with a delay of 3 seconds between each, then it shows all the records ie; 1709 rows.

If you think that in future there could be more records so you could increase the Loop no. of 57 to some higher number. There is no harm if you give it to a higher number even if there are no more records. But unnecessary processing time will increase.

You can also remove the first "Extract data" action because there is already one in the loop.

Starting from 0, the 1708th row is the last record with Z alphabet.

@@source: 'Recorder'
WebAutomation.LaunchChrome.AttachToChromeByUrl TabUrl: 'https://global-industrie.com/en/liste-des-exposants' AttachTimeout: 10 BrowserInstance=> Browser
WebAutomation.ExtractData.ExtractList BrowserInstance: Browser Control: $'''html > body > div:eq(0) > div:eq(1) > section > section > div > div > div:eq(1) > div > div > div:eq(0) > a''' ExtractionParameters: {[$'''div > span''', $'''Own Text''', $''''''] } PostProcessData: False TimeoutInSeconds: 60 ExtractedData=> OutputData
LOOP LoopIndex FROM 1 TO 57 STEP 1
    WebAutomation.ExecuteJavascript BrowserInstance: Browser Javascript: $'''function ExecuteScript() 
{
window.scrollTo(0, document.body.scrollHeight);
}''' Result=> Result
    WAIT 3
    WebAutomation.ExtractData.ExtractList BrowserInstance: Browser Control: $'''html > body > div:eq(0) > div:eq(1) > section > section > div > div > div:eq(1) > div > div > div:eq(0) > a''' ExtractionParameters: {[$'''div > span''', $'''Own Text''', $''''''] } PostProcessData: False TimeoutInSeconds: 60 ExtractedData=> OutputData
END
Display.ShowMessageDialog.ShowMessage Message: $'''COMPLETE''' Icon: Display.Icon.None Buttons: Display.Buttons.OK DefaultButton: Display.DefaultButton.Button1 IsTopMost: False ButtonPressed=> ButtonPressed

# [ControlRepository][PowerAutomateDesktop]

{
  "ControlRepositorySymbols": [],
  "ImageRepositorySymbol": {
    "Name": "imgrepo",
    "ImportMetadata": {},
    "Repository": "{\r\n  \"Folders\": [],\r\n  \"Images\": [],\r\n  \"Version\": 1\r\n}"
  }
}

Paste the above flow in a new editor and test it. Once working fine then add it to your original flow.

Currently it takes 5 mins to grab the entire records but is far much better than not being able to fetch the records at all.

[PAuserFromFranc]

@VJRHello VJR, i don't understand why it's not working for me the solution you provide...

It seems the page is not scrolling down with the script...and after 57 loop, i'm still having 30 records, don't forget i'm retreiving the URLs maybe it's différent for the h or div tag o?

[PAuserFromFranc]

Sorry @VJR my missunderstood, i just figure out i only need to copy paste the window.scroll function ....

now it's working well, thank agin for all those tips!

may i ask you what's your job? if we can keep in touch via whatsapp (0033652022485) , and are u guy working for Microsoft or are u MVP? all those questions because i might have job opportunity ...what is your speciality in coding? which languages and for what?

regards,

Fred

[PAuserFromFranc]

I'm really sorry @VJR but now access to get the company card request a JWT Token and even logged in i'm unable to get the cards

do you have a way to have access? then i can proceed my flow? it was not the case few days before and now struggling to do my database ...cheers