CSS Selector help for Extract data from web page advanced settings

[jfk86d]

I am trying to select the price below ($64.99) in the div element and the CSS selector retrieves the contents of the span element as well ($64.99reg $71.99). How do I modify my CSS selector term (and Attribute??) to only get the price ($64.99)?

<div class="prices f23442L">$64.99<span class="Pricereg b98723s">reg $71.99</span></div>

This is the current CSS selector and Attribute.

CSS selector: html > body > div:eq(1) > main > div > div:eq(2) > form > div:eq(1) > div:eq(0)
Attribute: Own Text

Thank you for sharing your expertise and help.

John

[BLawless]

Without looking at the element in the context of the page it's in, the first idea that comes to mind would be to use some regex parsing of the text that's captured and supply that as an argument in the third column for this element (I'd need to test it but I think something similar to "^(\$\d.\.\d{2})" would be a good starting point).

You could also use the split text action after extracting and split on a regex of any letter (or space if there's a space between "$64.99" and "reg") and then reference the first index in the resultant variable.  There is likely still a way to do this with the CSS selector alone but I'd need to see the site to be able to test it. 

If you would like to share the URL to the webpage, or a sample webpage anyways that contains this element, it would better help me with identifying the issue with the selector (if you need you are welcome to PM me with that information). 

[jfk86d]

Hi @BLawless ,

Thanks for the reply. I like the split suggestion as Plan B. Here is the URL.

https://www.petco.com/shop/en/petcostore/product/purina-pro-plan-focus-sensitive-skin-and-stomach-sa...

Thanks,

John

[BLawless · Accepted solution]

Hey @jfk86d, 
So I have bad news, OK news, and some interesting news.

The bad news is that I recall a similar case of where a div had text shared inline with a span containing more text, and I couldn't find a way to get the text I wanted by itself using CSS selectors alone(much the same way we're trying here).

The OK news is that regex should work (of course you'll need to do some more validation to check prices on other sample pages, you know how online storefronts can be with different CSS layouts...). Here's a picture of the settings I used to make it possible (with resultant outputs): 

For reference:

    CSS selector: html > body > div:eq(1) > main > div > div:eq(2) > form > div:eq(1) > div:eq(0)
    Attribute: Own Text
    Regex: ^\$\d.\.\d{2}

Just in case I've included a second variation to the regex that also worked for this case but again do some testing on the variations that will be presented (I'm doing this with another very large online store so I'm very familiar with the tendency for product listing pages to be incredibly inconsistent):

    Regex: ^\$\d*\.?\d*

While it was desirable to try and parse out only the numeric value without the currency symbol (so you can save yourself a step), every attempt I made at regex capture specification doesn't seem to play well with PAD (I think they use a slightly different syntax than I'm used to). 

Now here's the interesting news. You can actually very easily do this with the "Run JavaScript function on web page" function. I did some fudging around with the dev console in Firefox and Chrome and managed to get the following function pieced together: 

function GetPrice() {
    let price = document.querySelector('div.PriceRowContainer-sc-1wlo6zv-1:nth-child(1)').childNodes[0].textContent;
    return price;
}

This successfully returned the value in question, and if you wanted you could trim out the dollar sign first before returning the value.

document.querySelector('div.PriceRowContainer-sc-1wlo6zv-1:nth-child(1)').childNodes[0].textContent.replace('\$','');

This is a little bit outside of the general scope of what PAD is designed for but I find the JavaScript function to be a lot more reliable for some cases. The only concern I have here is the slew of seemingly random characters shown after "div.PriceRowContainer-sc" but again I tested this on a couple different pages and it didn't give me any issues. 

If anyone else that is more gifted than I about reducing this back to just using the CSS selector, I'd love to hear their approach as problems like this have been a nightmare for my data validation. 

Below I have included the text version of a sample flow, you should be able to select it, copy, and then paste it into a flow editor and it will import all of the actions and their settings to run for yourself and verify. (Actually this doesn't work... which is a bit of a bummer really, see my edit down below).
------------------------------- FLOW STARTS BELOW THIS LINE-------------------------------

WebAutomation.LaunchFirefox Url: $'''https://www.petco.com/shop/en/petcostore/product/purina-pro-plan-focus-sensitive-skin-and-stomach-salmon-and-rice-formula-adult-dry-dog-food''' WindowState: WebAutomation.BrowserWindowState.Normal ClearCache: False ClearCookies: False Timeout: 60 BrowserInstance=> Browser
WebAutomation.DataExtraction.ExtractSingleValue BrowserInstance: Browser ExtractionParameters: {[$'''html > body > div:eq(1) > main > div > div:eq(2) > form > div:eq(1) > div:eq(0)''', $'''Own Text''', $'''^\\$\\d.\\.\\d{2}'''] } ExtractedData=> arrayPriceUsingExtractorRegexA
SET stringA TO arrayPriceUsingExtractorRegexA[0][0]
WebAutomation.DataExtraction.ExtractSingleValue BrowserInstance: Browser ExtractionParameters: {[$'''html > body > div:eq(1) > main > div > div:eq(2) > form > div:eq(1) > div:eq(0)''', $'''Own Text''', $'''^\\$\\d*\\.?\\d*'''] } ExtractedData=> arrayPriceUsingExtractorRegexB
SET stringB TO arrayPriceUsingExtractorRegexB[0][0]
WebAutomation.ExecuteJavascript BrowserInstance: Browser Javascript&colon; $'''function GetPrice() {
let price = document.querySelector(\'div.PriceRowContainer-sc-1wlo6zv-1:nth-child(1)\').childNodes[0].textContent;
return price;
}''' Result=> stringPriceUsingJavaScript
WebAutomation.ExecuteJavascript BrowserInstance: Browser Javascript&colon; $'''function GetPrice() {
let price =document.querySelector(\'div.PriceRowContainer-sc-1wlo6zv-1:nth-child(1)\').childNodes[0].textContent.replace(\'\\$\',\'\');
return price;
}''' Result=> stringPriceUsingJavaScriptWithoutDollarSign
Text.ToNumber Text: stringPriceUsingJavaScriptWithoutDollarSign Number=> TextAsNumber
WebAutomation.CloseWebBrowser BrowserInstance: Browser
-------------------------------- FLOW ENDS ABOVE THIS LINE--------------------------------
See below for the example flow in the editor with the results of the last run performed. 

As a note here I'm gonna fiddle around with this to make it look a prettier. Just need to get it submitted first so the draft is saved. But please let me know if you have any questions about any of this. Over the last several months I've grown very familiar with the kinds of automations that are required when interfacing with online store fronts. 

Edit: So I recognize that the text gets re-encoded when it's posted into the html so it's not possible to directly copy and paste from here (though I've had no issue copying and pasting to and from text editors when trying to modify some of my flows at a code level) and I know the coding syntax of this can be a bit of a headache to follow. If you need more details on the flow setup I used for my solution feel free to PM me and I'll go over everything I did. 

[jfk86d]

Hi @BLawless ,

Thank you very much for your expert advice. I used the RegEx solution and it is working well. Thank you very much!

[BLawless]

Hey @jfk86d,
Happy to help! I'm sure someone else will come along with a better solution, but in the meantime if you wanna mark my explanation post as a solution so others can easily reference it, feel free to do so! 
Whoops, you were already on top of that. Thank you!

[kwikretails]

Hi @BLawless ,

Thanks for the reply. I like the split suggestion as Plan B. Here is the URL.

https://www.kwikpets.com/collections/dog-vitamins-supplements

Thanks,

John