Click on “Show more deals” in webpage with Selenium
up vote
-2
down vote
favorite
I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
I'm stuck having the following error:
AttributeError: 'NoneType' object has no attribute 'find_element'
My code is the following:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
driver = webdriver.Chrome(r'C:tempchromedriver.exe')
browser = driver.get(url)
while True:
button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
button.click()
Any idea?
python selenium web-scraping lazy-loading webdriverwait
add a comment |
up vote
-2
down vote
favorite
I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
I'm stuck having the following error:
AttributeError: 'NoneType' object has no attribute 'find_element'
My code is the following:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
driver = webdriver.Chrome(r'C:tempchromedriver.exe')
browser = driver.get(url)
while True:
button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
button.click()
Any idea?
python selenium web-scraping lazy-loading webdriverwait
add a comment |
up vote
-2
down vote
favorite
up vote
-2
down vote
favorite
I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
I'm stuck having the following error:
AttributeError: 'NoneType' object has no attribute 'find_element'
My code is the following:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
driver = webdriver.Chrome(r'C:tempchromedriver.exe')
browser = driver.get(url)
while True:
button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
button.click()
Any idea?
python selenium web-scraping lazy-loading webdriverwait
I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
I'm stuck having the following error:
AttributeError: 'NoneType' object has no attribute 'find_element'
My code is the following:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
driver = webdriver.Chrome(r'C:tempchromedriver.exe')
browser = driver.get(url)
while True:
button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
button.click()
Any idea?
python selenium web-scraping lazy-loading webdriverwait
python selenium web-scraping lazy-loading webdriverwait
edited Nov 21 at 9:06
DebanjanB
37.5k73373
37.5k73373
asked Nov 20 at 12:37
sammtt
609
609
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
up vote
1
down vote
accepted
To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/
you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
browser.get(url)
while True:
try:
browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
print("Button clicked")
except:
print("No more Buttons")
break
browser.quit()
Console Output:
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
No more Buttons
Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
– sammtt
Nov 20 at 13:43
@sammtt Check out my updated answer and let me know the status
– DebanjanB
Nov 20 at 13:58
add a comment |
up vote
0
down vote
Try the following which uses a CSS attribute = value selector to target the data-event-action
attribute of the button by its value
driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()
Replace driver
with browser
if required.
add a comment |
up vote
0
down vote
Try it like this:
while not re.search(r"Showing (d+) of 1 ", browser.page_source):
browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
time.sleep(1)
This avoids the selenium errors that will eventually drive you nuts.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53393158%2fclick-on-show-more-deals-in-webpage-with-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/
you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
browser.get(url)
while True:
try:
browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
print("Button clicked")
except:
print("No more Buttons")
break
browser.quit()
Console Output:
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
No more Buttons
Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
– sammtt
Nov 20 at 13:43
@sammtt Check out my updated answer and let me know the status
– DebanjanB
Nov 20 at 13:58
add a comment |
up vote
1
down vote
accepted
To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/
you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
browser.get(url)
while True:
try:
browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
print("Button clicked")
except:
print("No more Buttons")
break
browser.quit()
Console Output:
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
No more Buttons
Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
– sammtt
Nov 20 at 13:43
@sammtt Check out my updated answer and let me know the status
– DebanjanB
Nov 20 at 13:58
add a comment |
up vote
1
down vote
accepted
up vote
1
down vote
accepted
To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/
you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
browser.get(url)
while True:
try:
browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
print("Button clicked")
except:
print("No more Buttons")
break
browser.quit()
Console Output:
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
No more Buttons
To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/
you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
browser.get(url)
while True:
try:
browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
print("Button clicked")
except:
print("No more Buttons")
break
browser.quit()
Console Output:
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
Button clicked
No more Buttons
edited Nov 20 at 13:58
answered Nov 20 at 13:29
DebanjanB
37.5k73373
37.5k73373
Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
– sammtt
Nov 20 at 13:43
@sammtt Check out my updated answer and let me know the status
– DebanjanB
Nov 20 at 13:58
add a comment |
Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
– sammtt
Nov 20 at 13:43
@sammtt Check out my updated answer and let me know the status
– DebanjanB
Nov 20 at 13:58
Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
– sammtt
Nov 20 at 13:43
Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
– sammtt
Nov 20 at 13:43
@sammtt Check out my updated answer and let me know the status
– DebanjanB
Nov 20 at 13:58
@sammtt Check out my updated answer and let me know the status
– DebanjanB
Nov 20 at 13:58
add a comment |
up vote
0
down vote
Try the following which uses a CSS attribute = value selector to target the data-event-action
attribute of the button by its value
driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()
Replace driver
with browser
if required.
add a comment |
up vote
0
down vote
Try the following which uses a CSS attribute = value selector to target the data-event-action
attribute of the button by its value
driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()
Replace driver
with browser
if required.
add a comment |
up vote
0
down vote
up vote
0
down vote
Try the following which uses a CSS attribute = value selector to target the data-event-action
attribute of the button by its value
driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()
Replace driver
with browser
if required.
Try the following which uses a CSS attribute = value selector to target the data-event-action
attribute of the button by its value
driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()
Replace driver
with browser
if required.
answered Nov 20 at 12:45
QHarr
28.7k81839
28.7k81839
add a comment |
add a comment |
up vote
0
down vote
Try it like this:
while not re.search(r"Showing (d+) of 1 ", browser.page_source):
browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
time.sleep(1)
This avoids the selenium errors that will eventually drive you nuts.
add a comment |
up vote
0
down vote
Try it like this:
while not re.search(r"Showing (d+) of 1 ", browser.page_source):
browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
time.sleep(1)
This avoids the selenium errors that will eventually drive you nuts.
add a comment |
up vote
0
down vote
up vote
0
down vote
Try it like this:
while not re.search(r"Showing (d+) of 1 ", browser.page_source):
browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
time.sleep(1)
This avoids the selenium errors that will eventually drive you nuts.
Try it like this:
while not re.search(r"Showing (d+) of 1 ", browser.page_source):
browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
time.sleep(1)
This avoids the selenium errors that will eventually drive you nuts.
answered Nov 21 at 1:49
pguardiario
35.7k979112
35.7k979112
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53393158%2fclick-on-show-more-deals-in-webpage-with-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown