Click on “Show more deals” in webpage with Selenium











up vote
-2
down vote

favorite












I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
I'm stuck having the following error:



 AttributeError: 'NoneType' object has no attribute 'find_element'


My code is the following:



from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"

driver = webdriver.Chrome(r'C:tempchromedriver.exe')

browser = driver.get(url)
while True:
button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
button.click()


Any idea?










share|improve this question




























    up vote
    -2
    down vote

    favorite












    I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
    I'm stuck having the following error:



     AttributeError: 'NoneType' object has no attribute 'find_element'


    My code is the following:



    from selenium import webdriver
    from selenium.webdriver.support.wait import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.by import By

    url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"

    driver = webdriver.Chrome(r'C:tempchromedriver.exe')

    browser = driver.get(url)
    while True:
    button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
    button.click()


    Any idea?










    share|improve this question


























      up vote
      -2
      down vote

      favorite









      up vote
      -2
      down vote

      favorite











      I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
      I'm stuck having the following error:



       AttributeError: 'NoneType' object has no attribute 'find_element'


      My code is the following:



      from selenium import webdriver
      from selenium.webdriver.support.wait import WebDriverWait
      from selenium.webdriver.support import expected_conditions as EC
      from selenium.webdriver.common.by import By

      url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"

      driver = webdriver.Chrome(r'C:tempchromedriver.exe')

      browser = driver.get(url)
      while True:
      button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
      button.click()


      Any idea?










      share|improve this question















      I'd like to click on every 'Show 10 more deals' on the following page: "https://www.uswitch.com/broadband/compare/deals_and_offers/" but it does not seem to work.
      I'm stuck having the following error:



       AttributeError: 'NoneType' object has no attribute 'find_element'


      My code is the following:



      from selenium import webdriver
      from selenium.webdriver.support.wait import WebDriverWait
      from selenium.webdriver.support import expected_conditions as EC
      from selenium.webdriver.common.by import By

      url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"

      driver = webdriver.Chrome(r'C:tempchromedriver.exe')

      browser = driver.get(url)
      while True:
      button = WebDriverWait(browser,10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, 'Show 10 more deals')))
      button.click()


      Any idea?







      python selenium web-scraping lazy-loading webdriverwait






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 21 at 9:06









      DebanjanB

      37.5k73373




      37.5k73373










      asked Nov 20 at 12:37









      sammtt

      609




      609
























          3 Answers
          3






          active

          oldest

          votes

















          up vote
          1
          down vote



          accepted










          To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/ you can use the following solution:





          • Code Block:



            from selenium import webdriver
            from selenium.webdriver.support.wait import WebDriverWait
            from selenium.webdriver.support import expected_conditions as EC
            from selenium.webdriver.common.by import By

            url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
            options = webdriver.ChromeOptions()
            options.add_argument("start-maximized")
            options.add_argument('disable-infobars')
            browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
            browser.get(url)
            while True:
            try:
            browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
            browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
            print("Button clicked")
            except:
            print("No more Buttons")
            break
            browser.quit()



          • Console Output:



            Button clicked
            Button clicked
            Button clicked
            Button clicked
            Button clicked
            Button clicked
            Button clicked
            Button clicked
            Button clicked
            Button clicked
            No more Buttons







          share|improve this answer























          • Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
            – sammtt
            Nov 20 at 13:43












          • @sammtt Check out my updated answer and let me know the status
            – DebanjanB
            Nov 20 at 13:58


















          up vote
          0
          down vote













          Try the following which uses a CSS attribute = value selector to target the data-event-action attribute of the button by its value



          driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()


          Replace driver with browser if required.






          share|improve this answer




























            up vote
            0
            down vote













            Try it like this:



            while not re.search(r"Showing (d+) of 1 ", browser.page_source):
            browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
            time.sleep(1)


            This avoids the selenium errors that will eventually drive you nuts.






            share|improve this answer





















              Your Answer






              StackExchange.ifUsing("editor", function () {
              StackExchange.using("externalEditor", function () {
              StackExchange.using("snippets", function () {
              StackExchange.snippets.init();
              });
              });
              }, "code-snippets");

              StackExchange.ready(function() {
              var channelOptions = {
              tags: "".split(" "),
              id: "1"
              };
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function() {
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled) {
              StackExchange.using("snippets", function() {
              createEditor();
              });
              }
              else {
              createEditor();
              }
              });

              function createEditor() {
              StackExchange.prepareEditor({
              heartbeatType: 'answer',
              convertImagesToLinks: true,
              noModals: true,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: 10,
              bindNavPrevention: true,
              postfix: "",
              imageUploader: {
              brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
              contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
              allowUrls: true
              },
              onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              });


              }
              });














              draft saved

              draft discarded


















              StackExchange.ready(
              function () {
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53393158%2fclick-on-show-more-deals-in-webpage-with-selenium%23new-answer', 'question_page');
              }
              );

              Post as a guest















              Required, but never shown

























              3 Answers
              3






              active

              oldest

              votes








              3 Answers
              3






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes








              up vote
              1
              down vote



              accepted










              To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/ you can use the following solution:





              • Code Block:



                from selenium import webdriver
                from selenium.webdriver.support.wait import WebDriverWait
                from selenium.webdriver.support import expected_conditions as EC
                from selenium.webdriver.common.by import By

                url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
                options = webdriver.ChromeOptions()
                options.add_argument("start-maximized")
                options.add_argument('disable-infobars')
                browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
                browser.get(url)
                while True:
                try:
                browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
                browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
                print("Button clicked")
                except:
                print("No more Buttons")
                break
                browser.quit()



              • Console Output:



                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                No more Buttons







              share|improve this answer























              • Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
                – sammtt
                Nov 20 at 13:43












              • @sammtt Check out my updated answer and let me know the status
                – DebanjanB
                Nov 20 at 13:58















              up vote
              1
              down vote



              accepted










              To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/ you can use the following solution:





              • Code Block:



                from selenium import webdriver
                from selenium.webdriver.support.wait import WebDriverWait
                from selenium.webdriver.support import expected_conditions as EC
                from selenium.webdriver.common.by import By

                url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
                options = webdriver.ChromeOptions()
                options.add_argument("start-maximized")
                options.add_argument('disable-infobars')
                browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
                browser.get(url)
                while True:
                try:
                browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
                browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
                print("Button clicked")
                except:
                print("No more Buttons")
                break
                browser.quit()



              • Console Output:



                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                No more Buttons







              share|improve this answer























              • Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
                – sammtt
                Nov 20 at 13:43












              • @sammtt Check out my updated answer and let me know the status
                – DebanjanB
                Nov 20 at 13:58













              up vote
              1
              down vote



              accepted







              up vote
              1
              down vote



              accepted






              To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/ you can use the following solution:





              • Code Block:



                from selenium import webdriver
                from selenium.webdriver.support.wait import WebDriverWait
                from selenium.webdriver.support import expected_conditions as EC
                from selenium.webdriver.common.by import By

                url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
                options = webdriver.ChromeOptions()
                options.add_argument("start-maximized")
                options.add_argument('disable-infobars')
                browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
                browser.get(url)
                while True:
                try:
                browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
                browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
                print("Button clicked")
                except:
                print("No more Buttons")
                break
                browser.quit()



              • Console Output:



                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                No more Buttons







              share|improve this answer














              To click on the element with text as Show 10 more deals on the page https://www.uswitch.com/broadband/compare/deals_and_offers/ you can use the following solution:





              • Code Block:



                from selenium import webdriver
                from selenium.webdriver.support.wait import WebDriverWait
                from selenium.webdriver.support import expected_conditions as EC
                from selenium.webdriver.common.by import By

                url = "https://www.uswitch.com/broadband/compare/deals_and_offers/"
                options = webdriver.ChromeOptions()
                options.add_argument("start-maximized")
                options.add_argument('disable-infobars')
                browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
                browser.get(url)
                while True:
                try:
                browser.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(browser,20).until(EC.visibility_of_element_located((By.XPATH, "//button[@class='us-btn us-btn--action' and contains(.,'Show 10 more deals')]"))))
                browser.execute_script("arguments[0].click();", WebDriverWait(browser,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.us-btn.us-btn--action[name='visible_products']"))))
                print("Button clicked")
                except:
                print("No more Buttons")
                break
                browser.quit()



              • Console Output:



                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                Button clicked
                No more Buttons








              share|improve this answer














              share|improve this answer



              share|improve this answer








              edited Nov 20 at 13:58

























              answered Nov 20 at 13:29









              DebanjanB

              37.5k73373




              37.5k73373












              • Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
                – sammtt
                Nov 20 at 13:43












              • @sammtt Check out my updated answer and let me know the status
                – DebanjanB
                Nov 20 at 13:58


















              • Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
                – sammtt
                Nov 20 at 13:43












              • @sammtt Check out my updated answer and let me know the status
                – DebanjanB
                Nov 20 at 13:58
















              Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
              – sammtt
              Nov 20 at 13:43






              Thank you. I do get the following error eventually: WebDriverException: unknown error: Element <button class="us-btn us-btn--action" name="visible_products" value="53" data-event="true" data-event-category="Responsive Compare" data-event-action="Show 10 more products" data-event-label="Show 10 more products" data-event-value="43">...</button> is not clickable at point (981, 920). Other element would receive the click: <div class="us-loader__overlay"></div>
              – sammtt
              Nov 20 at 13:43














              @sammtt Check out my updated answer and let me know the status
              – DebanjanB
              Nov 20 at 13:58




              @sammtt Check out my updated answer and let me know the status
              – DebanjanB
              Nov 20 at 13:58












              up vote
              0
              down vote













              Try the following which uses a CSS attribute = value selector to target the data-event-action attribute of the button by its value



              driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()


              Replace driver with browser if required.






              share|improve this answer

























                up vote
                0
                down vote













                Try the following which uses a CSS attribute = value selector to target the data-event-action attribute of the button by its value



                driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()


                Replace driver with browser if required.






                share|improve this answer























                  up vote
                  0
                  down vote










                  up vote
                  0
                  down vote









                  Try the following which uses a CSS attribute = value selector to target the data-event-action attribute of the button by its value



                  driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()


                  Replace driver with browser if required.






                  share|improve this answer












                  Try the following which uses a CSS attribute = value selector to target the data-event-action attribute of the button by its value



                  driver.find_element_by_css_selector('[data-event-action="Show 10 more products"]').click()


                  Replace driver with browser if required.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered Nov 20 at 12:45









                  QHarr

                  28.7k81839




                  28.7k81839






















                      up vote
                      0
                      down vote













                      Try it like this:



                      while not re.search(r"Showing (d+) of 1 ", browser.page_source):
                      browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
                      time.sleep(1)


                      This avoids the selenium errors that will eventually drive you nuts.






                      share|improve this answer

























                        up vote
                        0
                        down vote













                        Try it like this:



                        while not re.search(r"Showing (d+) of 1 ", browser.page_source):
                        browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
                        time.sleep(1)


                        This avoids the selenium errors that will eventually drive you nuts.






                        share|improve this answer























                          up vote
                          0
                          down vote










                          up vote
                          0
                          down vote









                          Try it like this:



                          while not re.search(r"Showing (d+) of 1 ", browser.page_source):
                          browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
                          time.sleep(1)


                          This avoids the selenium errors that will eventually drive you nuts.






                          share|improve this answer












                          Try it like this:



                          while not re.search(r"Showing (d+) of 1 ", browser.page_source):
                          browser.execute_script("document.querySelector('[data-event-label="Show 10 more products"]').click()")
                          time.sleep(1)


                          This avoids the selenium errors that will eventually drive you nuts.







                          share|improve this answer












                          share|improve this answer



                          share|improve this answer










                          answered Nov 21 at 1:49









                          pguardiario

                          35.7k979112




                          35.7k979112






























                              draft saved

                              draft discarded




















































                              Thanks for contributing an answer to Stack Overflow!


                              • Please be sure to answer the question. Provide details and share your research!

                              But avoid



                              • Asking for help, clarification, or responding to other answers.

                              • Making statements based on opinion; back them up with references or personal experience.


                              To learn more, see our tips on writing great answers.





                              Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                              Please pay close attention to the following guidance:


                              • Please be sure to answer the question. Provide details and share your research!

                              But avoid



                              • Asking for help, clarification, or responding to other answers.

                              • Making statements based on opinion; back them up with references or personal experience.


                              To learn more, see our tips on writing great answers.




                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function () {
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53393158%2fclick-on-show-more-deals-in-webpage-with-selenium%23new-answer', 'question_page');
                              }
                              );

                              Post as a guest















                              Required, but never shown





















































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown

































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown







                              Popular posts from this blog

                              404 Error Contact Form 7 ajax form submitting

                              How to know if a Active Directory user can login interactively

                              TypeError: fit_transform() missing 1 required positional argument: 'X'