Cannot bypass the check box of the CF of this website

[Orbiszeus]

This is where the iframe is located:
<

This is the website:

I am having a problem bypassing this CF checkbox in this website. I am using the newly updated functions. When the browser opens the newly updated uc_click() method which is now handled by uc_gui_handle_cf() does not work. What can I do to get through this? The iframe located in this id="cf-chl-widget-om6tb". My code structure looks like this :

  with SB(uc=True, test=True, rtf=True,incognito=True, agent=agent, headless=False) as sb:
      sb.driver.uc_open_with_disconnect(url, 6)
      # sb.scroll_to("iframe")
      sb.uc_gui_handle_cf("#cf-chl-widget-om6tb iframe")
      sb.set_messenger_theme(location="bottom_center")
      sb.post_message("SeleniumBase wasn't detected!")

Exception I am getting :

urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x111201e40>: Failed to establish a new connection: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/orbiszeus/.vscode/extensions/ms-python.debugpy-2024.6.0-darwin-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/Users/orbiszeus/.vscode/extensions/ms-python.debugpy-2024.6.0-darwin-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/Users/orbiszeus/.vscode/extensions/ms-python.debugpy-2024.6.0-darwin-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/Users/orbiszeus/.vscode/extensions/ms-python.debugpy-2024.6.0-darwin-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/orbiszeus/.vscode/extensions/ms-python.debugpy-2024.6.0-darwin-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/orbiszeus/.vscode/extensions/ms-python.debugpy-2024.6.0-darwin-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/Users/orbiszeus/metro_analyst/menu_crawler.py", line 570, in <module>
    main()
  File "/Users/orbiszeus/metro_analyst/menu_crawler.py", line 545, in main
    sb.uc_gui_handle_cf("#cf-chl-widget-om6tb iframe")
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/seleniumbase/core/browser_launcher.py", line 4031, in <lambda>
    lambda *args, **kwargs: uc_gui_handle_cf(
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/seleniumbase/core/browser_launcher.py", line 653, in uc_gui_handle_cf
    source = driver.get_page_source()
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/seleniumbase/core/sb_driver.py", line 50, in get_page_source
    return self.driver.page_source
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/seleniumbase/undetected/__init__.py", line 330, in __getattribute__
    return super().__getattribute__(item)
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 455, in page_source
    return self.execute(Command.GET_PAGE_SOURCE)["value"]
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 352, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/selenium/webdriver/remote/remote_connection.py", line 302, in execute
    return self._request(command_info[0], url, body=data)
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/selenium/webdriver/remote/remote_connection.py", line 322, in _request
    response = self._conn.request(method, url, body=body, headers=headers)
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 136, in request
    return self.request_encode_url(
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 183, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 444, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/Users/orbiszeus/metro_analyst/myenv/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=53964): Max retries exceeded with url: /session/33d97a286c7d6d58961e0ec9b48dcc5f/source (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x111201e40>: Failed to establish a new connection: [Errno 61] Connection refused'))

[mdmintz]

I'm running this:

from seleniumbase import SB

with SB(uc=True) as sb:
    url = "https://www.yemeksepeti.com/city/istanbul/area/besiktas-etiler-mah"
    sb.driver.uc_open_with_reconnect(url, 6)
    sb.uc_gui_handle_cf()
    sb.highlight('span[aria-label="yemeksepeti"]')
    breakpoint()

And seeing this:

---

seleniumbase version: 4.28.2
urllib3 version: 2.2.2

[Orbiszeus]

Thank you!!! You are the best. Saved me..

[Orbiszeus]

Even this discussion is closed, I have encountered another problem with the same site, I can get bypass the CF however during crawling the links inside the site, it is randomly asking Google Recapatcha with "Select all vehicles goes on water, etc." How can I crawl inside undetected? The one suggestion about the audio clicking on Google Captcha does not work. Can you please help me? Any suggestions or anything that you can add to my code:

def main_crawler(url):
    menu_items = []
    with SB(uc=True) as sb:
        sb.driver.uc_open_with_reconnect(url, 6)
        try:
            sb.uc_gui_handle_cf()
            print(len(sb.find_elements("li.vendor-tile-new-l a")))
            for i in range(10):
                sb.click_nth_visible_element("li.vendor-tile-new-l a", i)
                all_items = sb.find_elements("li[data-testid='menu-product']")
                for li in all_items:
                    product_name = li.find_element("css selector", "[data-testid='menu-product-name']").text
                    product_description = li.find_element("css selector", "[data-testid='menu-product-description']").text
                    product_price = li.find_element("css selector", "[data-testid='menu-product-price']").text            
                    menu_item = {
                        "Menu Item": product_name,
                        "Menu Ingredients": product_description,
                        "Price": product_price
                    }
                    menu_items.append(menu_item)
                sb.go_back()
                
        except Exception as e:
            print(f"Exception : {e}")
            logging.info("No CAPTCHA iframe found, assuming automatic verification")
    return menu_items  

[mdmintz]

Just use uc_open(url). That method didn't have the reconnect_time arg.

[Orbiszeus]

Oh okay!! What about how can I pass the Selenium Base driver into here:

import os,urllib,random,pydub,speech_recognition,time

class RecaptchaSolver:
    def __init__(self, driver):
        self.driver = driver
    
    def solveCaptcha(self):
        iframe_inner = self.driver("@title=reCAPTCHA")
        time.sleep(0.1)
        
        # Click on the recaptcha
        iframe_inner('.rc-anchor-content',timeout=1).click()
        self.driver.wait.ele_displayed("xpath://iframe[contains(@title, 'recaptcha')]",timeout=3)
          
        # Sometimes just clicking on the recaptcha is enough to solve it
        if self.isSolved():
            return
        
        # Get the new iframe
        iframe = self.driver("xpath://iframe[contains(@title, 'recaptcha')]")

        # Click on the audio button
        iframe('#recaptcha-audio-button',timeout=1).click()
        time.sleep(.3)
        
        # Get the audio source
        src = iframe('#audio-source').attrs['src']
        
        # Download the audio to the temp folder
        path_to_mp3 = os.path.normpath(os.path.join((os.getenv("TEMP") if os.name=="nt" else "/tmp/")+ str(random.randrange(1,1000))+".mp3"))
        path_to_wav = os.path.normpath(os.path.join((os.getenv("TEMP") if os.name=="nt" else "/tmp/")+ str(random.randrange(1,1000))+".wav"))
        
        urllib.request.urlretrieve(src, path_to_mp3)

        # Convert mp3 to wav
        sound = pydub.AudioSegment.from_mp3(path_to_mp3)
        sound.export(path_to_wav, format="wav")
        sample_audio = speech_recognition.AudioFile(path_to_wav)
        r = speech_recognition.Recognizer()
        with sample_audio as source:
            audio = r.record(source)
        
        # Recognize the audio
        key = r.recognize_google(audio)
        
        # Input the key
        iframe('#audio-response').input(key.lower())
        time.sleep(0.1)
        
        # Submit the key
        iframe('#audio-response').input(Keys.ENTER)
        time.sleep(.4)

        # Check if the captcha is solved
        if self.isSolved():
            return
        else:
            raise Exception("Failed to solve the captcha")

    def isSolved(self):
        try:
            return "style" in self.driver.ele(".recaptcha-checkbox-checkmark",timeout=1).attrs
        except:
            return False

And change it to be used for the SB driver? Because there are a lot of normal driver functions. Can you help me with this ? Thank you.

[mdmintz]

Getting the driver from sb is just: sb.driver

[Orbiszeus]

However, there are I think difference in the above code that a sb.driver cannot do or make it with different methods, right?

[mdmintz]

sb.driver has all the methods of the regular driver, plus some new ones back-ported from SeleniumBase.

[Orbiszeus]

Dear Micheal,
I am trying to access this site, however the cookies that are popping when I try to use sb.driver.uc_open_with_reconnect(url, 6) on this website the cookie that comes up just does not go away even though I am using the sb.click(...) to click on of the buttons on the cookie pop. Could you please help me with this?

[mdmintz]

The left one is 'button[data-testid="uc-save-button"]' and the right one is 'button[data-testid="uc-accept-all-button"]'.
Those will always appear because every time you spin up a new browser, it's a new session with no history, meaning that you have to accept cookies again.

[Orbiszeus]

They don't work however, I did this and worked : sb.execute_script("UC_UI.denyAllConsents().then(UC_UI.closeCMP);") #closing the cookies Thank you.

[Orbiszeus]

However, the only thing I couldn't figure is the RECAPTCHA part where I tried to change the Chromium driver code to SB code. It just does not work. This is my error. :
It looks like you are trying to call a SeleniumBase method
from outside the scope of your test class's self object,
which is initialized by calling BaseCase's setUp() method.
The self object is where all test variables are defined.
If you created a custom setUp() method (that overrided the
the default one), make sure to call super().setUp() in it.
When using page objects, be sure to pass the self object
from your test class into your page object methods so that
they can call BaseCase class methods with all the required
variables, which are initialized during the setUp() method
that runs automatically before all tests called by pytest.
The code looks like this, can you please help me :

import os
import random
import time
import urllib.request
import pydub
import speech_recognition
from selenium.webdriver.common.keys import Keys
from seleniumbase import BaseCase
from seleniumbase import SB as sb

class RecaptchaSolver():
     
    def __init__(self, driver:sb):
        super().__init__() 
        self.driver = driver

    def solveCaptcha(self):
        try :   
          self.driver.wait_for_element_present("@title=reCAPTCHA", timeout=3)
          iframe_inner = sb.get_element("@title=reCAPTCHA")
          sb.click(".rc-anchor-content")
          sb.wait_for_element_present("xpath://iframe[contains(@title, 'recaptcha')]", timeout=3)
          
          # Sometimes just clicking on the recaptcha is enough to solve it
          if self.isSolved():
               return
          
          # Get the new iframe
          iframe = sb.get_element("xpath://iframe[contains(@title, 'recaptcha')]")
          
          # Click on the audio button
          sb.click("#recaptcha-audio-button")
          time.sleep(0.3)
          
          # Get the audio source
          src = sb.get_attribute("#audio-source", "src")
          
          # Download the audio to the temp folder
          path_to_mp3 = os.path.normpath(os.path.join((os.getenv("TEMP") if os.name=="nt" else "/tmp/") + str(random.randrange(1,1000)) + ".mp3"))
          path_to_wav = os.path.normpath(os.path.join((os.getenv("TEMP") if os.name=="nt" else "/tmp/") + str(random.randrange(1,1000)) + ".wav"))
          
          urllib.request.urlretrieve(src, path_to_mp3)
          
          # Convert mp3 to wav
          sound = pydub.AudioSegment.from_mp3(path_to_mp3)
          sound.export(path_to_wav, format="wav")
          sample_audio = speech_recognition.AudioFile(path_to_wav)
          r = speech_recognition.Recognizer()
          with sample_audio as source:
               audio = r.record(source)
          
          # Recognize the audio
          key = r.recognize_google(audio)
          
          # Input the key
          sb.update_text("#audio-response", key.lower())
          time.sleep(0.1)
          
          # Submit the key
          sb.type("#audio-response", Keys.ENTER)
          time.sleep(0.4)
          
        except Exception as e:
             print(f"RECAPTCHA problem: {e}")
        # Check if the captcha is solved
        if self.isSolved():
            return
        else:
            raise Exception("Failed to solve the captcha")
    
    def isSolved(self):
        try:
            return "style" in sb.get_attribute(".recaptcha-checkbox-checkmark", "style")
        except:
            return False

[mdmintz]

SB must be initialized:

with SB() as sb:
    # ...

[nguyendinhsang1908]

It's not working when I use the same solution. The checkbox was checked, but it didn't bypass the Cloudflare.

from seleniumbase import SB
import time

with SB(uc=True) as sb:
    url = "https://www.top-rated.online/cities/Rotterdam/place/p/2227631/La+Pizza+Centrum#google_vignette"
    sb.open(url)
    sb.scroll_to('//*[@id="app"]/main/div/div[1]/article/section/b/a')
    sb.execute_script("document.querySelector('a.btn.btn-lg.btn-success').click()")
    sb.switch_to_window(1, timeout=None)

    #switch_frame
    sb.wait_for_element_visible('iframe[title*="challenge"]', timeout=15)
    sb.switch_to_frame('iframe[title*="challenge"]')
    time.sleep(5)
    
    sb.uc_gui_handle_cf()  # Ready if needed!
    time.sleep(20)
    current_url = sb.get_current_url()
    print(current_url)

[Orbiszeus]

I think you don't need to scroll to where your CF iframe is with xpath, you just have to got to the website and and sb.uc_gui_handle_cf() use this function. And that's it!

[nguyendinhsang1908]

mean I have used sb.uc_gui_handle_cf() but it doesn't work. It still clicks on the checkbox but cannot bypass cf.