How to get past hCaptcha asking to select similar images ?

[vivekthoppil]

While implementing an automated screenshot taking logic for a Url, the site is asking for selecting similar images possibly due to the detection of bot by hCaptcha . I am using uc mode.

Manually navigating to the Url in the browser is working.

Code

    from seleniumbase import SB
    fallback_ua = 'Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36'

    chromium_arg = '--disable-gpu,--disable-blink-features=AutomationControlled'
    with SB(uc=True, headless2=True, agent=fallback_ua, chromium_arg=chromium_arg) as sb:
        url = "https://www.royalmail.com/track-your-item#/tracking-results/QF085212272GB"
        sb.uc_open_with_reconnect(url, 3)
        sb.save_screenshot('myimage.png')

Result

Dockerized environment: python3.9-slim

seleniumbase: 4.28.0

Chromium Browser: Chromium 126.0.6478.126 built on Debian 12.5, running on Debian 12.5

Chrome Driver: ChromeDriver 126.0.6478.126 (d36ace6122e0a59570e258d82441395206d60e1c-refs/branch-heads/6478@{#1591})

[mdmintz]

It looks like you're changing things that you shouldn't be. Use this instead:

from seleniumbase import SB

with SB(uc=True) as sb:
    url = "https://www.royalmail.com/track-your-item#/tracking-results/QF085212272GB"
    sb.uc_open_with_reconnect(url, 4)
    sb.save_screenshot("myimage.png")

Don't use headless mode, don't change the default user-agent, and don't set chromium args in UC Mode.

[vivekthoppil]

I deployed a Flask app in heroku which returns the screenshot of a url.

@app.route('/test')
def test():
    from seleniumbase import SB
    with SB(uc=True) as sb:
        url = "https://www.royalmail.com/track-your-item#/tracking-results/QF085212272GB"
        sb.uc_open_with_reconnect(url, 4)
        sb.save_screenshot('myimage.png')
    
        return send_file('./myimage.png', mimetype='image/png')

The above code gave an error

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: cannot connect to chrome at 127.0.0.1:9222

The log contained ( not the error log )

X11 display failed! Will use regular xvfb!

I tried your comment here and put xvfb=True ( which did not work ) followed by headless2=True . The issue with chrome is now over, but now the SeleniumBase is getting detected by hCaptcha , which is the original issue.

The heroku-22 stack, which is the environment for the app is based on Ubuntu 22.04

[mdmintz]

It sounds like the Linux environment you used does not support the required X11 display.

To try launching the X11 display on your own, set pyautogui._pyautogui_x11._display to Xlib.display.Display(os.environ['DISPLAY']) on Linux in order to sync up pyautogui with the X11 virtual display after calling sbvirtualdisplay.Display(visible=True, size=(1366, 768), backend="xvfb", use_xauth=True).start(). (For Xlib, use import Xlib.display after pip install python-xlib.)

[vivekthoppil]

What exactly is os.environ['DISPLAY'] ? I dont see any env variables set in that name .

I am getting

2024-07-02T13:40:49.795786+00:00 app[web.1]: _display = Display(os.environ['DISPLAY'])
2024-07-02T13:40:49.795786+00:00 app[web.1]: ~~~~~~~~~~^^^^^^^^^^^
2024-07-02T13:40:49.795786+00:00 app[web.1]: File "", line 714, in getitem
2024-07-02T13:40:49.795786+00:00 app[web.1]: KeyError: 'DISPLAY'

Code

from sbvirtualdisplay import Display
import Xlib.display
import pyautogui

    try:
        disp = Display(visible=True, size=(1366, 768), backend="xvfb", use_xauth=True)
        disp.start()
        pyautogui._pyautogui_x11._display = Xlib.display.Display(os.environ['DISPLAY'])
        from seleniumbase import SB
        with SB(uc=True) as sb:
            sb.uc_open_with_reconnect(url, 4)
            sb.save_screenshot('myimage.png')
            return send_file('./myimage.png', mimetype='image/png')
    finally:
        disp.stop()

[mdmintz]

It is automatically set when running an X11 / Xvfb display on Linux. If the variable isn't there, it means the display didn't start. See https://github.com/ponty/PyVirtualDisplay for more info. SeleniumBase uses a modified fork called sbvirtualdisplay, but 90% of the code is the same.

[vivekthoppil]

I installed the below libraries

xvfb 
xauth
python3-tk

and ran the below code in both Docker environment and Heroku environment

    from seleniumbase import SB
    from sbvirtualdisplay import Display
    import Xlib.display

    disp = Display(visible=True, size=(1920, 1080), backend="xvfb", use_xauth=True)
    disp.start()
    try:
        import pyautogui
        pyautogui._pyautogui_x11._display = Xlib.display.Display(os.environ['DISPLAY'])
        with SB(uc=True) as sb:
            url = "https://www.royalmail.com/track-your-item#/tracking-results/QF085212272GB"
            sb.uc_open_with_reconnect(url, 4)
            sb.save_screenshot('myimage.png')
            return send_file('./myimage.png', mimetype='image/png')
    finally:
        disp.stop()

Docker -- > The program ran without error but the hCaptcha issue still persists. Something to do with docker ?

Heroku -- > Program did not run. Below exception is happening.

b.1]: Traceback (most recent call last):
2024-07-03T11:25:41.800524+00:00 app[web.1]:   File "/app/app.py", line 17, in test
2024-07-03T11:25:41.800525+00:00 app[web.1]:     import pyautogui
2024-07-03T11:25:41.800525+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.12/site-packages/pyautogui/__init__.py", line 246, in <module>
2024-07-03T11:25:41.800525+00:00 app[web.1]:     import mouseinfo
2024-07-03T11:25:41.800525+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.12/site-packages/mouseinfo/__init__.py", line 223, in <module>
2024-07-03T11:25:41.800525+00:00 app[web.1]:     _display = Display(os.environ['DISPLAY'])
2024-07-03T11:25:41.800525+00:00 app[web.1]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-03T11:25:41.800526+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.12/site-packages/Xlib/display.py", line 89, in __init__
2024-07-03T11:25:41.800526+00:00 app[web.1]:     self.display = _BaseDisplay(display)
2024-07-03T11:25:41.800526+00:00 app[web.1]:                    ^^^^^^^^^^^^^^^^^^^^^
2024-07-03T11:25:41.800527+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.12/site-packages/Xlib/display.py", line 71, in __init__
2024-07-03T11:25:41.800527+00:00 app[web.1]:     protocol_display.Display.__init__(self, *args, **keys)
2024-07-03T11:25:41.800527+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.12/site-packages/Xlib/protocol/display.py", line 89, in __init__
2024-07-03T11:25:41.800527+00:00 app[web.1]:     self.socket = connect.get_socket(name, protocol, host, displayno)
2024-07-03T11:25:41.800527+00:00 app[web.1]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-03T11:25:41.800528+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.12/site-packages/Xlib/support/connect.py", line 87, in get_socket
2024-07-03T11:25:41.800528+00:00 app[web.1]:     return mod.get_socket(dname, protocol, host, dno)
2024-07-03T11:25:41.800528+00:00 app[web.1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-03T11:25:41.800528+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.12/site-packages/Xlib/support/unix_connect.py", line 112, in get_socket
2024-07-03T11:25:41.800528+00:00 app[web.1]:     raise error.DisplayConnectionError(dname, str(val))
2024-07-03T11:25:41.800528+00:00 app[web.1]: Xlib.error.DisplayConnectionError: Can't connect to display ":1001": [Errno 111] Connection refused

Any idea why this is happening ?

Basically i just need to run SB in Linux environment.