Seleniumbase within webapp in heroku

[Kamran-ov]

Hi,

Does anybody have experience in using seleniumbase within a Flask web app deployed in Heroku platform?

I have tried pythonanywhere but after paying the subscription and testing it, it didn't work out for me.
I just want to know if it works before paying for the service in heroku as well.

If somebody could share his/her experience would appreciate it.

Thank you

[mdmintz]

People have used SeleniumBase within Flask. Specify options from within one of the pure Python Syntax Formats: #1986 (comment)

[Kamran-ov]

In the end deployed to a dummy account in Heroku but for some reason did not make it work.
I received the below error in Heroku and it is the first time trying to use Selenium in Linux environment.
Any suggestion on why it can not initiate Selenium and what can I do to make it work?

App runs with
python-3.12.1
set up with gunicorn and flask

My Selenium base initiator (script) within webapp:

from seleniumbase import Driver

driver = Driver(uc=True,
                headless2=True,  # Headless mode
                log_cdp=True,  # Log Events
                ad_block_on=True,  # Ad-blocking
                block_images=True,  # block images
                proxy=None,  # Add proxy
                )
# Extend the size of xhr Request data_temp received
driver.execute_cdp_cmd("Network.enable", {
    'maxResourceBufferSize': 1024 * 1204 * 100,
    'maxTotalBufferSize': 1024 * 1204 * 200,
})
driver.get(url)

Error Logs:

2024-01-12T16:55:56.503921+00:00 app[web.1]: Traceback (most recent call last):
2024-01-12T16:55:56.503987+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/seleniumbase/core/browser_launcher.py", line 3359, in get_local_driver
2024-01-12T16:55:56.503987+00:00 app[web.1]:     driver = undetected.Chrome(
2024-01-12T16:55:56.503988+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/seleniumbase/undetected/__init__.py", line 217, in __init__
2024-01-12T16:55:56.503988+00:00 app[web.1]:     options.binary_location = (
2024-01-12T16:55:56.503988+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/chromium/options.py", line 52, in binary_location
2024-01-12T16:55:56.503988+00:00 app[web.1]:     raise TypeError(self.BINARY_LOCATION_ERROR)
2024-01-12T16:55:56.503988+00:00 app[web.1]: TypeError: Binary Location Must be a String
2024-01-12T16:55:56.503989+00:00 app[web.1]: 
2024-01-12T16:55:56.503990+00:00 app[web.1]: During handling of the above exception, another exception occurred:
2024-01-12T16:55:56.503990+00:00 app[web.1]: 
2024-01-12T16:55:56.504004+00:00 app[web.1]: Traceback (most recent call last):
2024-01-12T16:55:56.504036+00:00 app[web.1]:   File "/app/homer.py", line 49, in run
2024-01-12T16:55:56.504036+00:00 app[web.1]:     sts_scrapper.run()
2024-01-12T16:55:56.504047+00:00 app[web.1]:   File "/app/scrapers/sts_scraper.py", line 35, in run
2024-01-12T16:55:56.504047+00:00 app[web.1]:     self.create_session()
2024-01-12T16:55:56.504058+00:00 app[web.1]:   File "/app/scrapers/sts_scraper.py", line 28, in create_session
2024-01-12T16:55:56.504059+00:00 app[web.1]:     cookies,  self.headers = chr_cookies_generator(url=self.main_url, headers=self.headers)
2024-01-12T16:55:56.504069+00:00 app[web.1]:   File "/app/scrapers/scraper_utils.py", line 25, in chr_cookies_generator
2024-01-12T16:55:56.504070+00:00 app[web.1]:     driver = Driver(uc=True,
2024-01-12T16:55:56.504081+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/seleniumbase/plugins/driver_manager.py", line 453, in Driver
2024-01-12T16:55:56.504081+00:00 app[web.1]:     driver = browser_launcher.get_driver(
2024-01-12T16:55:56.504093+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/seleniumbase/core/browser_launcher.py", line 1541, in get_driver
2024-01-12T16:55:56.504093+00:00 app[web.1]:     return get_local_driver(
2024-01-12T16:55:56.504105+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/seleniumbase/core/browser_launcher.py", line 3719, in get_local_driver
2024-01-12T16:55:56.504105+00:00 app[web.1]:     driver = webdriver.Chrome(service=service)
2024-01-12T16:55:56.504117+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
2024-01-12T16:55:56.504118+00:00 app[web.1]:     super().__init__(
2024-01-12T16:55:56.504128+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 56, in __init__
2024-01-12T16:55:56.504128+00:00 app[web.1]:     super().__init__(
2024-01-12T16:55:56.504140+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 208, in __init__
2024-01-12T16:55:56.504140+00:00 app[web.1]:     self.start_session(capabilities)
2024-01-12T16:55:56.504153+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 292, in start_session
2024-01-12T16:55:56.504153+00:00 app[web.1]:     response = self.execute(Command.NEW_SESSION, caps)["value"]
2024-01-12T16:55:56.504164+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
2024-01-12T16:55:56.504164+00:00 app[web.1]:     self.error_handler.check_response(response)
2024-01-12T16:55:56.504177+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
2024-01-12T16:55:56.504177+00:00 app[web.1]:     raise exception_class(message, screen, stacktrace)
2024-01-12T16:55:56.504209+00:00 app[web.1]: selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
2024-01-12T16:55:56.504209+00:00 app[web.1]:   (session not created: DevToolsActivePort file doesn't exist)
2024-01-12T16:55:56.504209+00:00 app[web.1]:   (The process started from chrome location /app/.cache/selenium/chrome/linux64/120.0.6099.109/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

[Kamran-ov]

I saw few stackoverflow posts
https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t
https://stackoverflow.com/questions/67580639/selenium-on-heroku-devtoolsactiveport-file-doesnt-exist

In one of them even you commented, options which I can see webbrowser_launcher.py and all are being added to options.

    if IS_LINUX or not is_using_uc(undetectable, browser_name):
        chrome_options.add_argument("--no-sandbox")
    if IS_LINUX:
        chrome_options.add_argument("--disable-dev-shm-usage")
       
    elif not is_using_uc(undetectable, browser_name):
        chrome_options.add_argument("--disable-gpu")

Not sure only about the last one, disable_gpu can be the cause of it or not.
if yes then I should not use undetected, to be able to have it in the options?

[mdmintz]

Use SB() instead of Driver() so that you get the built-in virtual display that you need when running on a Linux Server that has no GUI:

from seleniumbase import SB

with SB(uc=True, xvfb=True) as sb:
    sb.driver.uc_open_with_reconnect("https://pixelscan.net", 20)

(Otherwise you'll need to use https://github.com/mdmintz/sbVirtualDisplay)

If for some reason xvfb=True didn't work for you, use headless2=True instead. From all the Linux environments I've seen, either xvfb=True OR headless2=True does the trick when using the SB() format.

If you're sticking with the Driver() format, then know how to use sbvirtualdisplay correctly:

from sbvirtualdisplay import Display
from seleniumbase import Driver

display = Display(visible=0, size=(1440, 1880))
display.start()

driver = Driver(uc=True, headless=False, incognito=True)
try:
    driver.uc_open_with_reconnect("https://pixelscan.net", 20)
finally:
    driver.quit()

display.stop()

(That overrides the default headless mode on Linux and forces your script to go through the virtual display instead.)