Login issue

[sebbourgeois]

Hey @alexlaverty (happy new year !),

Seems they did some change (again) on the login page :

Launching Headless Browser...
Username and password fields not found.
Login failed.
Downloading screenshots... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Traceback (most recent call last):
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 440, in <module>
    process_submissions(submissions)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 64, in process_submissions
    process_submission(submission)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/app.py", line 118, in process_submission
    vid.create(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/video_generation/video.py", line 500, in create
    download_screenshots_of_reddit_posts(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/comments/screenshot.py", line 179, in download_screenshots_of_reddit_posts
    page.locator(f"#t1_{comment.id}").screenshot(path=comment_path)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 17227, in screenshot
    self._sync(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
    return task.result()
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 491, in screenshot
    return await self._with_element(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 103, in _with_element
    handle = await self.element_handle(timeout=timeout)
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_locator.py", line 297, in element_handle
    handle = await self._frame.wait_for_selector(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 322, in wait_for_selector
    await self._channel.send("waitForSelector", locals_to_params(locals()))
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 461, in wrap_api_call
    return await cb()
  File "/home/sb/videos-creation/python-reddit-youtube-bot/env/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 96, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("#t1_khe3o65")
============================================================

Tried with accounts that have access to both old and new UI, same error (#t1_khe3o65 for old UI, #t1_khjepkk for new UI)

[johnnyvng]

I got the same issue

page.type("#loginUsername", auth.praw_username)

File "/home/aibots/.local/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 11470, in type
self._sync(
File "/home/aibots/.local/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 104, in _sync
return task.result()
File "/home/aibots/.local/lib/python3.10/site-packages/playwright/_impl/_page.py", line 910, in type
return await self._main_frame.type(**locals_to_params(locals()))
File "/home/aibots/.local/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 710, in type
await self._channel.send("type", locals_to_params(locals()))
File "/home/aibots/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 61, in send
return await self._connection.wrap_api_call(
File "/home/aibots/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 461, in wrap_api_call
return await cb()
File "/home/aibots/.local/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 96, in inner_send
result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("#loginUsername")

[alexlaverty]

Feels like reddit is purposely trying to break 3rd party apps these days... I'm away on leave at the moment will have a look when I'm back next week.

[alexlaverty]

In the screenshot.py you can change headless to false and it'll show the chrome browser as it scrapes the comments might give a clue as to what's wrong

[alexlaverty]

Doing some investigation tonight, seems running it from Github Actions I'm getting the following error, will keep troubleshooting to see if there's any way around it... :

which leads to this page :

https://www.reddit.com/wiki/api/

We want to allow developers to build great products powered by Reddit and we recognize our developer community is integral to the success of the Reddit platform. We also want to protect our users’ privacy and security.

In order to access the Reddit API directly, you must read our Reddit’s Developer Terms and Data API Terms.

When you are ready, you must register in order to use the Reddit API. Select “I’m a Developer” and “I want to register to use the Reddit API.” Then, you can create credentials here.

Helpful resources:
All API clients must authenticate with OAUTH 2 See our API Specification and FAQ and additional documentation

Sign up for Dev Platform, a new improved way to build bots and apps!

[sebbourgeois]

Hey Alex,

Yeah, I have the same result when I run the command in my terminal :

Launching Headless Browser...
Username and password fields not found. Printing HTML:
<!DOCTYPE html><html><head>
    <title>Blocked</title>
    <style>
      body {
          font: small verdana, arial, helvetica, sans-serif;
          width: 600px;
          margin: 0 auto;
      }

      h1 {
          height: 40px;
          background: transparent url(//www.redditstatic.com/reddit.com.header.png) no-repeat scroll top right;
      }
    </style>
  </head>
  <body>
    <h1>whoa there, pardner!</h1>

<p>Your request has been blocked due to a network policy.</p>

<p>Try logging in or creating an account <a href="https://www.reddit.com/login/">here</a> to get back to browsing.</p>

<p>If you're running a script or application, please register or sign in with your developer credentials <a href="https://www.reddit.com/wiki/api/">here</a>. Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again. if you're supplying an alternate User-Agent string,
try changing back to default as that can sometimes result in a block.</p>

<p>You can read Reddit's Terms of Service <a href="https://www.reddit.com/wiki/api/">here</a>.</p>

<p>if you think that we've incorrectly blocked you or you would like to discuss
easier ways to get the data you want, please file a ticket <a href="https://support.reddithelp.com/hc/en-us/requests/new?ticket_form_id=21879292693140">here</a>.</p>

<p>when contacting us, please include your ip address which is: <strong>x.x.x.x</strong> and reddit account</p>


    </body></html>
Login failed.

They are really making it hard and annoying these days. I'm making a ticket on the link they provided so maybe I'll have more information about this.

EDIT : no answer yet from Reddit, it will probably take a while.
I did some tests though, with your updated code. If I run it headless, I have the error message. If I disable the headless mode in the screenshot.py file, the script is able to log in and starts downloading comments. However, it generates a very short video, around 2mins :

ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 videos/18o0rhl_What_warning_signs_are_you_seeing_that_no_one_is_p/final.mp4
140.959000

The weird thing is, in the assets/work_dir/ directory, if I have a look at the length of all the mp3 files, it makes a length of 14:37 :

(~/.../assets/work_dir/18o0rhl/) $ mp3info -p '%S\n' *.mp3 | awk '{s+=$1} END {printf"%d:%02d:%02d\n",s/3600,s%3600/60,s%3600%60}'
0:14:37

So it seems that there is some problem with the video making as well.
Let me know if I you need me to share some logs/screenshots/info/whatever.

EDIT2 :
tried with another video, same weird issue, got 11mins of audio files, but the video is not even 2mins long :

EDIT3 :
Next mornng, with a bit of coffee, it helps :D
Actually, I just had a look at the logs, and I saw numerous warnings like this ;
Comment larger than video height : assets/work_dir/19d0tf4/comment_kj2v5s2.png

Maybe it comes from the fact I'm not running it in headless mode, but that's why the videos are so short, it is skipping a lot of comments during the creation.

[sebbourgeois]

@alexlaverty I just made a PR (#101) to fix that issue. Enforcing the user agent works like a charm. Also made a change in the requirements.txt to enforce Pillow to v9.5.0, latest version (10+) throws errors.

[alexlaverty]

Nice work with the PR i have merged it in, for the comment larger than video height error the code is :

  if img_clip.h > settings.video_height:
      logging.info("Comment larger than video height : %s", img_path)
      continue

python-reddit-youtube-bot/video_generation/video.py

Lines 542 to 544 in 38de9b6

	if img_clip.h > settings.video_height:
	logging.info("Comment larger than video height : %s", img_path)
	continue

Basically if the comment height is bigger then height of the video skip it, maybe check what your video_height is set to or how big the images are i will run with the latest from master and see if i can reproduce.

Update : have looked into the comment large than video height issue, it's because now when screenshotting the comments some banner is coming up which is causing the screenshot to be very tall which is exceeding the video height, will need to now update the screenshot logic to handle the banner that's coming up...

having to parse html and css is brittle, any time they change it means it will probably break the code.. : (