Fix aws-waf

[Felixoid]

The work is done on top of @Rick7221's commit from #319 (comment)

• Vendor https://github.com/xKiian/awswaf into the repository since it is:
  • not available in PyPI
  • Depends on unmaintained pyscrypt (use hashlib instead for scrypt xKiian/awswaf#4)
  • Does not provide the resolver for mp_verify, AKA network bandwidth challenge
• Implement WAF challenge resolution
• Add optional dependencies

fixes #319

[kugel-]

I tried this and it worked. Many great thanks for that!

[RealCLanger]

I tried it and it worked once. But after that it doesn't any more. Trying to figure out what happens...

[Felixoid]

Should be cookies in ~/.pytr

Probably, worth cleaning them on 403?

[RealCLanger]

Hm, ok, interesting. But shouldn't the cookies be updated, at least the aws-waf-token which is the one that counts. I'll try/check that.

Update: Cleaning the cookies helped.

[Felixoid]

I manually deleted the aws-waf-token cookie from a file, and it worked too.

Probably, we need to filter it when reading from the cookie file

[RealCLanger]

Generally, the change looks very good. Thanks for the contribution. I'll try to review it more thoroughly later today.

[Felixoid]

Thanks!

I added a fix. We clearly don't need to save the WAF cookie. And if there's one in the file, we need to overwrite it with the fresh one.

[RealCLanger]

Thanks!

I added a fix. We clearly don't need to save the WAF cookie. And if there's one in the file, we need to overwrite it with the fresh one.

Maybe the WAF token could be stored, though and used. And only if we get a 403 in authentication we update/regenerate it. I guess that would at least improve performance.

I don't know whether there's (another) way to check whether the waf token is still valid?

[MedAzizKhayati]

They changed the login yet again it seems, I am getting 405 response code when I try to login via API.

[RealCLanger]

They changed the login yet again it seems, I am getting 405 response code when I try to login via API.

I saw it failing again and needed to remove the cookies once more, did you try that? I also didn't add the latest commit yet to my build that filters/updates the WAF cookie.

[Felixoid]

I saw it failing again and needed to remove the cookies once more

Didn't affect me with the latest code, works after 2h of the last launch

Maybe the WAF token could be stored, though and used. And only if we get a 403 in authentication we update/regenerate it. I guess that would at least improve performance.

It's a subsecond. I think automatically obtained token is a safe side:

In [8]: def bench():
   ...:     time_start = time.perf_counter()
   ...:     i = 50
   ...:     for _ in range(i):
   ...:         api.TradeRepublicApi()
   ...:     time_end = time.perf_counter()
   ...:     time_duration = time_end - time_start
   ...:     avg_time = time_duration / i
   ...:     print(f'Took {time_duration:.3f} seconds, average time {avg_time:.3f} seconds')
   ...:

In [9]: bench()
12:08:30 AWS WAF token obtained automatically
12:08:30 AWS WAF token obtained automatically
12:08:30 AWS WAF token obtained automatically
12:08:31 AWS WAF token obtained automatically
12:08:31 AWS WAF token obtained automatically
12:08:31 AWS WAF token obtained automatically
12:08:32 AWS WAF token obtained automatically
12:08:32 AWS WAF token obtained automatically
12:08:32 AWS WAF token obtained automatically
12:08:32 AWS WAF token obtained automatically
12:08:33 AWS WAF token obtained automatically
12:08:33 AWS WAF token obtained automatically
12:08:33 AWS WAF token obtained automatically
12:08:34 AWS WAF token obtained automatically
12:08:34 AWS WAF token obtained automatically
12:08:34 AWS WAF token obtained automatically
12:08:35 AWS WAF token obtained automatically
12:08:35 AWS WAF token obtained automatically
12:08:35 AWS WAF token obtained automatically
12:08:35 AWS WAF token obtained automatically
12:08:36 AWS WAF token obtained automatically
12:08:36 AWS WAF token obtained automatically
12:08:36 AWS WAF token obtained automatically
12:08:37 AWS WAF token obtained automatically
12:08:37 AWS WAF token obtained automatically
12:08:37 AWS WAF token obtained automatically
12:08:37 AWS WAF token obtained automatically
12:08:38 AWS WAF token obtained automatically
12:08:38 AWS WAF token obtained automatically
12:08:38 AWS WAF token obtained automatically
12:08:38 AWS WAF token obtained automatically
12:08:39 AWS WAF token obtained automatically
12:08:39 AWS WAF token obtained automatically
12:08:39 AWS WAF token obtained automatically
12:08:40 AWS WAF token obtained automatically
12:08:40 AWS WAF token obtained automatically
12:08:40 AWS WAF token obtained automatically
12:08:41 AWS WAF token obtained automatically
12:08:41 AWS WAF token obtained automatically
12:08:41 AWS WAF token obtained automatically
12:08:41 AWS WAF token obtained automatically
12:08:42 AWS WAF token obtained automatically
12:08:42 AWS WAF token obtained automatically
12:08:42 AWS WAF token obtained automatically
12:08:43 AWS WAF token obtained automatically
12:08:43 AWS WAF token obtained automatically
12:08:43 AWS WAF token obtained automatically
12:08:43 AWS WAF token obtained automatically
12:08:44 AWS WAF token obtained automatically
12:08:44 AWS WAF token obtained automatically
Took 14.585 seconds, average time 0.292 seconds

[RealCLanger]

I saw it failing again and needed to remove the cookies once more

Didn't affect me with the latest code, works after 2h of the last launch

Yes, I had the error again, then updated to the latest version and it got resolved.

Maybe the WAF token could be stored, though and used. And only if we get a 403 in authentication we update/regenerate it. I guess that would at least improve performance.

It's a subsecond. I think automatically obtained token is a safe side

Probably yes, if we don't have a client-side mechanism to check validity of the token.

[tobias-goertz]

@Felixoid Great work on this PR — tested it and the WAF token generation works perfectly.

One issue I ran into: the WAF token is fetched using curl_cffi (browser TLS fingerprint), but the actual login POST to /api/v1/auth/web/login still goes through plain requests.Session. TR's ALB appears to validate TLS fingerprints on auth endpoints — I consistently get 405 from awselb/2.0 with plain requests, even with a valid WAF token.

Replacing _websession with a curl_cffi.Session(impersonate="chrome") for the auth requests fixes this. Since curl_cffi is already a dependency of this PR, it would just be a matter of using it for _websession too (or at least for the auth-related HTTP calls).

This might explain why some people in #319 report 405 errors even after applying this fix — it depends on whether TR's ALB enforces TLS fingerprinting for their specific requests/region.

Also minor: the WAF host extraction via response.text.split('src="https://')[1].split("/challenge.js")[0] is a bit fragile — a regex like re.search(r'src="(https://[^"]+)/challenge\.js"', response.text) would be more robust against HTML changes.

[Felixoid]

Replacing _websession with a curl_cffi.Session(impersonate="chrome") for the auth requests fixes this. Since curl_cffi is already a dependency of this PR, it would just be a matter of using it for _websession too (or at least for the auth-related HTTP calls).

Currently, it's an optional dependency.

I can think of conditional import in this place, so if a user has curl_cffi installed, it will be used.

Although, for a manual set token it's a slippery slope, since an original browser can be anything.

I don't think an issue is a session. Most probably it's an outdated cookie from the file with previous broken code

[Felixoid]

the WAF host extraction via response.text.split('src="https://')[1].split("/challenge.js")[0] is a bit fragile — a regex like re.search(r'src="(https://[^"]+)/challenge.js"', response.text) would be more robust against HTML changes.

After AwsWaf started parsing challenge.js, the logic should be twisted around, as you suggested. Thanks, it's much more reliable indeed!

[Felixoid]

This might explain why some people in #319 report 405 errors even after applying this fix — it depends on whether TR's ALB enforces TLS fingerprinting for their specific requests/region.

There was a state, when aws-waf-token cookie was written into a cookie jar on script exit. Then, on another run a new token was received, but after that, the cookies were restored from the file, overwriting aws-waf-token with an outdated value

It was fixed in c81d47f, please test it

[tobias-goertz]

@Felixoid You were right — the 405 I was seeing was caused by stale WAF cookies, not TLS fingerprinting. After updating to your latest commits (cookie cleanup fix), I removed my curl_cffi BrowserSession workaround entirely and everything works with plain requests.

Tested the full flow end-to-end: WAF token → login → 2FA → websocket data sync. All good on latest a9f6116.

Thanks for the quick iteration on the cookie handling!

[RealCLanger]

A formal remark: Could you put the MIT license first in the vendored files and the comment from where it was vendored etc. afterwards?

Also, I suggest to remove the additional LICENSE file in the aswwaf subdirectory as it is the same license (MIT) as for the other files and additionally, it is included as file header.

[Felixoid]

Let me rebase the staff a little bit, so it will be cleaner git log.

The work looks finished.

[RealCLanger]

Let me rebase the staff a little bit, so it will be cleaner git log.

The work looks finished.

We'll squash anyway 😄

[RealCLanger]

OK, this looks close to merging now. Two small remarks in the code.

[RealCLanger]

Thanks @Felixoid for this great piece of work.

[fft1010]

Hi. I can confirm, that it now works for non programmers as me. Thanks a lot for this. But if I do
pytr -V
it said, I am on the latest version, doing a

pip install --upgrade --force-reinstall git+https://github.com/pytr-org/pytr.git

got the newest version, and it worked. Again. Thanks a lot, this is for others, that might not find the "trick" on their own.

Juergen

[RealCLanger]

Hi. I can confirm, that it now works for non programmers as me. Thanks a lot for this. But if I do pytr -V it said, I am on the latest version, doing a

pip install --upgrade --force-reinstall git+https://github.com/pytr-org/pytr.git

got the newest version, and it worked. Again. Thanks a lot, this is for others, that might not find the "trick" on their own.

Version 0.4.7 with the fix is now officially released and available via standard pip upgrade.