LinkedIn Scraper

[HenryAllen04]

LinkedIn scraper was set to disabled by you Eddy, it was falling back to fake data. This needs to be on and working ensuring we collect basic info.

We also need to understand rate limits associated to scraping etc.. if you need help pull @mousberg into this.

If we want to be able to disable the scraper this is best done through a variable change in settings.

[EddyDavies]

It was not disabled intentionally, clearly missed claude code deciding to do this, sorry.

[EddyDavies]

The current BrightData has a limitation that it can only access data that is visible without being logged in, which you can see by looking at a page in incognito mode.

https://www.linkedin.com/in/georgia-gibson-a89b57163/

See my conversation with BrightData Support

[EddyDavies]

I think it is just some profiles that hit this issue, so we just need to add a flag that catches the experience data missing and not compare when that happens, making sure no fake data is created.