Detect private accounts

[Sidnioulz]

@huang-julien told me agentscan can fail to detect bot accounts if their contribution list is private. However, he also told me there is a way to detect, in fetch response payloads, if contributions are private. This could be used as a signal to put accounts in a mixed bucket, or inconclusive bucket.

[github-actions]

✅ Organic activity

No automation signals detected in the analyzed events.

View full analysis →

_{This is an automated analysis by AgentScan}

[huang-julien]

yeah @MatteoGabriele @graphieros , I found this while taking a look at https://github.com/peakoss/anti-slop

there's a profile_view_type info coming from GH

[MatteoGabriele]

You're saying to add some kinda of flag if the user has that profile type?

[graphieros]

@MatteoGabriele

We had a chat about this yesterday with @huang-julien's after his talk, around AgentScan's categories.

I think the cool thing with AgentScan is its 3-category simplicity.

Instead of increasing the number of categories, it might be wise to just have some inner taxonomy that would allow, for example, private profiles to be classified under 'mixed'. However, 'mixed' does not seem to fit quite exactly, it is an in-between limbo which is basically saying to the user "beware", "investigate", etc, whereas both organic and automated offer more assertive conclusions. A private profile is not really "mixed", more like impossible to tell, this grey area we might find a way to shed light on later.

Just starting a conversation here, with a recommendation to keep the 3 cat system which makes AgentScan effective in my opinion, and perhaps a renaming of "mixed" could make it easier to extend with more of these in-between flags.

Also, Julien talked about maybe surfacing AgentScan notifications only to maintainers (through email, or whatever GH makes it possible to hide from other users), in order to avoid clanks training on causes of PR closing after an AgentScan flagging. Just throwing that here so we remember.

[MatteoGabriele]

I like the idea, but I don't think private accounts fit into any of the existing classifications. We might need a 4th one that acts as a catch-all for anything we can't classify.
Unresolved could work well for that. I'd prefer to keep it at 3 like u said, but a 4th bucket for edge cases like this could work.
what you think? 👀

[mootari]

agentscan can fail to detect bot accounts if their contribution list is private

In case it's relevant: GitHub returns public user events even for profiles that have been set to private. I don't know whether that's a feature or an oversight.

[MatteoGabriele]

I think it's because there's a delay between 30 minutes to 48 hours between updates, sometimes. So if u put yourself to private but were public might take time.
Nonetheless, unreliable I think.

[mootari]

I think it's because there's a delay between 30 minutes to 48 hours between updates, sometimes

No, I've been pulling profiles from Google by searching for "activity is private", and I currently have a list of 15 usernames for which I got events (1/4 or 1/3 of the accounts I checked, which I believe tracks considering how many users are inactive).

[huang-julien]

that's probably a bug, we couldn't retrieve private profile's data at the beginning of agent-scan

[MatteoGabriele]

any thoughts about this one?
not sure where to go exactly. we shouldn't rely on possible bugs, just use what we have which is nothing since it's private.

what story should that tell? I don't think we can put it in any of our existing classifications since they are pretty well defined around the automation classification: no automation (organic), some automation (mixed), lots of automation (automation).
I'd say a 4th will be necessary but also, is it worth it?