PROJECT: Basic statistics in Opencast

[oas777]

This goes back to a discussion at the DACH meeting where we agreed to reactivate the existing statistics tab in the admin UI. Not with Matomo, but with a similar intention of presenting basic (!) statistics to the admin, powered by Opencast (and/or the player), and encompassing different distribution channels.

So the idea is to present the 2-3 most common statistical numbers

• Number of plays/views/clicks (must-have)
• Number of users who have viewed (“unique views”, likely a must)
• Minutes watched
• Completion rate

The tab should differentiate, ideally also merge data from different channels (LMS, Tobira), more complexity via the player to also include "exotic" distribution channels.

More complexity:

• per video/per series
• up until now or with the option to set a time period

I will create the issue once we had the discussion at the roadmap meeting.

[mtneug]

The tab should differentiate, ideally also merge data from different channels (LMS, Tobira), more complexity via the player to also include "exotic" distribution channels.

Would that include the Opencast Annotation Tool?

IMO providing a common player embedded as iFrame/script in all UIs would simplify this goal (see JWT discussion). Otherwise all UIs need to implement the sending of data.

[oas777]

The tab should differentiate, ideally also merge data from different channels (LMS, Tobira), more complexity via the player to also include "exotic" distribution channels.

Would that include the Opencast Annotation Tool?

How did you guess when I said "exotic"? Answer: It could, but I presume this would add complexity. Which would increase resources needed. Which would make me ask where the budget is coming from for that.

[dagraf]

In meeting: Proposition of @LukasKalbertodt

• Goal:
  • Simple solution built in Opencast, with only basic statistics, features and numbers
  • should work out of the Opencast box
  • practice data minimalism (only collect what we need)
  • should not require a consent banner (to improve the data we gather)
• basic idea:
  • all data should be stored in Opencast
  • external apps report to Opencast
  • two layers information gets stored: (i) raw event data (event = thing happening in time, e.g. 'this video was started'); (ii) aggregated data which will be presented to the users
  • no user identifier information should be stored, so we do not need a consent banner > solution: storing a hash, like other tools do
  • possible problem: there are 'network address translation'
  • should be integrated into the Paella player

Notes with more details will be made available shortly here.

[LukasKalbertodt]

Meeting notes

I was tasked to develop a concept for this, so this is what I did today. In the meeting just now I presented a summary of that concept, most of which David already mentioned above. For completeness, here are my notes for todays meeting:

• What we want: simple, out of the box statistics.
  • Simple: View count, watchtime, both potentially over time. Maybe retention graph. No arbitrary analytics. Not a replacement for Matomo.
  • Out of the box: no additional setup required, people expect view count to just be there.
• Privacy: stick to our Open Source principles
  • "Data minimalism", not "collect first think later"
  • No consent banner: user-privacy, UX, brand credibility, and most importantly: more reliable stats
• Basic architecture:
  • All data stored in OC
  • Apps report to OC (e.g. octoka, Tobira, LMS)
• Store two things:
  • Raw events (not videos): timestamp, session id, type (fetch, play, seek, ...), arbitrary metadata
  • Aggregated: hour granularity, retention graph
• Unique user tracking:
  • Without cookies or storing UII (user identifiable information)
  • Session ID from hash(salt + IP + UA)
    • NAT problem?
• API will be added to OC

Details

Below, I describe more details that I didn't get to explaining in the meeting.

What exactly to store

There are two levels of storage: raw and aggregated. The latter is derived from the former. Since the raw data can potentially grow in size quickly, it is expected that some installations will delete older entries. Keeping the raw data around has the advantage of being able to re-derive aggregated data in case the aggregation algorithm is improved or produced more detailed information. We store no user-identifiable information at any point, neither in raw nor aggregated data.

Aggregated

My suggestion for what to end up with:

• Views in hour granularity
• Watchtime in hour granularity
• Timeline data:
  • Divide the video into a number of buckets (max 256, that's 20s per bucket for 90min video, probably reduce buckets for shorter videos)
  • Per bucket store: number of seeks, pauses and plays

Of course, one can start simpler, e.g. no timeline data for now, or the other two metrics without time component, but just as a single number each. As long as raw data is not deleted, more sophisticated aggregated data can be re-derived from raw data.

If storing views & watchtime across time, one likely wants to have a cached total number for them. Also, one would not store hours or days where there weren't any views.

In terms of storage space:

• Views & Watchtime in hour granularity: 24 * 2 * 4 bytes = 192 bytes per day
• Timeline data is just 256 * 3 * 4 bytes = 3KB per event

For a large OC installation (100k videos), assuming most videos are only watched on 100 days, that's 22KB per event, or ~2GB total. That's acceptable I think.

Raw data

This is just a list of "events" (not OC events, the normal use of the term). We store the following data:

• timestamp
• item_kind: event (OC), series, playlist (we might want to have some stats about non-video things too?)
• item_id (e.g. video UUID)
• session: unique identifier for a unique user, kind of. See below.
• event_kind: describes what happened, see below for a list
• event_data: arbitrary metadata about the event, basically JSON?
• Maybe later: browser, OS, desktop/mobile, country/region/city

Here are some important events kinds with their metadata:

• play: user starts playing a video. No metadata.
• seek: user seeks to a position in the video. Metadata: target duration of the seek.
• fetch-file: whenever a video file is actually requested from the file server. Metadata:
  • type is presenter-video, captions, ...
  • range is an optional byte range that was requested
• visit: user visits a page of that item (e.g. the video page in Tobira or the /play URL). Metadata: referrer.
• watched: when some part of the video was fully watched. Metadata: what part was watched?
• pause & resume: when a user pauses or resumes. Metadata: current play position when it was paused/resumed.
• Maybe later: video quality, speed, subtitle usage, fullscreen, video layout

Counting unique users

A typical interaction with Opencast will trigger a number of events to happen, e.g. visit, then play, then fetch-file and so on. It is very beneficial to know which events stem from the same person to correlate them to one another, and to not over-count, for example. Classically, this is done with a tracking cookie, but this is no bueno for many reasons.

A number of privacy focused analytics products have come up with a different solution, that involves storing a hash of IP address and user agent (UA). More precisely they store sha256(salt + IP + UA) with a random salt that rotates daily. The salt is necessary as otherwise, brute-forcing is theoretically viable: 4 billion IP addresses (likely somewhat reducible, by only considering residential IPs, ...) times a few hundreds of common user agent strings. When rotating the salt, the old one is completely removed to make brute forcing non-viable. This also means that after the rotation, every event is regarded as stemming from a new unique user. The rotation should ideally happen in the middle of the night to make it unlikely that the unability to correlate events matters.

It is important to ensure old salts are never kept, but this also means that it should ideally not be stored in the DB as otherwise regular DB backups might retain it. At the same time, there should only be one unique salt per day, so if multiple OC nodes should be able to deal with this, they need to agree on a single salt. This potentially means the service can only run on one node? (Open question!)

The raw event data should be kept for at least until the salt rotates, to be able to correlate events to one another.

Protection against abuse, bots, crawlers, ...

Imagine for a second: web development if there were no bad actors out there. But alas, in the real world we have to deal with all kinds of stupid stuff:

• Users that want to manipulate statistics, e.g. driving up view count. By refreshing the page a lot or even writing a script.
• Normal, well behaved web crawlers
• Malicious crawlers
• Click fraud bots & spam bots
• DDoS attacks

First the bad news: we cannot defend reliably against all of that. No one can. But the good news is that we can protect fairly easily against the most common of these and there is little incentive for random malicious crawlers and bots to influence Opencast statistics. Possible counter-measures include:

• Well behaved web crawlers can be ignored via user agent filter
• Ignoring requests from certain IP ranges (data centers or known spam source). IP ranges for VPNs (usually located in data centers) should be not be excluded.
• Detecing unusual patterns
  • rejecting lots of repeated events from the same IP
  • Ignore play events without fetch-file event following

I think implementing some basic UA filter and simple measure to prevent a single student from screwing with statistics should be absolutely fine. These things need to happen immediately when the statistics data is ingested, as later on we don't have the UA/IP data anymore. Some protections can also be implemented outside of Opencast, e.g. there are some "AI crawler blockers" out there that can be installed in front of any website.

API

Opencast needs a new API for this.

• Trusted push: this is for external apps (octoka, Tobira, LMS, ...) to push some raw events. Those external apps would buffer events in their memory and batch push them to Opencast every minute or so. Apps need to authenticate to do that. Normal users browser would never send a request to this.
• Client push (maybe?): close to the endpoint that normal analytics systems use, used directly by a users browser. The IP and UA would not be sent in the payload, but derived from the request, and there are likely some additional validity checks, e.g. not allowing fetch-file events.
• Reading statistic information

A note on external tools and trusted push: most of the actual statistics logic needs to happen in Opencast. For one, because there can only be one single salt for each day, and communicating that to external tools is hard. But we also don't want to reimplement any logic in other tools. Finally, the trusted push has to contain the actual UA and IP data, but that shouldn't be a problem when communicated over a safe channel.

Open questions

• How would Tobira get up to date statistics?
  • Not via event updates in the normal harvest API: that would mean repeating the same event data everytime it gets a view.
  • Maybe an extra harvesting API?
• Using the normal DB or a time series DB?
  • For especially the raw event data, a time series DB sounds like a great fit. They are usually able to compress the data quite a bit, while also having better query times for common queries. On the other hand, we likely don't have thaaaat much data and setting up a time series DB like influx-DB complicates the setup considerably. The idea was to build a out-of-the-box experience. Or allow both as storage backend? But start only with normal DB

Privacy focused analytics solutions

These are just some random notes and links.

Details

Plausible

• Seems to be pretty good and popular
• Hash of salt + IP + UA + domain
• Collects: URL, referrer, browser, OS, desktop/mobile, country/region/city
• Bot detection: UA list, known referrer spam domains (?), data center IP ranges (minus VPN ranges), detecting unusual traffic patterns
• https://plausible.io/data-policy#how-we-count-unique-users-without-cookies

Usermaven

• Advertises as GDPR compliant, cookieless tracking

• But seems to flying close to the sun, optimizing for understanding user behavior

• IP + (part of?) the UA are hashed for unique visitor tracking

• Store approx location (country, city); for UK/EU it masks last octet of IP address (i thought those were only hashed?)

• https://usermaven.com/blog/cookieless-tracking

• https://usermaven.com/blog/privacy-first-analytics-stack

Fathom

• user signature hash: salt + IP + UA + Hostname

• page request signature hash: user signature hash + pathname

• https://usefathom.com/legal/compliance

• https://usefathom.com/data

Ackee

• Also uses salt + IP + UA as identifier, salt rotated daily, salt never in DB
• https://docs.ackee.electerious.com/#/docs/Anonymization

Matomo

• Requires extra configuration to get away without cookie banner
• Aims to be more like a full GA replacement, whereas the solutions above are more minimal

Some law stuff:

• https://matomo.org/faq/general/eprivacy-directive-national-implementations-and-website-analytics/
• https://matomo.org/faq/general/do-i-need-consent-to-use-web-analytics-on-my-website/

Other links

• https://www.vioffice.de/blog/privacy-respecting-analytics

TODO

• Figure out whether university NAT is a significant problem for the IP+UA based session system
• Work out way more details about this system (this is a first sketch!)
• Find funding
• Implement it everywhere

[mtneug]

• Timeline data:
  • Divide the video into a number of buckets (max 256, that's 20s per bucket for 90min video, probably reduce buckets for shorter videos)
  • Per bucket store: number of seeks, pauses and plays

If the number of buckets is not fixed anyway, why not directly divide the video in, say, blocks of 20s? That would automatically adjust the bucket number based on the video length and give a uniform resolution, but of course the upper limit would be harder to calculate. Still, most lecture recordings probably would fall in the 90min or so category.

It is important to ensure old salts are never kept, but this also means that it should ideally not be stored in the DB as otherwise regular DB backups might retain it. At the same time, there should only be one unique salt per day, so if multiple OC nodes should be able to deal with this, they need to agree on a single salt. This potentially means the service can only run on one node? (Open question!)

Running on a single node is a big no-go. Are you suggesting to keep the salt in-memory? What about restarts?

Regarding GDPR: The goal is to basically fall under Article 6(1)(f), i.e. you have a valid interest and don't have to ask for permission (which would be Article 6(1)(a); I don't think reasons (b)-(e) would apply generally). Strictly speaking, the proposal is still processing personal data, e.g. by putting the IP in the trusted push message. But my guess is that this is similar to the "retaining web logs for X days" case, although there are some differences (X is potentially way longer, use of pseudonym, reason for the processing is different). I want to stress that the following two things should be done

• Check the design with a lawyer / data protection officer.
• Provide a description of the data processing. This should be useful for adopters and Opencast hosters that need to document what, how, why and how long personal data is processed.

Regarding the data collection process: I think it makes sense to collect data with the nodes that already distribute static files, i.e. nodes running octoka in the future. As a pointer: it may make sense to look at how Common Media Client Data (CMCD) is doing this. Just to be clear: I'm not suggesting implementing CMCD; this is very much not basic and has a different focus. However, CMCD (especially the upcoming second edition) has some ideas of how data is collected from clients, which may be helpful.

Random notes:

• What about statistics for live stream events?
• Next to ease of installation, my way bigger concern for the choice of data store is performance and scalability. Storing this data must not (significantly) impact normal operations of Opencast.

[LukasKalbertodt]

If the number of buckets is not fixed anyway, why not directly divide the video in, say, blocks of 20s? That would automatically adjust the bucket number based on the video length and give a uniform resolution, but of course the upper limit would be harder to calculate. Still, most lecture recordings probably would fall in the 90min or so category.

I don't care too much, but I think a fixed maximum is helpful to avoid ugly edge cases. I also think 20s buckets generally isn't amazing, especially for 3min videos (which marketing departments of universities often produce) it's too low of a resolution. One can also come up with a function to cleanly scale between max buckets and min bucket length, but all in all: this is a detail question that we can figure out when the time comes.

Running on a single node is a big no-go. Are you suggesting to keep the salt in-memory? What about restarts?

Yep, I assumed that this would be a no go :D I was kind of suggesting keeping it in memory. Restarts wipe it, but that only means one cannot cross correlate events across the restart point. That's not a disaster, especially given that restarts should be rare. But yep so we need to figure out some way to communicate a salt to all relevant nodes that does not run the risk of being stored indefinitely.

Regarding GDPR: The goal is to basically fall under Article 6(1)(f), i.e. you have a valid interest and don't have to ask for permission (which would be Article 6(1)(a); I don't think reasons (b)-(e) would apply generally). Strictly speaking, the proposal is still processing personal data, e.g. by putting the IP in the trusted push message. But my guess is that this is similar to the "retaining web logs for X days" case, although there are some differences (X is potentially way longer, use of pseudonym, reason for the processing is different). I want to stress that the following two things should be done

* Check the design with a lawyer / data protection officer.

* Provide a description of the data processing. This should be useful for adopters and Opencast hosters that need to document what, how, why and how long personal data is processed.

We should absolutely have a "data policy" document describing exactly what we do. And I'm also in favor of asking a lawyer about all of this, but I have two remarks: first, i just want to remind everyone that this is not uncharted territory, we basically just doing what four other analytics solutions already do. Second, for a lawyer being asked about this, "no" is always an easy and safe answer, so we should make sure we find someone who is interested in making it work if possible, not just go the comfortable route.

The only way how my proposal differs from the other analytic solutions I posted is that octoka would send the non-hashed IP and UA to Opencast. But I'd argue it can just be seen as one system. As long as the channel is secure, I don't really see a practical difference to the other solutions.

What about statistics for live stream events?

Mhh I guess for those, only certain events will be valid? E.g. no seek. And depending on the streaming solution, octoka will also not report fetch-file events.

Next to ease of installation, my way bigger concern for the choice of data store is performance and scalability. Storing this data must not (significantly) impact normal operations of Opencast.

Of course, I agree. But I'm fairly sure even using the main DB will not be a concern for most OC users. Opt-in time series DB sounds like a good solution for big institutions.

[snoesberger]

What about statistics for live stream events?

Mhh I guess for those, only certain events will be valid? E.g. no seek. And depending on the streaming solution, octoka will also not report fetch-file events.

Keep in mind that you can buffer live streams. At our institution, live streams are buffered for 30min. This means users can pause a live stream and seek within this 30min. However, I would still consider this to be watching the live stream.
HLS live streaming means generally serving a playlist and video files/chunks. So I assume octoka could also be used with this sort of live streams and would report fetch-file events?