Missing submissions in web interface & cost of submissions

[ssardina]

Which part is affected?

Web app (classroom50.org)

What happened?

1. Student has committed & pushed to main many times, but teacher interface shows no submission,
2. Cost in large classes rises an alarm ⏰ 💲

but in teacher web:

whereas in student view:

What did you expect to happen?

1. Every commit in main should be seen as a submission, as far as I understood from DOC
2. GitHub Teams has already used 20% and I have done nothing yet

I am a bit concerned about the $ of the actions. Only 50% of my students have accepted the Project 0 which is just a warm up. I do not have here any autograding, but I believe the autograder will run anyways every push to main. With 500 students pushing on main for more involved projects, I am very concerned I will run out of quota fairly quickly:

Question: wouldn't it be safer to at least have the option that the student must tag their submission with a particular name. I use submission. The teacher can specify in the web interface if the assignment will have submissions every time a push to main is done, OR every time a tag is added?

Steps to reproduce

1. Student commits and push to main.
2. Web interface does not show any submissions

Version and environment

OS: LInux Mint 22.3
Web teacher: preview version
Web student: production version
GH plan: Teams (Academic)

Relevant logs or output

Before submitting

• I searched existing issues and this is not a duplicate.
• I removed any tokens, secrets, or private student data from the logs above.

[ssardina]

Maybe I should split the two issues: submissions showing up and impact of cost of GH actions 🤦🏽

[rongxin-liu]

This is by design, not a bug. You must run Collect Scores for the submission to appear.

[rongxin-liu]

More context, a few of these touch the core architecture, so let me answer each directly.

1. Live submissions vs. live scores. We can show submissions live in the teacher view, listing a repo's submit/* releases is a plain api.github.com call that works fine from the browser, so a pushed submission shows up without waiting for a collection run. Scores, however, are the part that can't be fully live in the browser. The score lives in a result.json release asset, and GitHub's asset-download endpoint 302-redirects to S3 (objects.githubusercontent.com), which sends no CORS headers, so a browser fetch is blocked, and the browser_download_url path additionally 404s for private repos with fine-grained tokens. That's inherent to GitHub's asset hosting, not something we can configure away. So rendering grades has to go through a server-side step (GitHub Actions), which is what score collection does today. In short: submissions can be live; grades stay collection-backed.

2. The autograder is also what opens the Feedback PR. In the current design, the reusable autograde workflow both grades and opens/updates the one long-lived Feedback PR per repo (it's an opt-in step with pull-requests: write). So disabling the autograder outright also means no automated feedback PRs, the two are coupled by design.

3. Student-tagged submissions. Technically supported already, the runner treats a pushed submit/* tag as a submission trigger, so a "tag to submit" mode is feasible. But two caveats: it doesn't actually bound cost (a student can keep pushing tagged submissions and still burn minutes, though that's arguably a course-policy matter, not a technical one), and more importantly, it pushes Git tagging onto students. A core goal of Classroom 50 is that students shouldn't need to learn GitHub mechanics, so asking them to create tags is a bit counter-intuitive.

4. A better lever for cost: defer grading. The more promising option is to let a teacher disable Actions by default and run grading on demand, e.g. once at the due date, kicking off the autograder for every repo in one batch and opening the feedback PRs then. This reuses machinery we already have (the collect/regrade workflows are teacher-dispatched batch jobs). The tradeoff is that students lose immediate per-push feedback, which is a real cost for iterative assignments, so this would be an opt-in per-assignment mode, not the default.

[ssardina]

Hi @rongxin-liu . Thanks for taking the time to go over all this in detail. 🥇

I have some follows-up, if I may:

1. All good, yes. I do not have autograding now and I was just referring to the interface saying "no submissions" while there were some pushed already. But it is now showing up as submitted, although it says 1 submission while there has been 3 tags/release:

Am I still missing something about submissions? I thought that every push to main was seen as a submission? All those repo have more than one release submit/*

2. Regarding Feedback PR and autograding, shouldn't they be de-coupled? What does it mean to "update the Feedback PR"? Being a PR, it will just list the related commits, what other updates are needed on it once it is created? I do not want to lose the PR facility (I asked for it, ha!), it is great. But one thing is to create a PR and another is to run an expensive automarking system every push to main for 500 students. Right?
3. Right, the **student can do an explicit tag submit/****but the system is already doing that on every push, right?

What I was proposing was the OPTION for teachers to select between: "every push generates a submit/date tag" or "act only on X tag".

Students will not be tagging all the time, they will be pushing all the time though, even on a single typo fix. But tagging will be done once, twice to fix, maybe three, at most 4 times in a homework. I have ran this for 5 years and never had any repo re-tagging more than 3 times.

It is true that in many courses you do not want students to need to know the GH mechanics, but for many, many, courses tagging is not much more than committing and pushing, and one can give clear instructions, It can even be done with GH desktop. And this is a neat way to avoid expensive use of the autograder on every push, it is a minimal price to pay, almost zero. I have been doing this for years and thousands of students that are not git experts, but they figure it out in the first assignment (and I have clear instructions)

4. Yes disabling Actions all together is an option, and running grading at due date and PR creation. That is in fact what I have done in one of my courses for years, because if I let them run on every push I ran out of quota in just 1 week. My question where you would disable Actions? At the individual student repo I guess right? You cannot do it at org level as classroom50 depends on workflows running all the time.

Unless I am missing something, the best solution is to offer two config options:

1. Every push on main is treated as a submission (as it is now) OR only act on a specific submission tag X
2. Allow teacher to enable/disable students' actions.

If 1 is not possible, at least 2. Otherwise I suspect lots of teachers will ran out of quota very very quickly.

Maybe we can do 2 already by disabling actions in the whole org except classroom50 repo? Still much be better if it is handled in the web interface

Sorry for the long post, but I think this issue is quite important @rongxin-liu

[rongxin-liu]

@ssardina I agree that this is an important issue. We considered GitHub Actions usage when we designed and built the current autograding system. We recognized that, given the constraints of the GitHub Team plan, our approach probably would not scale efficiently to a large class, so deferred grading might be best for now in your case.

Regarding Feedback PR and autograding, shouldn't they be de-coupled?

Opening the feedback pull request is intentionally part of the autograder workflow. It is simpler and usually more cost-effective, because GitHub rounds each billable job up to the next full minute. Running the PR step separately could turn a few seconds of work into an additional billed minute. Keeping the steps together also makes recovery easier: if GitHub temporarily fails to open the pull request, rerunning the autograder retries the same step without requiring a separate workflow or recovery process. That said, we could add functionality to the web app that allows course owners to manually trigger feedback pull requests for student assignment repositories. Or we open the PR when student accepts the assignment.

Right, the **student can do an explicit tag submit/****but the system is already doing that on every push, right?

The autograder triggers on every push to the default main branch (as well as on submit/* tags), and on a branch push it creates the submit/- tag and release for you, so a student doesn't need to tag manually.

Basically, today every "real" push to main is treated as a submission. The UI used to render the results from scores.json statically, so if collect-scores did not run for a day or two, you would expect to see stale data. We have made some changes to the submissions page so that it now shows at least some live submission data, but scores (parsed from result.json in the student repo) still come from scores.json.

We will need more time to determine and rethink what works best for autograding, but for now we have added a GitHub Actions section on the org settings page, so you can easily disable Actions at the org level for all student repos except the classroom50 config repo. We also added a spending-cap alert to remind you to set a GitHub Actions spending cap (by default, GitHub should have set a $0 cap when the org was created).