running out of space on grader

[tansaku]

Berkeley SPOC TAs and MOOC students reported the grader was down.

SSHing into the instance I got a filespace warning, and couldn't re-start the grader. This is the filespace stats I saw.

ubuntu@ip-10-182-65-242:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  4.2G  3.2G  57% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.7G  8.0K  3.7G   1% /dev
tmpfs           752M  208K  752M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdb        30G   45M   28G   1% /mnt

I followed the solution here

http://stackoverflow.com/a/9728026/316729

Creating a 160Gig volume based on the existing snapshot - one wrinkle was that I had to make sure the volume was in us-east-1c

Here are the stats afterwards and the grader seems to be working

ubuntu@ip-10-229-30-162:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      158G  4.2G  147G   3% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.7G  8.0K  3.7G   1% /dev
/dev/xvdb        30G   45M   28G   1% /mnt

Ideally we need monitoring on this and some form of log/submission archiving or deleting