What happened?
Found my server is crashed by out of disk space, and mongodb data is huge.
root@Cosmos:~# du -ahx / | sort -rh | head -n 50
...
13G /var/lib/docker/volumes/cosmos-mongo-data-0OR/_data
13G /var/lib/docker/volumes/cosmos-mongo-data-0OR
13G /var/lib/docker/volumes
8.5G /var/lib/docker/volumes/cosmos-mongo-data-0OR/_data/index-2--2112421272775686701.wt
...
In mongosh
> use COSMOS
< switched to db COSMOS
> db.COSMOS_events.totalSize();
< Long('12373024768')
COSMOS_events collection use 12,373,024,768 bytes (~= 12GB)
What should have happened?
Event log should delete by cronjob.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/CRON.go#L217
I found exist events log is older than one month ago.
date: 2025-09-01T17:58:45.536+00:00
Log in events collection shows that deleted count is always 0,
"cosmos.database.cleanup {"collection":"events","deleted":0}"
but notifications is work.
"cosmos.database.cleanup {"collection":"notifications","deleted":3}"
How to reproduce the bug?
Start a server and add an old log then run the cronjob i guess?
Relevant log output
Other details
The problem is cronjob delete documents where the field "Date" less then one month ago.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/cleanup.go#L22
But events use "date" to record time, and mongodb field is case-sensitive.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/events.go#L24
In my mongodb looks like:
{
"_id": {
"$oid": "68b5df0ff75fea67e3ae5d8b"
},
"data": {
"bytes": {
"$numberLong": "1523"
},
"hostname": "pve.mysite.com",
"url": "/api2/json/cluster/resources",
"route": "ProxmoxVE",
"status": 200,
"method": "GET",
"clientID": "192.168.1.1",
"time": 0.005945966
},
"object": "route@ProxmoxVE",
"_search": "cosmos.proxy.response.ProxmoxVE.success {\"bytes\":1523,\"clientID\":\"192.168.1.1\",\"hostname\":\"pve.mysite.com\",\"method\":\"GET\",\"route\":\"ProxmoxVE\",\"status\":200,\"time\":0.005945966,\"url\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api2/json/cluster/resources\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"}}",
"eventId": "cosmos.proxy.response.ProxmoxVE.success",
"label": "Proxy Response ProxmoxVE success",
"application": "Cosmos",
"level": "info",
"date": {
"$date": "2025-09-01T17:58:45.536Z"
}
}
Notification use "Date" so cronjob works.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/notifications.go#L205
{
"_id": {
"$oid": "69a66167aaab42ac54518c05"
},
"Read": true,
"Recipient": "mysite",
"Vars": "",
"Icon": "",
"Actions": null,
"Title": "header.notification.title.serverError",
"Message": "Your server has no internet connection! : Get \"https://www.google.com\": dial tcp: lookup www.google.com on 192.168.1.1:53: read udp 192.168.1.8:33636->192.168.1.1:53: i/o timeout",
"Link": "",
"Date": {
"$date": "2026-03-03T04:19:11.173Z"
},
"Level": "error"
}
Addition note:
Do not just makes it delete events log by correct letter case, maybe batch delete with limited size?
My events collection have 23M documents since July 2025.
I try delete a lot by one statement and my mongodb stuck until i reboot the container.
> const sixMonthsAgo = new Date();
> sixMonthsAgo.setMonth(sixMonthsAgo.getMonth() - 6);
< 1759288902393
> db.COSMOS_events.find({ "date": { "$lt": sixMonthsAgo } }).count();
< 2359651
> db.COSMOS_events.deleteMany({ "date": { "$lt": sixMonthsAgo } })
# stuck here
# ctrl + c
> Error: Async script execution was interrupted
# reboot after few minutes and service alive
System details
Cosmos version: 0.22.1
Runs on Debian GNU/Linux 12 (bookworm) with LXC container(from Proxmox Virtual Environment 9.0.5).
I just assign 30GB to the container.
Host machine info:
CPU(s) 8 x Intel(R) Core(TM) i3-N305 (1 Socket)
Kernel Version Linux 6.14.8-2-pve (2025-07-22T10:04Z)
What happened?
Found my server is crashed by out of disk space, and mongodb data is huge.
In
mongoshCOSMOS_events collection use 12,373,024,768 bytes (~= 12GB)
What should have happened?
Event log should delete by cronjob.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/CRON.go#L217
I found exist events log is older than one month ago.
Log in events collection shows that deleted count is always 0,
but notifications is work.
How to reproduce the bug?
Start a server and add an old log then run the cronjob i guess?
Relevant log output
Other details
The problem is cronjob delete documents where the field "Date" less then one month ago.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/cleanup.go#L22
But events use "date" to record time, and mongodb field is case-sensitive.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/events.go#L24
In my mongodb looks like:
{ "_id": { "$oid": "68b5df0ff75fea67e3ae5d8b" }, "data": { "bytes": { "$numberLong": "1523" }, "hostname": "pve.mysite.com", "url": "/api2/json/cluster/resources", "route": "ProxmoxVE", "status": 200, "method": "GET", "clientID": "192.168.1.1", "time": 0.005945966 }, "object": "route@ProxmoxVE", "_search": "cosmos.proxy.response.ProxmoxVE.success {\"bytes\":1523,\"clientID\":\"192.168.1.1\",\"hostname\":\"pve.mysite.com\",\"method\":\"GET\",\"route\":\"ProxmoxVE\",\"status\":200,\"time\":0.005945966,\"url\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api2/json/cluster/resources\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"}}", "eventId": "cosmos.proxy.response.ProxmoxVE.success", "label": "Proxy Response ProxmoxVE success", "application": "Cosmos", "level": "info", "date": { "$date": "2025-09-01T17:58:45.536Z" } }Notification use "Date" so cronjob works.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/notifications.go#L205
{ "_id": { "$oid": "69a66167aaab42ac54518c05" }, "Read": true, "Recipient": "mysite", "Vars": "", "Icon": "", "Actions": null, "Title": "header.notification.title.serverError", "Message": "Your server has no internet connection! : Get \"https://www.google.com\": dial tcp: lookup www.google.com on 192.168.1.1:53: read udp 192.168.1.8:33636->192.168.1.1:53: i/o timeout", "Link": "", "Date": { "$date": "2026-03-03T04:19:11.173Z" }, "Level": "error" }Addition note:
Do not just makes it delete events log by correct letter case, maybe batch delete with limited size?
My events collection have 23M documents since July 2025.
I try delete a lot by one statement and my mongodb stuck until i reboot the container.
System details
Cosmos version: 0.22.1
Runs on Debian GNU/Linux 12 (bookworm) with LXC container(from Proxmox Virtual Environment 9.0.5).
I just assign 30GB to the container.
Host machine info:
CPU(s) 8 x Intel(R) Core(TM) i3-N305 (1 Socket)
Kernel Version Linux 6.14.8-2-pve (2025-07-22T10:04Z)