Skip to content

[BUG]: Events log not deleted by cronjob #535

@Linniem

Description

@Linniem

What happened?

Found my server is crashed by out of disk space, and mongodb data is huge.

root@Cosmos:~# du -ahx / | sort -rh | head -n 50
...
13G     /var/lib/docker/volumes/cosmos-mongo-data-0OR/_data
13G     /var/lib/docker/volumes/cosmos-mongo-data-0OR
13G     /var/lib/docker/volumes
8.5G    /var/lib/docker/volumes/cosmos-mongo-data-0OR/_data/index-2--2112421272775686701.wt
...

In mongosh

> use COSMOS
< switched to db COSMOS
> db.COSMOS_events.totalSize();
< Long('12373024768')

COSMOS_events collection use 12,373,024,768 bytes (~= 12GB)

What should have happened?

Event log should delete by cronjob.

https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/CRON.go#L217

I found exist events log is older than one month ago.

date: 2025-09-01T17:58:45.536+00:00

Log in events collection shows that deleted count is always 0,

"cosmos.database.cleanup {"collection":"events","deleted":0}"

but notifications is work.

"cosmos.database.cleanup {"collection":"notifications","deleted":3}"

How to reproduce the bug?

Start a server and add an old log then run the cronjob i guess?

Relevant log output

Other details

The problem is cronjob delete documents where the field "Date" less then one month ago.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/cleanup.go#L22

But events use "date" to record time, and mongodb field is case-sensitive.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/events.go#L24

In my mongodb looks like:

{
  "_id": {
    "$oid": "68b5df0ff75fea67e3ae5d8b"
  },
  "data": {
    "bytes": {
      "$numberLong": "1523"
    },
    "hostname": "pve.mysite.com",
    "url": "/api2/json/cluster/resources",
    "route": "ProxmoxVE",
    "status": 200,
    "method": "GET",
    "clientID": "192.168.1.1",
    "time": 0.005945966
  },
  "object": "route@ProxmoxVE",
  "_search": "cosmos.proxy.response.ProxmoxVE.success {\"bytes\":1523,\"clientID\":\"192.168.1.1\",\"hostname\":\"pve.mysite.com\",\"method\":\"GET\",\"route\":\"ProxmoxVE\",\"status\":200,\"time\":0.005945966,\"url\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api2/json/cluster/resources\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"}}",
  "eventId": "cosmos.proxy.response.ProxmoxVE.success",
  "label": "Proxy Response ProxmoxVE success",
  "application": "Cosmos",
  "level": "info",
  "date": {
    "$date": "2025-09-01T17:58:45.536Z"
  }
}

Notification use "Date" so cronjob works.
https://github.com/azukaar/Cosmos-Server/blob/v0.22.4/src/utils/notifications.go#L205

{
  "_id": {
    "$oid": "69a66167aaab42ac54518c05"
  },
  "Read": true,
  "Recipient": "mysite",
  "Vars": "",
  "Icon": "",
  "Actions": null,
  "Title": "header.notification.title.serverError",
  "Message": "Your server has no internet connection! : Get \"https://www.google.com\": dial tcp: lookup www.google.com on 192.168.1.1:53: read udp 192.168.1.8:33636->192.168.1.1:53: i/o timeout",
  "Link": "",
  "Date": {
    "$date": "2026-03-03T04:19:11.173Z"
  },
  "Level": "error"
}

Addition note:
Do not just makes it delete events log by correct letter case, maybe batch delete with limited size?

My events collection have 23M documents since July 2025.
I try delete a lot by one statement and my mongodb stuck until i reboot the container.

> const sixMonthsAgo = new Date();
> sixMonthsAgo.setMonth(sixMonthsAgo.getMonth() - 6);
< 1759288902393
> db.COSMOS_events.find({ "date": { "$lt": sixMonthsAgo } }).count();
< 2359651
> db.COSMOS_events.deleteMany({ "date": { "$lt": sixMonthsAgo } }) 
# stuck here

# ctrl + c
> Error: Async script execution was interrupted

# reboot after few minutes and service alive

System details

Cosmos version: 0.22.1

Runs on Debian GNU/Linux 12 (bookworm) with LXC container(from Proxmox Virtual Environment 9.0.5).
I just assign 30GB to the container.

Host machine info:
CPU(s) 8 x Intel(R) Core(TM) i3-N305 (1 Socket)
Kernel Version Linux 6.14.8-2-pve (2025-07-22T10:04Z)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions