Feature request: Configurable Event Processing Trace Level

[ig-quantum]

IncidentRelay stores a detailed processing trace for every event, which is very useful for troubleshooting and auditing event handling.

For deployments with a high event volume, it would be beneficial to make the trace level configurable. This would allow administrators to choose the appropriate balance between observability and storage usage.

For example, the application could support trace levels such as:

Full (current behavior)
Short processing (no data included)
Disabled

This would provide greater deployment flexibility while preserving the existing functionality as the default.

[Aidaho12]

Looks like usefull. btw trace db is cleaned up every 30 days by default. I have ~20k alerts and my db is ~2mb, how many alerts do you have? )

[ig-quantum]

Currently, I have ~6k alerts in the PostgreSQL DB.

incidentrelay=# select count(*) from "alert";
-------
count 
  6124
(1 row)

But problem is in related tables

SELECT
    schemaname,
    relname AS table_name,
    pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
    pg_size_pretty(pg_relation_size(relid)) AS table_size,
    pg_size_pretty(
        pg_total_relation_size(relid) - pg_relation_size(relid)
    ) AS indexes_and_toast
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;
 schemaname |              table_name              | total_size | table_size | indexes_and_toast
 
------------+--------------------------------------+------------+------------+------------------
-
 public     | alertexplainstep                     | 3042 MB    | 2331 MB    | 711 MB
 public     | alertexplaintrace                    | 855 MB     | 743 MB     | 112 MB
 public     | alertevent                           | 416 MB     | 276 MB     | 140 MB
 public     | alert_group                          | 52 MB      | 41 MB      | 11 MB
 public     | alert                                | 16 MB      | 12 MB      | 3672 kB

Today in was much bigger (20 k alerts for 7 days and 10Gb total), but I performed cascade deletions for all alerts older than 3 days

[Aidaho12]

Not bad. Will do it the next release too