feat(paths): keep user files out of the directory a package manager owns

CHANGELOG.md

@@ -52,6 +52,19 @@ The format follows [Keep a Changelog](https://keepachangelog.com/); versions fol
   ignored rather than duplicated. Like every other row action they fill the form only - press
   "Apply changes" to put them into a running session.
 
+### Changed
+- **Your profiles, window state and CSV exports now live in your own user folder**
+  (`%LOCALAPPDATA%\BeanNetworkTester`) instead of the program folder. Files from an earlier
+  version are copied over on first start, and the originals stay where they were, so going back
+  to an older build still finds them. The reason is updates: a package manager owns the program
+  folder and replaces it, which would take your profiles with it. Both CSV exports now log the
+  whole path. Set `BEAN_DATA_DIR` to keep everything somewhere else, for example on a stick.
+- **"About" and `--doctor` now tell you which folder your files are in.** The folder belongs to
+  the Windows account the program runs as, so starting it elevated on an account that is not an
+  administrator uses that administrator's folder instead of yours. Nothing used to say so, which
+  made saved profiles look lost. Both READMEs explain it, and an administrator can set
+  `BEAN_DATA_DIR` system-wide to give every account one shared folder.
+
 ### Docs
 - **Four more guides on the website: no internet, timed scenarios, game lag and chaos testing.**
   How to take the internet away from one app while the local network keeps working, how a scenario

README.md

@@ -829,8 +829,21 @@ meant delivered: a row could read 5 MB received while its application got 0.4 MB
 
 ## CSV exports
 
-Two buttons write two files, and they behave **differently on purpose**. Both land next to the
-executable (or in the project root when running from source).
+Two buttons write two files, and they behave **differently on purpose**. Both land in your own
+folder, `%LOCALAPPDATA%\BeanNetworkTester`, together with your profiles and the window state (or
+in the project root when running from source). The program writes the full path into the log
+every time it saves one. They are kept out of the program folder so that updating the program -
+by hand or through a package manager, which replaces that folder - cannot take your files with
+it. Set `BEAN_DATA_DIR` to a folder of your choosing to keep everything somewhere else, for
+example on the same stick as a portable copy.
+
+The folder belongs to the Windows account the program is running as. On an account without
+administrator rights, agreeing to the elevation prompt runs the program as the administrator
+account whose password was entered, and it then uses THAT account's folder - so the profiles you
+saved without elevation are not the ones you see with it. "About" and `--doctor` both print the
+folder in use, so you can always tell which one you are looking at. To give every account on the
+machine one shared folder, an administrator can set `BEAN_DATA_DIR` as a system-wide environment
+variable.
 
 | | **Statistics** ("Export CSV", Statistics -> Live) | **Connections** ("Export connections CSV") |
 |---|---|---|
@@ -1184,6 +1197,7 @@ beantester/              the implementation package
     rates.py             throughput averaging (a pure, testable helper)
     scope.py             what the numbers on screen cover (one pure verdict)
     crash.py             what the GUI tells the crash logger: report context, breadcrumb
+    csv_export.py        the two CSV exports and the column names they write
     theme.py  chart.py  tooltip.py  profiles.py  icon.py  labels.py
 lang/                    translations (en, pl)
 tests/                   pytest tests

README.pl.md

@@ -688,8 +688,20 @@ dostarczone: wiersz mógł pokazywać 5 MB odebranych, gdy aplikacja dostała 0,
 
 ## Eksporty CSV
 
-Dwa przyciski zapisują dwa pliki i **celowo zachowują się inaczej**. Oba lądują obok pliku
-wykonywalnego (albo w korzeniu projektu przy uruchomieniu ze źródeł).
+Dwa przyciski zapisują dwa pliki i **celowo zachowują się inaczej**. Oba lądują w Twoim własnym
+folderze, `%LOCALAPPDATA%\BeanNetworkTester`, razem z profilami i stanem okna (albo w korzeniu
+projektu przy uruchomieniu ze źródeł). Program przy każdym zapisie wpisuje do logu pełną ścieżkę.
+Leżą poza folderem programu po to, żeby aktualizacja - ręczna albo przez menedżer pakietów, który
+ten folder podmienia - nie zabrała Twoich plików razem z nim. Ustaw `BEAN_DATA_DIR` na wybrany
+folder, żeby trzymać wszystko gdzie indziej, na przykład na tym samym pendrivie co kopia programu.
+
+Folder należy do konta Windows, na którym program działa. Na koncie bez uprawnień administratora
+zgoda na pytanie o podniesienie uprawnień uruchamia program jako to konto administratora, którego
+hasło zostało podane, więc używa TAMTEGO folderu - profile zapisane bez podniesienia uprawnień to
+nie są te, które widzisz z nim. Okno „O programie" i `--doctor` pokazują folder, z którego program
+właśnie korzysta, więc zawsze wiadomo, na który patrzysz. Żeby wszystkie konta na maszynie miały
+jeden wspólny folder, administrator może ustawić `BEAN_DATA_DIR` jako systemową zmienną
+środowiskową.
 
 | | **Statystyki** („Eksportuj CSV", Statystyki → Na żywo) | **Połączenia** („Eksportuj połączenia CSV") |
 |---|---|---|
@@ -1040,6 +1052,7 @@ beantester/              pakiet z implementacją
     rates.py             uśrednianie przepustowości (czysty, testowalny helper)
     scope.py             co obejmują liczby na ekranie (jeden czysty werdykt)
     crash.py             co GUI mówi logowi awarii: kontekst raportu i okruszek
+    csv_export.py        dwa eksporty CSV i nazwy kolumn, które zapisują
     theme.py  chart.py  tooltip.py  profiles.py  icon.py  labels.py
 lang/                    tłumaczenia (en, pl)
 tests/                   testy pytest

beantester/cli.py

@@ -25,7 +25,7 @@
 from .fields import BOOL, FIELD_DEFS
 from .filters import CLI_FILTERS
 from .i18n import T
-from .paths import is_frozen
+from .paths import is_frozen, user_data_dir
 from .presets import (PRESETS, closest_preset, preset_to_settings,
                       resolve_preset)
 from .repro import save_repro_report, settings_to_cli_string
@@ -438,12 +438,19 @@ def _run_license(log):
 
 def _run_doctor(log):
     ok, checks = driver.doctor()
+    # Where the user's own files are is an environment fact like the rest of this
+    # report, and it is the only one a person cannot look up: the folder follows
+    # the ACCOUNT the program runs as, so it moves when the same person starts it
+    # elevated onto another account. Not a check - it has no pass or fail - so it
+    # is a line of its own rather than a made-up state in the status column.
+    where = user_data_dir()
     if log.fmt == clilog.JSON:
-        log.data(dict(event="doctor", ok=ok,
+        log.data(dict(event="doctor", ok=ok, data_dir=where,
                       checks=[dict(check=c, state=st, detail=d) for c, st, d in checks]), "")
     else:
         for check, state, detail in checks:
             log.data(dict(), f"{state.upper():<4} {check:<18} {detail}")
+        log.data(dict(), f"user files: {where}")
     return exitcodes.OK if ok else exitcodes.RUNTIME
 
 

beantester/crashlog.py

@@ -65,7 +65,7 @@
 from datetime import datetime, timezone
 
 from .appinfo import __version__
-from .paths import app_dir
+from .paths import user_data_dir
 
 CRASH_DIR_NAME = "crashes"
 LOG_NAME = "crashes.ndjson"
@@ -91,8 +91,8 @@
 
 # -- where ------------------------------------------------------------------- #
 def crash_dir():
-    """Directory the crash files live in. Next to the executable, like the profiles."""
-    return os.path.join(app_dir(), CRASH_DIR_NAME)
+    """Directory the crash files live in - the user's data directory, by the profiles."""
+    return os.path.join(user_data_dir(), CRASH_DIR_NAME)
 
 
 def _ensure_dir():

beantester/gui/app.py

@@ -14,7 +14,6 @@
     purpose - no mixed languages),
   * every visible text is looked up through ``T()`` at widget-build time.
 """
-import csv
 import os
 import queue
 import sys
@@ -34,9 +33,10 @@
 from ..fields import FIELD_DEFS, SECTIONS, UI_ONLY_KEYS, off_value
 from ..filters import cli_key_for, i18n_key_for, i18n_keys, windivert_for
 from .. import crashlog
+from . import csv_export
 from ..i18n import (FALLBACK_LANGUAGE, T, available_languages, current_language,
                     field_name, set_language)
-from ..paths import CONNECTIONS_CSV_FILE, CSV_FILE, scenarios_dir
+from ..paths import prepare_user_data, scenarios_dir
 from ..presets import (PRESETS, preset_to_settings, resolve_preset,
                        settings_to_preset)
 from ..processes import port_process_map
@@ -47,7 +47,6 @@
                         settings_from_raw, warn_if_unbounded)
 from ..summary import settings_summary
 from ..utils import number_string
-from ..views import avg_packet_bytes, connection_proc, filter_sort_connections
 from . import crash as gui_crash
 from . import dialogs
 from .icon import (apply_window_icon, make_gear_icon, show_idle_icon,
@@ -105,6 +104,12 @@ def __init__(self, root):
         root.configure(bg=BG)
         apply_dark_titlebar(root)     # while still hidden, so it never flashes light
 
+        # BEFORE any store reads a file: create the data directory and adopt what an
+        # older build left next to the executable (see paths.user_data_dir). Both
+        # stores bind their path at import, so this cannot wait until the log exists -
+        # whatever went wrong is reported by _report_storage_problems instead.
+        self._data_problems = prepare_user_data()
+
         self.ui = UiStateStore()
         # Secondary windows live in a registry, exactly like the pages and the
         # settings fields do (gui/windows.py): a new window is one entry, and it
@@ -1029,152 +1034,15 @@ def _persist_profiles(self):
             self.log(f"{T('log.profiles_not_saved')}: {err}")
 
     # -- session actions ---------------------------------------------------------- #
-    # Internal stat keys are engine-speak ("seen"); a CSV is read by people and
-    # by spreadsheets, so it gets column names that mean something.
-    CSV_COLUMNS = {"seen": "packets_seen", "scoped_seen": "packets_in_scope",
-                   "drop_loss": "dropped_loss",
-                   "drop_overflow": "dropped_overflow", "drop_syn": "dropped_syn",
-                   "drop_mtu": "dropped_mtu", "drop_nat": "dropped_nat",
-                   "drop_rst": "dropped_rst", "rst_reset": "connections_reset",
-                   "drop_lan": "dropped_lan",
-                   "drop_block": "dropped_block",
-                   "drop_flap": "dropped_link_outage", "drop_rate": "dropped_rate_limit",
-                   "drop_shutdown": "dropped_at_stop",
-                   "drop_send": "dropped_send_failed",
-                   "queue": "queue_len",
-                   "peak_queue": "queue_peak",
-                   # The stats CSV deliberately does NOT follow the "show only the
-                   # targeted traffic" preference: it is an APPEND log, and a file
-                   # whose columns mean one thing in some rows and another in the
-                   # rest is worse than useless for the spreadsheet it exists for.
-                   # It gains both totals instead, so the reader can do the
-                   # narrowing themselves and see which is which.
-                   "bytes_in_scoped": "delivered_in_scope_bytes_down",
-                   "bytes_out_scoped": "delivered_in_scope_bytes_up"}
-
-    # Columns that come from the SESSION rather than the counters: they say which
-    # world the counters were measured in. The comment above explains why this
-    # file must not follow the view preference - a column meaning one thing in
-    # some rows and another in the rest is useless to the spreadsheet it exists
-    # for. Capture narrowing is the STRONGER version of that problem and had no
-    # column at all: it does not pick between two totals, it changes `seen`
-    # itself, so a narrowed row and a wide row were indistinguishable while
-    # counting completely different traffic. The CLI has carried the same fact in
-    # its JSON summary (`capture_narrowed`) since narrowing shipped, and the repro
-    # report carries the whole `session_info` - only this file could not say.
-    # Keyed by session_info() key, like CSV_COLUMNS is keyed by counter key.
-    CSV_SESSION_COLUMNS = {"narrowed": "capture_narrowed"}
-
-    @staticmethod
-    def _csv_session_value(key, info):
-        """One session cell. Booleans as yes/no in English, like `impaired` in the
-        connections export - a CSV is read by scripts and spreadsheets, so it does
-        not follow the interface language."""
-        value = info.get(key)
-        return ("yes" if value else "no") if isinstance(value, bool) else value
-
+    # The two exports live in gui/csv_export.py - they are file work, not window
+    # work, and this file is the one at the size ceiling (convention 32b). The
+    # methods stay because every caller uses them: two pages, the smoke script,
+    # the tests and the `_export_csv` alias at the bottom of this file.
     def export_csv(self):
-        snap = self.engine.stats_snapshot()
-        info = self.engine.session_info()
-        session_cells = [self._csv_session_value(k, info)
-                         for k in self.CSV_SESSION_COLUMNS]
-        header = ["time", *self.CSV_SESSION_COLUMNS.values(),
-                  *(self.CSV_COLUMNS.get(k, k) for k in snap)]
-        try:
-            write_header = not os.path.exists(CSV_FILE)
-            if not write_header:
-                with open(CSV_FILE, newline="", encoding="utf-8") as f:
-                    existing = next(csv.reader(f), [])
-                if existing != header:
-                    # the stat columns changed between versions: appending would
-                    # silently misalign rows against the old header
-                    backup = CSV_FILE[:-4] + time.strftime(".%Y%m%d-%H%M%S.csv")
-                    os.replace(CSV_FILE, backup)
-                    self.log(f"{T('log.csv_rotated')} {os.path.basename(backup)}")
-                    write_header = True
-            with open(CSV_FILE, "a", newline="", encoding="utf-8") as f:
-                writer = csv.writer(f)
-                if write_header:
-                    writer.writerow(header)
-                writer.writerow([time.strftime("%Y-%m-%d %H:%M:%S"),
-                                 *session_cells, *snap.values()])
-            self.log(f"{T('log.stats_saved_to')} {os.path.basename(CSV_FILE)}")
-        except Exception as e:
-            self.log(f"{T('log.csv_error')}: {e}")
-
-    # Mirrors the table's columns so the export is the table, on disk. Raw bytes,
-    # not KB: a CSV is read by a spreadsheet or a script, where exact, summable
-    # integers beat the one-decimal KB the table shows for people. `impaired` is
-    # "yes"/"no" in English, like the headers - the CSV is language-independent.
-    # delivered_* is what reached the application; captured_* is what the tool
-    # saw offered. The old download_bytes/upload_bytes/total_bytes held CAPTURED
-    # under names every other surface uses for delivered, so they are renamed
-    # rather than reused - a column that quietly changes meaning is worse than one
-    # that disappears.
-    CONN_CSV_HEADER = ["process", "pid", "proto", "remote_ip", "remote_port",
-                       "local_port", "packets", "impaired", "dropped",
-                       "delivered_down_bytes", "delivered_up_bytes",
-                       "delivered_total_bytes",
-                       "captured_down_bytes", "captured_up_bytes",
-                       "captured_total_bytes",
-                       "avg_bytes", "duration_s", "idle_s"]
+        csv_export.export_stats(self)
 
     def export_connections_csv(self):
-        """Write the CURRENT connection view (search + sort) to a CSV snapshot.
-
-        The display row-limit is a rendering cap, not part of what the user asked
-        to see, so the export carries every filtered row - sorted the same way the
-        table is. The file is overwritten atomically each time (tmp + os.replace):
-        it is a snapshot of "the connections as they are now", not an append log
-        like the stats CSV.
-        """
-        now = self.engine.now_ref()
-        snapshot = self.engine.connections_snapshot(limit=None)
-        # The export follows the view: what you exported and what you were looking
-        # at have to be the same set, or the file quietly disagrees with the screen
-        # that produced it.
-        if self.scoped_view():
-            snapshot = [c for c in snapshot if c.get("scoped")]
-        rows = filter_sort_connections(
-            snapshot, self.conn_query,
-            self.conn_sort["col"], self.conn_sort["reverse"],
-            now=now, proc_map=self.proc_map, limit=0)
-        path = CONNECTIONS_CSV_FILE
-        tmp = path + ".tmp"
-        try:
-            with open(tmp, "w", newline="", encoding="utf-8") as f:
-                writer = csv.writer(f)
-                writer.writerow(self.CONN_CSV_HEADER)
-                for c in rows:
-                    last = c.get("last", now)
-                    packets = c.get("packets", 0) or 0
-                    writer.writerow([
-                        connection_proc(c, self.proc_map) or "?",
-                        c.get("pid") or "",
-                        c.get("proto", "IP"), c.get("remote_ip", ""),
-                        c.get("remote_port", ""), c.get("local_port", ""),
-                        packets, "yes" if c.get("scoped") else "no",
-                        c.get("dropped", 0),
-                        c.get("sent_in", 0), c.get("sent_out", 0), c.get("sent", 0),
-                        c.get("bytes_in", 0), c.get("bytes_out", 0), c.get("bytes", 0),
-                        avg_packet_bytes(c),
-                        f"{max(0.0, last - c.get('first', now)):.1f}",
-                        f"{max(0.0, now - last):.1f}"])
-            os.replace(tmp, path)
-            self.log(f"{T('log.conns_saved_to')} {os.path.basename(path)} ({len(rows)})")
-        except Exception as e:
-            # Clean up the half-written temp file, the way jsonfile.write_json
-            # already does. Without this a failed export left a `.csv.tmp` next to
-            # the real file for the user to find and wonder about - and the next
-            # export silently overwrote it, so the litter was never even stable.
-            # A failure here must leave the previous export untouched and nothing
-            # else behind.
-            try:
-                if os.path.exists(tmp):
-                    os.remove(tmp)
-            except OSError as _exc:
-                crashlog.note(_exc, "gui.app")
-            self.log(f"{T('log.csv_error')}: {e}")
+        csv_export.export_connections(self)
 
     def mark_bug(self):
         if not self.running:
@@ -1744,6 +1612,10 @@ def on_close(self):
 
     def _report_storage_problems(self):
         """A profile file that vanished or broke must SAY so, not just be gone."""
+        # No reset, unlike the stores below: this list is filled once, before the
+        # window exists, and _check_environment reads it once.
+        for problem in self._data_problems:
+            self.log(f"{T('log.data_files_problem')}: {problem}")
         for store, key in ((self.profiles, "log.profiles_problem"),
                            (self.ui, "log.ui_state_problem")):
             problem = getattr(store, "problem", None)