Skip to content

perf(shift-type): stop and clean up last_sync_of_checkin versions - #396

Open
sagarvora wants to merge 3 commits into
Aakvatech-Limited:version-15from
sagarvora:skip-shift-type-version-on-checkin-sync
Open

perf(shift-type): stop and clean up last_sync_of_checkin versions#396
sagarvora wants to merge 3 commits into
Aakvatech-Limited:version-15from
sagarvora:skip-shift-type-version-on-checkin-sync

Conversation

@sagarvora

@sagarvora sagarvora commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Biometric checkin sync PUTs last_sync_of_checkin to /api/resource/Shift Type/<name> on every run — one Version record per sync, forever.

Stop creating them

New before_validate hook on Shift Type sets doc.flags.ignore_version when a resource PUT updates only that field.

  • Bails out first on has_value_changed(last_sync_of_checkin), so the request is only inspected when the timestamp actually moved.
  • Reads the payload via frappe.api.v1.get_request_form_data — the sync sends a raw JSON body with no Content-Type, so make_form_dict leaves form_dict empty and only request.get_data() recovers it. Also covers form-encoded data=<json> and query params.
  • Gated to PUT on /api/resource/, /api/v1/resource/, /api/v2/document/. Desk saves, background jobs and /api/method/* are untouched.
  • Strict single-key match — a PUT touching last_sync_of_checkin plus any other field versions normally.

Clean up the ones already there

csf_tz/patches/delete_shift_type_checkin_sync_versions.py, deliberately not in patches.txt — deleting history is a judgement call. Run via bench execute; estimate first, it only reads.

  • Keyset-paginated on name over the indexed ref_doctype slice, deleted in small autocommitted batches. Resumable, and work is proportional to the Shift Type rows, not the table.
  • Matching is done in Python, not SQL: Version.data's key-set depends on the frappe version that wrote the row (v11 emits four keys, v15 six, bare {"changed": …} also occurs), so it requires changed to hold only our field and no other key to carry a value. An unrecognised key with content means the row is kept.
  • reclaim_space rebuilds the table afterwards to return space to the filesystem — optional, and wants a quiet window.

Measured 7k–13k rows/s deleting locally, so under two hours for 35M rows — one bench, not a promise.

Verification

Hook: fires for the sync's exact shape (raw body, no content-type) on all three resource paths; stays off for unchanged values, extra fields, other-fields-only, empty/absent body, POST, GET, /api/method/frappe.client.save, desk/background saves, and malformed payloads.

Patch: deletes the v15/v11/changed-only/null-previous shapes; keeps two-field diffs, row_changed/added/removed/data_import/updater_reference, unknown-key-with-content, insert markers, and malformed input. End-to-end Version counts and idempotent re-runs confirmed, plus batch boundaries across multi-batch walks.

The biometric checkin sync PUTs `last_sync_of_checkin` on every run,
creating a `Version` record each time. Set `ignore_version` in
`before_validate` when a resource PUT updates only that field.
Bail out before inspecting the request when `last_sync_of_checkin`
hasn't actually changed.
@sagarvora
sagarvora marked this pull request as draft August 13, 2026 13:28
Removes historical `tabVersion` rows that only record a
`last_sync_of_checkin` bump on Shift Type — the versions the hook now
prevents from being created.

Not in `patches.txt`; run via `bench execute` (`estimate` first, it only
reads). Keyset-paginated over the indexed `ref_doctype` slice and deleted
in small autocommitted batches, so it is resumable and safe on a large
table. Shape matching is done in Python since the `Version.data` key-set
varies by the frappe version that wrote the row.
@sagarvora sagarvora changed the title perf(shift-type): skip versions for last_sync_of_checkin sync perf(shift-type): stop and clean up last_sync_of_checkin versions Aug 13, 2026
@sagarvora
sagarvora marked this pull request as ready for review August 13, 2026 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant