Split large pending writes queue entries by ohadzeliger · Pull Request #3970 · FoundationDB/fdb-record-layer

ohadzeliger · 2026-02-25T16:57:51Z

This PR allows Pending Writes Queue entries to be split using the SplitHelper so that we can save large entries in the queue. It contains both the SplitHelper work as well as the PendingWriteQueue work.
On the SplitHelper the work introduces an abstraction of the key construction and value writing in order to support VersionInKey - the concept of having an incomplete version in the keys of the split records. There are two implementations: DefaultKeyValueSplitHelper (the current behavior) and VersioningKeyValueSplitHelper (the new one in support of the queue).
Care was taken to ensure no changes occur to the existing behavior (for backwards compatibility). The original test SplitHelperTest remains almost intact, while a new test, SplitHelperTestMultipleTransactions was added to test the new functionality. This new test borrows heavily from the other test, with the intent of testing the behavior across multiple transactions (since we have to commit the first in order to complete the version in the key so that we can locate the entry).
Two test cases, saveWithSplitAndIncompleteVersions and deleteWithSplitAndVersion were not copied over as they had no relevance for the new VersionInKey.

This is considered a breaking change, as it changes the data structure of the Lucene index's pending writes queue. The index should be disabled before the deployment and re-enabled afterwards.

Resolves #3981

# Conflicts: # fdb-record-layer-lucene/src/main/java/com/apple/foundationdb/record/lucene/directory/FDBDirectory.java # fdb-record-layer-lucene/src/main/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueue.java # fdb-record-layer-lucene/src/test/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueueTest.java

Adds a useVersionInKey boolean to SplitHelperTestConfig, a keyHelper(int localVersion) factory method, and extends allValidConfigs() to produce versioning variants of all existing configs (excluding isDryRun=true, which is incompatible with versionstamps). This lays the groundwork for multi-transaction test methods that exercise VersioningSplitKeyValueHelper alongside the existing DefaultSplitKeyValueHelper tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix a bug in SplitHelper.writeSplitRecord where split keys were packed using subspace.subspace(key) + Tuple.from(index), producing an inconsistent layout compared to the unsplit path for VersioningSplitKeyValueHelper. Now both call packSplitKey(subspace, key.add(index)), which VersioningSplitKeyValueHelper handles as [subspace][vs][key][index] — consistent with [subspace][vs][key][0] for unsplit records. Add SplitHelperTestConfig.useVersionInKey and keyHelper(localVersion) to drive tests through VersioningSplitKeyValueHelper. Extend allValidConfigs() to cover both helpers. Add Assumptions.assumeFalse guards where tests are incompatible with versioning keys. Introduce saveOnly + verifySuccessfullySaved helpers split from saveSuccessfully, and new saveWithSplitMultipleTransactions test that writes in tx1 and verifies in tx2, covering both default and versioning key helpers. Add parameterized defaultHelperSplitKeyEquivalence test asserting that the old call site subspace.subspace(key).pack(index) equals the new subspace.pack(key.add(index)) for DefaultSplitKeyValueHelper across all relevant key/index combinations. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…eTransactions - Add useVersionInKey and localVersion parameters to writeDummyRecord, threading both through to writeDummyKV - writeDummyKV now handles versionstamp prepending and FDB write dispatch (SET_VERSIONSTAMPED_KEY vs plain set) based on useVersionInKey - Remove writeDummyVersionedRecord (now redundant with writeDummyRecord) - Add deleteWithSplitMultipleTransactions: write in tx1, delete in tx2, verify empty in tx3, covering both default and versioning key helpers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…eTransactions - Add writeDummyRecordsMultipleTransactions(boolean useVersionInKey): writes 50 records in tx1 (with FDBRecordVersion when useVersionInKey=false, with versionstamp keys when true), then scans back in tx2 to build expected FDBRawRecord list with accurate committed sizes - Add scanMultipleRecordsWithVersioningKey: parallel to scanMultipleRecords, uses writeDummyRecordsMultipleTransactions(true) for setup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Parallel to scanContinuations, uses writeDummyRecordsMultipleTransactions(true) for setup so records are written with versionstamp keys and read back in a separate transaction with accurate committed sizes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… key, added comments

…est to original, some tests converted.

ScottDugas

There are a fair number of comments about how the clarity of the test could be improved. For anything other than quick one-line changes, it probably makes sense to merge and followup with improved readability of the test, so that we can be confident we're testing the same thing as before.

...ain/java/com/apple/foundationdb/record/provider/foundationdb/DefaultSplitKeyValueHelper.java

ScottDugas · 2026-03-02T19:42:26Z

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

-            if (splitLongRecords || previousSizeInfo == null || previousSizeInfo.isVersionedInline()) {
+            if (splitKeyHelper.shouldClearBeforeWrite() && (splitLongRecords || previousSizeInfo == null || previousSizeInfo.isVersionedInline())) {
+                // Note that the clearPreviousSplitRecords also removes version splits from the context cache
+                // This is not currently supported for the case where we have versions in the keys since we can't trace the old values down


If we can't know the previous value, how can we have a previousSizeInfo?

When shouldClearBeforeWrite()==true the assumption is that we have values we can use to clear the previous keys (with or without the sizeInfo).
when shouldClearBeforeWrite()==false then sizeInfo is ignored as we don't call clear at all (and chances are we will see null there).

With the changed logic, if the key has an incomplete version, then we can't clear older records based on size info since the size info either has complete version (previous transaction was committed) or it has incomplete versions (in which case the keys are not in the RYW cache)

...ain/java/com/apple/foundationdb/record/provider/foundationdb/DefaultSplitKeyValueHelper.java

...e/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitKeyValueHelper.java

ScottDugas · 2026-03-02T22:08:10Z

...com/apple/foundationdb/record/provider/foundationdb/SplitHelperMultipleTransactionsTest.java

+ * -
+ */
+@Tag(Tags.RequiresFDB)
+public class SplitHelperMultipleTransactionsTest extends FDBRecordStoreTestBase {


Probably not as part of this PR, but it might be worth some sort of chaos testing. (i.e. random key shapes, random bytes for values of random lengths, etc.)

...r-lucene/src/main/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueue.java

...e/src/main/java/com/apple/foundationdb/record/lucene/directory/PendingWritesQueueHelper.java

...cene/src/test/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueueTest.java

alecgrieser · 2026-03-04T14:52:09Z

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

     * @param serialized serialized representation
     * @param version the version to store inline with this record
     */
    public static void saveWithSplit(@Nonnull final FDBRecordContext context, @Nonnull final Subspace subspace,


One to look out for is reading/deleting an item within the same transaction as the one that inserted it. That may mean fishing through the versionMutationCache on the FDBRecordContext for keys with versions that haven't been updated if that hasn't already been done.

alecgrieser · 2026-03-04T14:58:14Z

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

-            if (splitLongRecords || previousSizeInfo == null || previousSizeInfo.isVersionedInline()) {
+            if (splitKeyHelper.shouldClearBeforeWrite() && (splitLongRecords || previousSizeInfo == null || previousSizeInfo.isVersionedInline())) {
+                // Note that the clearPreviousSplitRecords also removes version splits from the context cache
+                // This is not currently supported for the case where we have versions in the keys since we can't trace the old values down


Maybe I don't understand this, but it seems like we should be able to do this. The versionMutationCache always uses a fixed value (all \xff bytes) for the uncommitted version, so two keys with the same FDBRecordVersion in it should appear in the same location of the versionMutationCache.

As it happens, I think this is fine for the use case of the pending writes queue, as a different local version is claimed with each insert. Theoretically, you could imagine leveraging this code to support versionstamps in primary keys (which we currently don't, but this gets us pretty close to being able to support), in which case this may be more important

The issue faced was that the cache contains specific keys with split suffixes which can span multiple cache entries. In the "incomplete version in key" case we would need to replace them all when a new record gets written with the same local version. One way to introduce that would be to group all the splits by the local version and replace the entire group every time, but that seems a little excessive since there is no use case for this currently.

I think we do group all of the keys by their local version. The mutations currently go into a NavigableMap, which is sorted by unsigned byte order. Incomplete versions all get assigned 10 \xff bytes as their global version, and so they'll all be there in the cache in consecutive keys. If there's no use case for updating large values within the same transaction as they're included, then we could punt this, though

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

...e/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitKeyValueHelper.java

.../java/com/apple/foundationdb/record/provider/foundationdb/VersioningSplitKeyValueHelper.java

...r-lucene/src/main/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueue.java

alecgrieser · 2026-03-04T17:04:21Z

...-core/src/test/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelperTest.java

-
-    public static Stream<Arguments> testConfigs() {
-        return SplitHelperTestConfig.allValidConfigs().map(Arguments::of);
+    public static Stream<Arguments> testConfigsNoVersionInKey() {


It's a bit odd that the way this turned out, all of the tests wound up with the NoVersionInKey case. It kind of seems like what you'd want is for testConfigs to return the same set of configs as earlier, and then you could test out instances where the key has a version in separate cases in this file

This was the original attempt. Things got complicated as the versionInKey required structural changes to the tests (save in one transaction, keep local and global version, construct actual key, assert in another transaction), which was incompatible with the structure of the test.
I think that a future enhancement (separate from the actual refactoring) would be to break the large sequential tests into smaller ones, where we could affect the way the assertions are made (one or two transactions) separately from the actual flow of the test logic.

...com/apple/foundationdb/record/provider/foundationdb/SplitHelperMultipleTransactionsTest.java

alecgrieser · 2026-03-04T17:25:42Z

...com/apple/foundationdb/record/provider/foundationdb/SplitHelperMultipleTransactionsTest.java

+public class SplitHelperMultipleTransactionsTest extends FDBRecordStoreTestBase {
+
+    // From the traditional nursery rhyme
+    private static final byte[] HUMPTY_DUMPTY =


I also noted the duplication. Given that the distinction seems to just be whether the methods use versionstamp or not, I think this could mostly just be combined with the other tests. I suppose a common helper could be extracted if we wanted to separate out the versionstamp and non-versionstamp tests for code organization purposes. But I don't think, for example, that the multiple transaction-ness of these causes much issue, if they lived in the same file as the others

...com/apple/foundationdb/record/provider/foundationdb/SplitHelperMultipleTransactionsTest.java

alecgrieser · 2026-03-04T19:04:57Z

I haven't totally gone through all of the tests, but I figured I'd add what I had before taking off for the day

alecgrieser

There's some concern about the particulars of the API here, but I think the approach is sound, in that I don't believe it would cause trouble for existing data, and it does seem like it would enable large values into the PendingWriteQueue. Fixing up the API in a follow up seems acceptable

ScottDugas

We discussed offline that we should take in some of the api changes, and test improvements, but I have some comments on the most recent PR changes.

...e/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitKeyValueHelper.java

...com/apple/foundationdb/record/provider/foundationdb/SplitHelperMultipleTransactionsTest.java

ScottDugas · 2026-03-05T16:32:46Z

...cene/src/test/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueueTest.java

    @Nonnull
    private TestDocument createHugeDocument() {
-        String hugeString = "Hello ".repeat(100_000);
+        Random random = ThreadLocalRandom.current();


Probably better to use @RandomSeedSource, although I would be quite surprised if this test changes behavior with a different seed.

The calling methods use @source already, not sure how to combine that with @RandomSeedSource

The RandomSeedSource itself is based on RandomizedTestUtils.randomSeeds:

fdb-record-layer/fdb-test-utils/src/main/java/com/apple/test/RandomSeedProvider.java

Line 46 in 155beb8

return RandomizedTestUtils.randomSeeds(fixedSeeds).map(Arguments::of);

I think the only provider is a boolean provider, so creating a method arguments provider that returns something like:

static Stream<Arguments> seedAndUseCompression() { return RandomizedTestUtils.randomSeeds(/* some array of fixed seeds */) .flatMap(seed -> Stream.of(Arguments.of(seed, false), Arguments.of(seed, true)); }

Would do it, I think

Yeah, you can't use @RandomSeedSource, but there's actually a cleaner way to do that, you could do:

static Stream<Arguments> seedAndUseCompression() { return ParameterizedTestUtils.cartesianProduct( RandomizedTestUtils.randomSeeds(/* some array of fixed seeds */), ParameterizedTestUtils.booleans("useCompression")); }

Using cartesianProduct just makes this code a bit cleaner (IMHO), using booleans means that the description of the arguments will be useCompression/!useCompression in our test logs/ides which is clearer than just true/false

ScottDugas · 2026-03-05T16:35:21Z

...cene/src/test/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueueTest.java


    @Nonnull
    private TestDocument createHugeDocument() {
-        String hugeString = "Hello ".repeat(100_000);


Is there any value in a different test that has a document that compresses well. I don't think there's any max size for SplitHelper, so I don't know that you could observe any change in behavior, other than trying to save something that is 10,000,000 bytes: https://apple.github.io/foundationdb/known-limitations.html but very compressible. I'm not sure we want a test using 10mb of memory just for this.

I don't think there is a limit on the total record size, so there is no discernible difference between N splits and 500*N splits, AFAICT

That limit, is the limit on the amount data you can have in a single transaction. So if you tried to save a 11,000,000 byte string of "repeat me" it would save successfully with compression, and fail if it just had the splits.

jjezra

LGTM

alecgrieser

LGTM. There are a few things in the tests, I suppose, but we could do that later. I do also think that there's an approach here that would let us remove any keys that were previously written within the same transaction, but I'm okay with punting that for a later improvement

alecgrieser · 2026-03-10T18:14:36Z

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

+            // an incomplete version in the key means we shouldn't delete previous values for the record (since they all have
+            // completed versions by now)


I don't think this is quite right. The keys only get associated with a complete version once they've been committed. Saying that we don't currently support updating records with incomplete versions in the key may be more accurate

alecgrieser · 2026-03-10T18:26:05Z

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

-            if (splitLongRecords || previousSizeInfo == null || previousSizeInfo.isVersionedInline()) {
+            if (splitKeyHelper.shouldClearBeforeWrite() && (splitLongRecords || previousSizeInfo == null || previousSizeInfo.isVersionedInline())) {
+                // Note that the clearPreviousSplitRecords also removes version splits from the context cache
+                // This is not currently supported for the case where we have versions in the keys since we can't trace the old values down


I think we do group all of the keys by their local version. The mutations currently go into a NavigableMap, which is sorted by unsigned byte order. Incomplete versions all get assigned 10 \xff bytes as their global version, and so they'll all be there in the cache in consecutive keys. If there's no use case for updating large values within the same transaction as they're included, then we could punt this, though

alecgrieser · 2026-03-11T09:30:44Z

...com/apple/foundationdb/record/provider/foundationdb/SplitHelperMultipleTransactionsTest.java

+
+    private List<FDBRawRecord> writeDummyRecordsMultipleTransactions(boolean useVersionInKey) {
+        final byte[] valueVersion = "_cushions_".getBytes(StandardCharsets.US_ASCII);
+        // Generate primary keys using a generalization of the Fibonacci formula: https://oeis.org/A247698


I believe I wrote the original version of this, and there's no reason besides it being a little more whimsical. That's also why it's _cushions_ (both are in oblique references to the same person, who is also referenced in the OEIS entry)

alecgrieser · 2026-03-11T09:35:48Z

...cene/src/test/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueueTest.java

    @Nonnull
    private TestDocument createHugeDocument() {
-        String hugeString = "Hello ".repeat(100_000);
+        Random random = ThreadLocalRandom.current();


The RandomSeedSource itself is based on RandomizedTestUtils.randomSeeds:

fdb-record-layer/fdb-test-utils/src/main/java/com/apple/test/RandomSeedProvider.java

Line 46 in 155beb8

return RandomizedTestUtils.randomSeeds(fixedSeeds).map(Arguments::of);

I think the only provider is a boolean provider, so creating a method arguments provider that returns something like:

static Stream<Arguments> seedAndUseCompression() { return RandomizedTestUtils.randomSeeds(/* some array of fixed seeds */) .flatMap(seed -> Stream.of(Arguments.of(seed, false), Arguments.of(seed, true)); }

Would do it, I think

ScottDugas · 2026-03-11T10:49:28Z

...-core/src/test/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelperTest.java

     */
    @MethodSource("splitKeyEquivalenceCases")
    @ParameterizedTest(name = "defaultHelperSplitKeyEquivalence[key={0}, index={1}]")
    void defaultHelperSplitKeyEquivalence(Tuple key, long index) {


The name here is probably a bit confusing given that DefaultSplitKeyValueHelper no longer exists. Perhaps this test is no longer necessary?

ScottDugas · 2026-03-11T10:52:16Z

...com/apple/foundationdb/record/provider/foundationdb/SplitHelperMultipleTransactionsTest.java


+    private Tuple toIncompleteKey(Tuple key, int localVersion, boolean versionInKey) {
+        if (versionInKey) {
+            return Tuple.from(Versionstamp.incomplete(localVersion)).addAll(key);


I don't have much concern, but it would probably be good to followup with some tests where the version is elsewhere in the key

ScottDugas · 2026-03-11T11:03:51Z

...cene/src/test/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueueTest.java

    @Nonnull
    private TestDocument createHugeDocument() {
-        String hugeString = "Hello ".repeat(100_000);
+        Random random = ThreadLocalRandom.current();


Yeah, you can't use @RandomSeedSource, but there's actually a cleaner way to do that, you could do:

static Stream<Arguments> seedAndUseCompression() { return ParameterizedTestUtils.cartesianProduct( RandomizedTestUtils.randomSeeds(/* some array of fixed seeds */), ParameterizedTestUtils.booleans("useCompression")); }

Using cartesianProduct just makes this code a bit cleaner (IMHO), using booleans means that the description of the arguments will be useCompression/!useCompression in our test logs/ides which is clearer than just true/false

ScottDugas · 2026-03-11T11:14:40Z

...cene/src/test/java/com/apple/foundationdb/record/lucene/directory/PendingWriteQueueTest.java


    @Nonnull
    private TestDocument createHugeDocument() {
-        String hugeString = "Hello ".repeat(100_000);


That limit, is the limit on the amount data you can have in a single transaction. So if you tried to save a 11,000,000 byte string of "repeat me" it would save successfully with compression, and fail if it just had the splits.

ScottDugas · 2026-03-11T11:17:57Z

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

+            if (current != null) {
+                // This should never happen. It means that the same key (and suffix) and local version were used for subsequent
+                // write. It is most likely not an intended flow and this check will protect against that.
+                // It is an incomplete check since the same record can have different suffixes to the primary key that would not collide.


Suggested change

// It is an incomplete check since the same record can have different suffixes to the primary key that would not collide.

// It is an incomplete check since the same record can have different suffixes to the primary key that would not collide.

// Namely, if one is split, and one is not split they will not overlap. Anything else and the would.

ScottDugas · 2026-03-11T11:21:18Z

...ayer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/SplitHelper.java

     * @param sizeInfo optional size information to populate
     */
    @SuppressWarnings("PMD.CloseResource")
    public static void saveWithSplit(@Nonnull final FDBRecordContext context, @Nonnull final Subspace subspace,


Does it make sense with this new api to support dryRun in the same way?
I don't think it should be done in this PR, and not really a priority, but there is some inconsistency that might be good resolving eventually.

ohadzeliger and others added 19 commits February 13, 2026 16:29

Initial commit of splitting queue entries

f3ded4a

Style

d7e130d

Implement serializer and allowed version/clear options

a3ef2ef

Rebase from main

6bdd1cf

Complete unsplitting

783c3b1

Small fixes.

8656bdb

Add more tests for large document.

fc24b5b

cleanup comments

7071e40

Add tests for the case where we overwrite same record with version in…

0f89ece

… key, added comments

Create separate test with multiple transaction, revert split helper t…

e7dd234

…est to original, some tests converted.

Convert more tests

cf54a2d

Cleanup

05c958e

More test changes, extract TestConfig and reuse from both test classes.

fb0085c

ohadzeliger changed the title ~~Draft PR for Split queue entries~~ Split large pending writes queue entries Mar 2, 2026

ohadzeliger requested review from ScottDugas and jjezra and removed request for ScottDugas March 2, 2026 15:57

ohadzeliger marked this pull request as ready for review March 2, 2026 15:57

ohadzeliger self-assigned this Mar 2, 2026

ohadzeliger added enhancement New feature or request Run mixed-mode Label to add to Pull Requests to have it run mixed mode tests labels Mar 2, 2026

ohadzeliger requested a review from ScottDugas March 2, 2026 16:44

typos, hashcode

8f7038e

ScottDugas requested changes Mar 2, 2026

View reviewed changes

PR comments.

c1b641c

ohadzeliger added the breaking change Changes that are not backwards compatible label Mar 3, 2026

alecgrieser requested changes Mar 4, 2026

View reviewed changes

alecgrieser reviewed Mar 4, 2026

View reviewed changes

ohadzeliger added 4 commits March 4, 2026 16:32

PR comments: Add backwards-compatible method to API

1a68f22

More PR comments

d7deeed

Style

e9da6a3

Split two test methods that were too long

f93aa9f

alecgrieser approved these changes Mar 5, 2026

View reviewed changes

ScottDugas requested changes Mar 5, 2026

View reviewed changes

jjezra reviewed Mar 6, 2026

View reviewed changes

ohadzeliger added 4 commits March 7, 2026 20:18

Revert API changes, use tuple.hasIncompleteVersion, adjust tests

96eccea

More refactoring, add asserts for key size in tests

4524208

Removed helper and implementation

db31556

More test fixes

3e68faa

alecgrieser approved these changes Mar 11, 2026

View reviewed changes

ScottDugas approved these changes Mar 11, 2026

View reviewed changes

		// an incomplete version in the key means we shouldn't delete previous values for the record (since they all have
		// completed versions by now)

	// It is an incomplete check since the same record can have different suffixes to the primary key that would not collide.
	// It is an incomplete check since the same record can have different suffixes to the primary key that would not collide.
	// Namely, if one is split, and one is not split they will not overlap. Anything else and the would.

Conversation

ohadzeliger commented Feb 25, 2026 • edited by ScottDugas Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ScottDugas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ohadzeliger Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alecgrieser commented Mar 4, 2026

Uh oh!

alecgrieser left a comment

Choose a reason for hiding this comment

Uh oh!

ScottDugas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjezra left a comment

Choose a reason for hiding this comment

Uh oh!

alecgrieser left a comment

Choose a reason for hiding this comment

ohadzeliger commented Feb 25, 2026 •

edited by ScottDugas

Loading

ohadzeliger Mar 9, 2026 •

edited

Loading