Overlap trace adjustment

[raymondwjang]

🔍 Type of Change

✨ New Feature: Introducing new functionality

📝 Description

Another way to alleviate negative residuals.

The extend-pipeline-tests (#132) deals with overlap by purging negative residuals. This has a benefit of "deprecating" components that do not fit well against the residuals. However, it can also cause correct components to be removed (GradualOnSource test gets more chances of failing.) Protecting the "correct" estimators is critical to reaching a solution convergence. Another solution is proposed to ensure correct estimators aren't compromised, while still managing incorrect overlap footprints.

Explanation

There are largely two circumstances that a negative residual can occur:

• Setup 1: A and B are overlapping. We know of A only. (Simulated in Gradual-on Test)
  1. B begins brightening, driving up trace for A due to the overlapping area becoming brighter.
  2. This causes an overestimation in A that does not overlap B (A - B hereon), causing a negative residual.
  3. At the same time, there's an underestimation in A ∩ B, since trace increase was modulated by the brightness of A - B, and thus couldn't fully reflect the brightness at the overlap.
  4. ii and iii causes a negative residual in A - B and a positive residual in A ∩ B.

*if B's brightness decreases instead, the residual will be negative in A ∩ B and positive in A - B.

• Setup 2: A and B are overlapping. We know of A U B only. (Simulated in Split-off Test)
  1. A and B have been synchronized so far, causing the model to mark the combined footprint as a single component (A U B)
  2. They begin deviating. Let's say B begins dimming.
  3. This pulls the trace estimation for the A U B down.
  4. This causes overestimation in B - A, and underestimation in A - B, and something indeterminable in A ∩ B.
  5. iv implies a negative residual in B - A, and a positive residual in A - B.

*if B grows brighter instead, the residual signs will be opposite. This is symmetric.

Solutions

Negative residuals are nonphysical. We propose two ways of dealing with them, while encouraging an accurate cell detection in the process.

• Purge Negative Footprints:
  The mantra here is: "Oh, we have detected a negative residual. Dismember all suspects that may have caused it."

  1. Given an incoming frame, we have a set of footprints and a newly estimated traces.
  2. We calculated the residuals on this new frame and get pixels with negative residuals.
  3. All footprint values at these negative residual pixels are floored to zero.
  4. Footprints that are "remnants" (thickness of < n pixels) are entirely deleted.
  5. Residual is recalculated (it should be same as the raw incoming frame value.)
  6. New components are detected with these residuals.
  • This solution is very good at keeping the number of estimators under control. However, it causes a critical issue in Setup 1: Even though we begin with a correct footprint for A, after 4., we end up removing a chunk of it. A hotfix is introduced, where the user can set a negative "buffer threshold" - giving a room for the negative value in Setup 1 to avoid getting purged while waiting for the detection step to build a new component for B (and thus A ∩ B). This assumes the negative residuals in Setup 2 will eventually go over the buffer threshold.

• Trace Throttling:
  This mantra is: "Yes, the residual is negative, but surely we saw something here at one point, since we have a footprint. Let's adjust the traces so that the residual minimum becomes zero. The amplified residuals everywhere else, when accounted for during the detection step, will give us a better picture."

  1. Given an incoming frame, we have a set of footprints and a newly estimated traces.
  2. We calculated the residuals on this new frame and get pixels with negative residuals.
  3. We get the non-overlapping parts of the footprints. (Still investigating what should be done for completely occluded components. Currently, we do nothing with them.)
  4. Using the pixels where the residual is negative and there exists a non-overlapping part of a component, we calculate how much the component's trace has to decrease by to make the residual zero.
  5. Thus, the latest traces for all components are updated.
  6. This naturally means the residuals overall increase everywhere (since the traces for entire components are decreasing, residuals necessarily increase.)
  7. New components are detected with the residuals.
  • This solution does not delete or reduce footprints, and thus can cause overfitting issues by proliferating number of estimators. However, it is very good at resolving the issue caused by the Purge Negative Footprints method precisely for the same reason.

When it comes to Gradual-On test and Split-Off test, the performances of the two are quite similar, with Trace Throttling with a better final trace results. This could be a stochastic result, and more tests/reasoning are necessary to produce a better conclusion.

🧪 Testing

• Ran unit tests
• Ran integration tests
• Performed manual testing
• Updated existing tests

🛠️ Dependencies

✅ Checklist

• My code follows the project's style guidelines
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

---

📚 Documentation preview 📚: https://cala--133.org.readthedocs.build/en/133/

[raymondwjang]

    Gotta be able to do at least ONE OF splitoff or gradualon.

    Negative residuals just need to go. There isn't much you can do with the value...?

    Two cases: (A & B Overlapping)
    1. GradualOn: Know A. B turns ON
        -> trace tries to chase (increases)
        -> footprint tries to chase
        -> residual becomes negative at A-B
            -> should just decrease, positive at A^B
            -> actually... should decrease (just more steeply)

    2. SplitOff: Know AB. B turns OFF
        -> trace tries to chase (decreases)
        -> footprint tries to chase
        -> residual becomes positive at A-B
            -> should increase, negative at A^B
            -> this should just decrease, MORE negative at B-A
            -> this going to zero makes sense
        OR
        keep B, remove A-B

        R = Y - A @ C

    What about the past frame residuals after?

    for GradualOn, nothing should go to zero.
    for SplitOff, a chunk needs to go to zero.

        So... how about we do something like (if it's been on for a long time,
        we become less likely to purge it?)

We subsequently clip R minimum to zero, since all significant negative residual spots
have been removed, and the remaining negative spots are noise level.