fix(core): enable fault-tolerant execution and incremental result persistence

[Rishabh-git10]

What type of PR is this?
/kind bug

What this PR does / why we need it:
Enables fault-tolerant execution and incremental data persistence to prevent data loss during multi-configuration benchmarks.

Previously, a downstream configuration failure caused a fatal RuntimeError, terminating the execution loop and erasing all prior successful results from memory.

Changes:

1. Exception Isolation: Modified run_testcases in testcasecontroller.py to catch, log, and isolate individual test exceptions, allowing the loop to continue safely.
2. Incremental Persistence: Introduced an incremental_save_cb callback in benchmarkingjob.py connected to the Rank module's concatenation logic. Successful results are now written to disk immediately after each test completes.

Which issue(s) this PR fixes:
Fixes #423

[kubeedge-bot]

Welcome @Rishabh-git10! It looks like this is your first PR to kubeedge/ianvs 🎉

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Rishabh-git10
To complete the pull request process, please assign jaypume after the PR has been reviewed.
You can assign the PR to them by writing /assign @jaypume in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gemini-code-assist]

Code Review

This pull request introduces incremental saving of test results and enhances fault tolerance by allowing the benchmark to continue after individual test case failures. Feedback highlights a significant performance bottleneck and potential data loss issue caused by frequent I/O operations in the saving logic. Additionally, it is recommended to wrap the incremental save callback in a try-except block to ensure that persistence failures do not interrupt the overall execution.