Add FPS benchmark metric to influx by tjb7 · Pull Request #214 · luxonis/modelconverter

tjb7 · 2026-04-02T05:27:31Z

Adds influx db pushing of fps results, is BEST EFFORT, depends on oakctl installed on running system.

tjb7 · 2026-04-02T05:33:35Z

NOTE: it's not really ready for review yet.

Basic idea is just to get FPS metrics into influx so we can watch them across all versions of everything, so nothing slips by

tjb7 · 2026-04-08T11:03:20Z

@@ -0,0 +1,85 @@
+name: HIL FPS Benchmark Influx


@danilo-pejovic not sure if this is needed, would you rather have a seperate workflow or instrument all the tests directly (in rvc4_test.yaml) ... it is my preference to instrument all tests/gather all data, but up to your choice.

Tests that are running on our hil are bom-test.yaml ones, not rvc4_test - those seem internal tests to ml team. I would just add influx thing to bom-tests and not create a new workflow.

We are still creating a new unnecessary workflow. Can we edit this one that is already part of the bom to also pass info to influx?

https://github.com/luxonis/modelconverter/blob/main/.github/workflows/bom-test.yaml

This commend is still unaddressed - lets not create separate workflow for no reason.

I edited bom-test workflow to run this - it will just need to be returned to pip install framework when confirmed to work with experimental hil framework. Same needs to be done for run_hil_tests.sh

tjb7 · 2026-04-09T11:23:31Z

Also, here is a screenshot of the final data in grafana:

And the direct link: http://10.12.139.254:3000/d/ad4vt7g/fps-model-converter-tests-v3?orgId=1&from=2026-04-07T10:26:51.553Z&to=2026-04-07T17:48:13.745Z&timezone=browser&var-test_name=luxonis%2Fyolov8-nano-pose-estimation:coco-512x288&var-test_name=luxonis%2Ffastsam-s:512x288&var-test_name=luxonis%2Fyolov6-nano:r2-coco-512x288&var-depthai_version=3.3.0&var-depthai_version=3.4.0&var-depthai_version=3.5.0&var-camera_os_version=$__all&var-run_id=$__all&var-camera_mxid=$__all&var-testbed_name=$__all&var-min_fps_threshold=

danilo-pejovic · 2026-04-10T08:18:42Z

+# Cache device metadata once for the whole run using oakctl. If oakctl is not
+# available, keep the benchmark runnable and record explicit placeholder values
+# for the missing oakctl-derived metadata.
+oakctl_output=$(oakctl list --format json 2>/dev/null || printf '')


Oakctl is not part of these tests. Why are we adding more moving parts that can fail?

Hil framework has built in functions to account get all necessary info from camera:

danilo@danilo:~/Downloads$ camera -t slo316-4pro -n test oak4_pro info -v +----------+----------+------------+------------+------------+------------+------------+--------------+----------+----------------+-------------------------------------------------+----------+ | name | model | platform | revision | mxid | username | password | hostname | adb | state | os_version | adb_id | +==========+==========+============+============+============+============+============+==============+==========+================+=================================================+==========+ | oak4_pro | oak4_pro | rvc4 | 9 | 1583798758 | root | oelinux123 | 10.12.143.89 | 5e66d9e6 | setup-complete | 1.30.0+955791f9014cfe160419deae2f64980f180f4d9c | 5e66d9e6 | +----------+----------+------------+------------+------------+------------+------------+--------------+----------+----------------+-------------------------------------------------+----------+ danilo@danilo:~/Downloads$ camera -t slo316-4pro -n test oak4_pro info -v -j [{"name": "oak4_pro", "model": "oak4_pro", "platform": "rvc4", "revision": 9, "mxid": "1583798758", "username": "root", "password": "oelinux123", "hostname": "10.12.143.89", "adb": "5e66d9e6", "state": "setup-complete", "os_version": "1.30.0+955791f9014cfe160419deae2f64980f180f4d9c", "adb_id": "5e66d9e6"}]

Alternatively - camera object in python can give all the necessary info in even more readable/maintainable way. That is probably the way we should supply that info.

I will sub-in the json version of the above:

(.hil) hil@slo130-fl:~$ camera -t slo316-4pro -n test oak4_pro info -j
[{"name": "oak4_pro", "model": "oak4_pro", "platform": "rvc4", "revision": 9, "mxid": "1583798758", "hostname": "10.12.143.89", "os_version": "1.27.1+9950fe7353d76fec53a99019be9e1d6130e8f69a", "adb_id": "5e66d9e6"}]

Q: how to determine the camera used if there are multiple cameras?

For now I will just use: camera -t ${HIL_TESTBED} -n test all info -j and take the first:

(.hil) hil@slo238-4s:~$ camera -t ${HIL_TESTBED} -n test all info -j use_ssh 2026-04-13 07:21:36,832 [INFO] ssh command failed with the following output: 2026-04-13 07:21:36,832 [INFO] 2026-04-13 07:21:36,832 [INFO] Error message (stderr): 2026-04-13 07:21:36,832 [INFO] 2026-04-13 07:21:36,832 [INFO] Exit code: 5 [{"name": "oak4_s", "model": "oak4_s", "platform": "rvc4", "revision": 9, "mxid": "2671960531", "hostname": "10.12.140.235", "os_version": "1.27.1+9950fe7353d76fec53a99019be9e1d6130e8f69a", "adb_id": "9f42e1d3"}]

As stated in dms via example provided and in first comment here I would prefer camera metadata to be gathered inside the script:

Alternatively - camera object in python can give all the necessary info in even more readable/maintainable way. That is probably the way we should supply that info.

It would be more readable, consistent and maintainable.

danilo-pejovic · 2026-04-10T08:49:05Z

+    return f'"{escaped}"'
+
+
+def _write_fps_result_to_influx(


Can this be implemented in hil_framework? We can probably hardcode influx variables there and then create simple function like camera/testbed.log_to_influx(arguments) and it will be pretty similar to this implementation and how sanity checks work for hil already. That way it is more maintainable - if we for example change where we host out database, it will be one change instead of changing every repo.

agreed, but let's do it in a follow up ticket, just so I can get this work finished and immediately useful.

I would prefer to do things properly from the start if we know it is something we are going to be building on - it shouldnt take long.

danilo-pejovic · 2026-04-10T09:12:00Z

@@ -0,0 +1,85 @@
+name: HIL FPS Benchmark Influx


Tests that are running on our hil are bom-test.yaml ones, not rvc4_test - those seem internal tests to ml team. I would just add influx thing to bom-tests and not create a new workflow.

danilo-pejovic · 2026-04-10T09:12:33Z

        default="rvc4",
        help="Target platform to benchmark (default: rvc4).",
    )
+    parser.addoption(


Comments above/below adds more info.

Probably should be implemented in more maintainable way to avoid piping info we already have multiple times.

So if I understand correctly, I should just remove this here, and fail hard if the env is not properly configured. I will make the change, please comment again if you mean something else here.

we are passing camera mxid, os versions,... as arguments to the script when there is no need to do that and that info is readily available inside the script via hil framework.

danilo-pejovic · 2026-04-10T09:33:13Z

Also, here is a screenshot of the final data in grafana:

And the direct link: http://10.12.139.254:3000/d/ad4vt7g/fps-model-converter-tests-v3?orgId=1&from=2026-04-07T10:26:51.553Z&to=2026-04-07T17:48:13.745Z&timezone=browser&var-test_name=luxonis%2Fyolov8-nano-pose-estimation:coco-512x288&var-test_name=luxonis%2Ffastsam-s:512x288&var-test_name=luxonis%2Fyolov6-nano:r2-coco-512x288&var-depthai_version=3.3.0&var-depthai_version=3.4.0&var-depthai_version=3.5.0&var-camera_os_version=$__all&var-run_id=$__all&var-camera_mxid=$__all&var-testbed_name=$__all&var-min_fps_threshold=

In the graph max and min values for models dont seem to be consistent? IIRC they are just some expected value with buffer on both sides so it should be always the same for the same model. Are we maybe mixing up models in tables?

Also as long as those min and max values are set constants we probably dont need to track them for every test. It just adds clutter

tjb7 · 2026-04-13T13:01:17Z

Will try to refactor out to the HIL environment

tjb7 · 2026-04-16T08:58:49Z

Refactor complete, ready for another look @danilo-pejovic

danilo-pejovic

https://github.com/luxonis/modelconverter/actions/runs/24509126353/job/71636481076

tests seem to pass. Lets fix last couple of issues and it should be good to go.

danilo-pejovic · 2026-04-16T08:35:14Z

@@ -0,0 +1,87 @@
+# FPS Influx Split Plan


Lets not commit and push task.md

danilo-pejovic · 2026-04-16T08:36:15Z

@@ -0,0 +1,85 @@
+name: HIL FPS Benchmark Influx


This commend is still unaddressed - lets not create separate workflow for no reason.

I edited bom-test workflow to run this - it will just need to be returned to pip install framework when confirmed to work with experimental hil framework. Same needs to be done for run_hil_tests.sh

danilo-pejovic · 2026-04-16T08:46:55Z

    return slug.rsplit("/", 1)[-1]


+def _normalize_tag(value: Any) -> str:


This seems like something we will want to do every time we would want to do everytime even on other performance tests? In that case it belongs in hil_framework and not here

danilo-pejovic · 2026-04-16T10:49:45Z

+  exit 1
+fi
+
+rvc4_camera=$(


from hil_framework.lib_testbed.db_source.InfluxClient import InfluxClient from hil_framework.lib_testbed.config.Config import Config from hil_framework.lib_testbed.utils.Testbed import Testbed from hil_framework.lib_testbed.camera.RVC4.RVC4CameraBase import RVC4CameraBase import os testbed_name=os.getenv("HIL_TESTBED") config=Config(testbed_name) testbed = Testbed(config) camera: RVC4CameraBase = next((cam for cam in testbed.cameras if cam.platform == 'rvc4'), None) print(camera.name) print(camera.platform) print(camera.mxid) print(camera.get_os_version()) # note if this creates issue as it has been affected by current refactor, fixed the issue but could still be bad on older versions of hil framework print(camera.revision) print(camera.model.name) print(camera.hostname)

As stated in:

#214 (comment) (preferred choice, towards the end)

#214 (comment)

https://github.com/luxonis/modelconverter/pull/214/changes#r3072218602

and multiple times in dms with examples

I would heavily prefer to collect data in python as camera object is less likely to be changed in the future compared to cli interface which is generally used for interactive purposes (granted both are unlikely), is more readable, maintainable and less likely to suffer random bugs. I will not block the release over this as it is unlikely to cause issues in short term, but I dont think I have gotten a response - so if we go for cli tool I would at least want it to be a conscious choice. Cli tool was just a quick example from my side before, I will be more clear from now on.

ok will change.

tjb7 requested a review from a team as a code owner April 2, 2026 05:27

tjb7 requested review from conorsim, klemen1999, kozlov721 and tersekmatija and removed request for a team April 2, 2026 05:27

tjb7 marked this pull request as draft April 2, 2026 05:27

tjb7 requested a review from danilo-pejovic April 2, 2026 05:33

tjb7 force-pushed the tjb_fps_graphs branch from e5bc2d2 to 9ae4b83 Compare April 7, 2026 07:40

tjb added 11 commits April 8, 2026 13:00

Add manual FPS benchmark Influx pipeline

cfc995e

Use model-based HIL selection for FPS Influx workflow

ce82fcd

Temporarily repurpose rvc4 workflow for FPS Influx dispatch

f14ff42

Map Influx GitHub variables into FPS workflow env

d0bf4d7

Read Influx workflow settings from GitHub secrets

f65c94d

Add fake Influx CSV fixture for FPS metrics

5fad3b6

Fix annotated CSV format for fake Influx fixture

c95f978

Add fake Influx line protocol fixture

d9308bd

Use correct Influx bucket name in FPS workflows

5e96309

Normalize FPS Influx tags and testbed fallback

e7e23cd

Temporary debugging logging

45768b9

tjb7 force-pushed the tjb_fps_graphs branch from 69f7149 to 45768b9 Compare April 8, 2026 11:00

tjb7 commented Apr 8, 2026

View reviewed changes

Cleanup Commit

22fd18c

tjb7 marked this pull request as ready for review April 8, 2026 11:10

kozlov721 reviewed Apr 9, 2026

View reviewed changes

Comment thread .github/workflows/rvc4_test.yaml

danilo-pejovic reviewed Apr 10, 2026

View reviewed changes

Comment thread tests/test_benchmark/run_hil_tests.sh Outdated

tjb added 7 commits April 13, 2026 07:24

Use HIL camera metadata in benchmark tests

1579062

Drop obsolete camera agent metadata

be2ccf9

Use HIL_TESTBED directly in benchmark script

63fdb79

Fail benchmark run on missing camera metadata

a962d3b

Use env metadata in benchmark tests

60a4f54

Select rvc4 camera metadata

ba2e946

Restore RVC4 workflow from main

c24c3ec

tjb added 4 commits April 14, 2026 15:17

Use shared HIL Influx writer for FPS benchmark

782600c

Pass explicit influx benchmark inputs

ca2df04

Use fixed benchmark bucket and env token

05a2cba

Move benchmark payload shaping into modelconverter

f3550f2

Danilo added 5 commits April 16, 2026 13:31

Bootleg test to check if it works with new branch

78b707a

bootleg to test on bom tests

73e2f88

python3 instead of python

3edc090

add token

d22fc9e

remove hil frameowrk before

c6f6076

danilo-pejovic reviewed Apr 16, 2026

View reviewed changes

tjb added 9 commits April 17, 2026 10:44

Pass raw benchmark tags to InfluxClient

191fc1f

Read benchmark metadata from HIL camera object

02f343f

Wire BOM test os_version input into env

34d97fa

Construct benchmark HIL testbed from Config

e4858c7

Remove uneeded workflow

d693c63

Removal of uneeded file

b3cb113

Remove benchmark device_ip pytest plumbing

a496eb7

Remove unused runner hostname logging

c739d8c

Restore packaged hil-framework install in BOM workflow

19f07b6

		return slug.rsplit("/", 1)[-1]


		def _normalize_tag(value: Any) -> str:

Conversation

tjb7 commented Apr 2, 2026

Uh oh!

tjb7 commented Apr 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tjb7 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danilo-pejovic Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danilo-pejovic Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

danilo-pejovic commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tjb7 commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tjb7 commented Apr 16, 2026

Uh oh!

danilo-pejovic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjb7 Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

tjb7 commented Apr 9, 2026 •

edited

Loading

danilo-pejovic Apr 10, 2026 •

edited

Loading

danilo-pejovic Apr 13, 2026 •

edited

Loading

danilo-pejovic commented Apr 10, 2026 •

edited

Loading

tjb7 commented Apr 13, 2026 •

edited

Loading

tjb7 Apr 17, 2026 •

edited

Loading