Skip to content

Add NWB current reporting#124

Merged
pgetta merged 73 commits intomainfrom
add-current-report
Mar 13, 2026
Merged

Add NWB current reporting#124
pgetta merged 73 commits intomainfrom
add-current-report

Conversation

@ilkilic
Copy link
Copy Markdown
Contributor

@ilkilic ilkilic commented Mar 10, 2026

Add support for exporting current report variables to a separate current_report.nwb.
Current densities recorded from the simulation are converted to currents using the segment area of the recorded compartment and stored as VoltageClampSeries.

If a SEClamp stimulus is present in the simulation configuration, the command waveform is reconstructed and written to the NWB stimulus group.

The simulation script was also refactored to rely on the BlueCellulab report helper pipeline instead of manually configuring recordings. This depends on the corresponding BlueCellulab API changes (report payload updates including segment area and time) and should be merged after that PR.

g-bar and others added 30 commits May 21, 2025 09:31
* generate typed dict

* generate entitycore schemas

* entitycore

* build_model

* fetching assets

* download morphologies, hoc_files, mechanisms

* pyright

* single_neuron_simulation

* entitycore routes

* bimodal

* Typings

* Migrating Nexus helper to entitycore helper

* fetch morphology

* fetch mechanisms

* Retrieve morphology entitycore

* download

* FORMAT PRIORITY

* direct current plot

* simulation

* remove dendrogram for now

* clean up 1

* add vlab info

* add vlab, project id

* run_simulation service

* model_factory

* simulation routes use prject contexts

* lint

* check token

* Headers

* morphology, plot

* clean up

* naming

* naming

* feedback

* typo

* types

* typing

* Typging

* Types and linting

* authors

* authors

* format

* comment

* response models

* type error

* format

* updated schemas

* cureate schemas

* remove trailing /

* add sinaptome

* Keep codegenerator for reference

* ass

* traling slash

* migrate synaptome to entitycore

---------

Co-authored-by: Gil Arturo Barrios del Villar <gilarturo.barriosdelvillar@openbraininstitute.org>
Co-authored-by: g-bar <gbarv2@gmail.com>
…rcuit simulation

* Add current changes

* Make single cell sim distributed

* Add circuit route and sim code draft

* Add current changes

* Add current progress

* Futher refactoring

* Futher refactoring

* Add simulation code, docker setup

* Fix container tag

* Provide ENVIRONMENT var to docker build

* Fix missing env vars

* Add current changes

* Convert to async, refactor

* Remove unused deps

* Make route handlers async

* Other changes

* Remove ARM platfor build from api

* Some improvements for the docker setup

* Ensure user group is created

* Use arm64 for api

* Entitycore integration: model download

* Auth and other fixes

* Register Simulation execution and results

* Provide custom libnrnmech path

* Fix output dir, add comments, enable NWB save

* Have a better name for ready marker file

* Refctor saving the results

* Rename  artefacts, cleanu p

* Check the the status

* Add current changes

* Try using qemu and buildx

* Try to build arm image in a different way

* Temporarily isable Qemu and buildx

* Put back normal build from makefile

* Set platform in docker compose to amd64

* Add missing IMAGE_NAME

* Add test script to run MPI in a container

* Add missing module init files

* Debug issue with storage

* Clean up app storage debug code

* Add was_generated_by field to generated NWBs

* Process on_failure and on_success from the api service

* Add sending the pending status to the redis stream

* Upgrade bluecellulab and entitysdk

* Use 8 workers to download circuit

* Update simulation script to correctly gather sim reports with MPI

* Use time rate for NWB instead of supplying time series data

* Make sure sim exec status is getting set to error if simulation fails

* Provide input spike files with absolute path

* Add production build (for now from the current branch)

* Update labels

* Fix image tag in prod, disable staging build for now

* Fix img tag suffix env resolution

* Disable prod deployment, enable staging deployment

* Dynamicly determine number of MPI processes

* Disable adding projections to allow BCL set up local synapses

* Build production image

* Add more logs

* Build in staging

* Call pc.quit() after simulation ended

* Provide v_init, init spikes dicts

* Remove hippocampus projections

* Add NWB compression

* Send queue position in extra prop

* Build for current tag

* Disable build from the current branch

* Refactor job_position fn into job monitoring fn to run callbacks

* Remove nexus, part 1

* Remove nexus, part 2

* Remove nexus, part 3

* Refactor Github actions

* Remove sample circuit

* Fix all checks

* Fix typo

* Pass exec id as string

* Clean up

* Better error handling and clean up

* Rename single_cell to single_neuron to be consistent

* Another round of clean up

* Migrate most of single cell functions

* Fix synapse generation

* Remove redundant endpoints, clean up

* validation and small fixes

* Final touches

* Add comment about glibc version conflict

* Split single neuron calibration and validation tasks

* Add accounting for small circuit simulations

* Integrate accounting for circuit simulations, make validation a single job

* Correct exc_type types

* Fix sync/async call in on_failed handler

* Clean up, add job fn reference check

* Clean up #2

* Upgrade deps

* Update pull request GH action

* Fix types for asset label and content type

* Clean up some entitycore types

* Fix some other types

* Provide default accounting base URL

* Add makefile entry to start job monitor

* Refactor circuit simulations, add timestamp to JobMessage

* Fix simulation constructor args

* Upgrade BlueCelluLab to use validation multiprocessing from released version

* Move logic between job - service, renaem send_one to send_once
* Better logs for a missing hoc file

* Make sure single neuron models have the same structure

* Bump ready marker version (to invalidate current cache)
* Small init model improvements, more logs and clean up

* Remove arch MD doc
* Add WIP

* Add endpoint to fetch output files

* Enable authentication

* Clean up, organize imports

* Add Ultralizer repo clone, test build

* Clean up

* Ignore missing dep for now

* Temporarily disable job-fn-reference check

* Finalize Github actions
* update entitysdk version

* uv lock
* Make sure only external entitySDK client is used

* Datamodel code generator dependency
* Add current recordings

* Use combination of sec and seg to index current recordings
@pgetta
Copy link
Copy Markdown
Collaborator

pgetta commented Mar 12, 2026

Relates to Bluenaas implementation of IC Simulator.

pgetta
pgetta previously approved these changes Mar 12, 2026
Comment thread app/core/circuit/simulation-mpi-entrypoint.py Outdated
Comment thread app/core/circuit/simulation-mpi-entrypoint.py
@ilkilic
Copy link
Copy Markdown
Contributor Author

ilkilic commented Mar 12, 2026

@pgetta is there a reason that the python version is locked at 3.12.13 now? Because I was unable to run uv lock with my python version. I can install it if this necessary.

@AurelienJaquier
Copy link
Copy Markdown
Contributor

AurelienJaquier commented Mar 12, 2026

EDIT: we would need this obi-one version to be used: https://github.com/openbraininstitute/obi-one/releases/tag/2026.3.6
and this version of BlueCelluLab: https://github.com/openbraininstitute/BlueCelluLab/releases/tag/2.6.87
could you guys please update it in this PR?

darshanmandge
darshanmandge previously approved these changes Mar 13, 2026
Copy link
Copy Markdown
Contributor

@darshanmandge darshanmandge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @ilkilic and @pgetta !

@pgetta
Copy link
Copy Markdown
Collaborator

pgetta commented Mar 13, 2026

@pgetta is there a reason that the python version is locked at 3.12.13 now? Because I was unable to run uv lock with my python version. I can install it if this necessary.

In general I like versions fixed, had too many issues with things suddenly not working in the past. I've updated it to the latest patch.

I remember trying to use 3.13 - but some packages failed, maybe the situation is different now, will try to upgrade at some point.

With UV, if I'm not mistaken, you can run uv sync and it will install required python version if you currently installed ones don't satisfy the requirements.

@darshanmandge
Copy link
Copy Markdown
Contributor

New release made for obi-one: https://github.com/openbraininstitute/obi-one/releases/tag/2026.3.7

Copy link
Copy Markdown
Contributor

@AurelienJaquier AurelienJaquier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks guys!

@pgetta pgetta merged commit 3bb85aa into main Mar 13, 2026
1 check passed
@pgetta pgetta deleted the add-current-report branch March 13, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants