Feat/fdb adapter#70
Closed
mathleur wants to merge 68 commits intofeature/rustfrom
Closed
Conversation
* remove mars list file * try to separate floats from general coords * separate float from coords * clean up * add all_unique_dim_coords method and tests
…at/cads_rust_adaptor
… into feat/cads_rust_adaptor
There was a problem hiding this comment.
Pull request overview
Adds new “meteo” adapters for building/serializing Qube structures from external list/constraint formats, while also refactoring core merge/union behavior and adding a new arena-style JSON serialization format.
Changes:
- Add FDB and MARS list adapters plus a JSON “constraints” export adapter in
qubed-meteo. - Refactor core merge API (
union→append/append_many) and add utilities for copying/leaf-path extraction +Qube→Datacubeconversion. - Add arena-style JSON serialization/deserialization for
Qube, and reorganize float coordinates into their own module.
Reviewed changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 18 comments.
Show a summary per file
| File | Description |
|---|---|
| qubed/tests/test_union.rs | Updates union tests to use append and adds a second test case. |
| qubed/src/serde/json.rs | Adds arena-style JSON (de)serialization + a roundtrip test. |
| qubed/src/qube.rs | Adds helper APIs (is_empty, node_dim, child dedup in create_child, path/copy helpers) and a new test. |
| qubed/src/merge.rs | Renames/rewrites union into append and adds append_many. |
| qubed/src/datacube.rs | Adds Datacube::coordinates() and Qube::to_datacubes(), wires append_datacube to append. |
| qubed/src/coordinates/mod.rs | Moves float coordinate type into a dedicated module and updates imports. |
| qubed/src/coordinates/floats.rs | New module for float coordinate storage/conversions. |
| qubed/src/compress.rs | Adds a merge_subtrees helper and tweaks doc comments. |
| qubed/Cargo.toml | Adds rayon dependency. |
| qubed-meteo/src/adapters/to_constraints.rs | New adapter to export Qube leaf paths into DSS-like JSON constraints objects. |
| qubed-meteo/src/adapters/mod.rs | Exposes new adapter modules. |
| qubed-meteo/src/adapters/mars_list.rs | New parser to build a Qube from an indented MARS list format. |
| qubed-meteo/src/adapters/fdb.rs | New parser to build a Qube from FDB-like comma-separated path strings. |
| qubed-meteo/src/adapters/dss_constraints.rs | Changes constraints ingestion to build/merge multiple Qubes using append_many. |
| qubed-meteo/examples/read_from_mars_list.rs | New example for timing MARS-list parsing and datacube conversion. |
| qubed-meteo/examples/read_from_dss_constraints.rs | Updates example input + adds timing output. |
| qubed-meteo/Cargo.toml | Adds rsfdb path dependency and a bin target configuration. |
| .gitignore | Ignores *.list files. |
| .github/workflows/ci.yml | Adjusts workflow triggers (adds pull_request_target labeled trigger). |
Comments suppressed due to low confidence (3)
qubed/src/merge.rs:53
internal_set_operationreturnsOption<Vec<NodeIdx>>but never returnsNone, and callers ignore the returned value. Consider simplifying this to return()(or return a meaningful result that’s actually used) to reduce dead/unclear code paths.
/// Performs a set operation between two groups of nodes from two Qubes.
fn internal_set_operation(
&mut self,
other: &mut Qube,
self_ids: &Vec<NodeIdx>,
other_ids: &Vec<NodeIdx>,
) -> Option<Vec<NodeIdx>> {
let mut return_vec = Vec::new();
qubed/src/coordinates/mod.rs:13
tiny_vec::TinyVecis imported in this module but no longer used (after moving float coordinates intofloats.rs). Please remove the unused import to avoid warnings.
use std::hash::Hash;
use floats::FloatCoordinates;
use integers::IntegerCoordinates;
use strings::StringCoordinates;
use tiny_vec::TinyVec;
use crate::utils::tiny_ordered_set::TinyOrderedSet;
.github/workflows/ci.yml:29
- The added
pull_request_targettrigger (on anylabeledevent) will run this workflow on the base repo context, but the currentactions/checkoutconfiguration checks out the default ref (base), so it won’t actually test the PR’s code. If the intent is “run CI after approval”, consider filtering on a specific label and explicitly checking out the PR merge/head ref; if you do check out PR code underpull_request_target, ensure the workflow does not expose secrets to untrusted code.
# Trigger after public PR approved for CI
pull_request_target:
types: [labeled]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| [dependencies] | ||
| qubed = { path = "../qubed" } | ||
| rsfdb = { path = "../../rsfdb" } |
| use crate::qube::Dimension; | ||
| use crate::{NodeIdx, Qube}; | ||
| use std::collections::HashMap; | ||
| use std::time::Instant; |
Comment on lines
104
to
105
| println!("{:#?}", Qube::to_ascii(&qube_a)); | ||
|
|
Comment on lines
1
to
+4
| pub mod dss_constraints; | ||
| pub mod fdb; | ||
| pub mod mars_list; | ||
| pub mod to_constraints; |
| @@ -0,0 +1,101 @@ | |||
| use qubed::{Coordinates, NodeIdx, Qube}; | |||
Comment on lines
+10
to
+31
| let datacubes: &Vec<Value> = | ||
| dss_constraints.as_array().expect("DSS constraints should be a JSON array"); | ||
|
|
||
| let datacubes = dss_constraints.as_array().expect("DSS constraints should be a JSON array"); | ||
| let order = vec![ | ||
| "origin".to_string(), | ||
| "forecast_type".to_string(), | ||
| "hday".to_string(), | ||
| "day".to_string(), | ||
| "hmonth".to_string(), | ||
| "hyear".to_string(), | ||
| "year".to_string(), | ||
| "month".to_string(), | ||
| "time".to_string(), | ||
| "leadtime_hour".to_string(), | ||
| "level_type".to_string(), | ||
| "variable".to_string(), | ||
| ]; | ||
|
|
||
| for datacube in datacubes { | ||
| // Parse the first datacube and initialize the main Qube | ||
| let first_datacube = parse_datacube(&datacubes[0])?; | ||
| let mut qube = Qube::from_datacube(&first_datacube, Some(&order)); | ||
|
|
Comment on lines
+197
to
+201
| pub fn all_unique_dim_coords(&mut self) -> BTreeMap<String, Coordinates> { | ||
| // TODO | ||
| let mut map: BTreeMap<String, Coordinates> = BTreeMap::new(); | ||
|
|
||
| for (_id, node) in self.nodes.iter() { |
Comment on lines
+117
to
+131
| for (dim, source_child_ids) in source_children { | ||
| for source_child_id in source_child_ids { | ||
| let target_children = { | ||
| let target_node = self.node_ref(target_id).unwrap(); | ||
| target_node.children().get(&dim).cloned().unwrap_or_default() | ||
| }; | ||
|
|
||
| let mut merged_children = target_children.clone(); | ||
| merged_children.push(source_child_id); | ||
|
|
||
| let target_node = self.node_mut(target_id).unwrap(); | ||
| target_node.children_mut().insert(dim, merged_children.into()); | ||
| } | ||
| } | ||
|
|
Comment on lines
+48
to
+49
| println!("{:#?}", Qube::to_ascii(&qube_a)); | ||
|
|
|
|
||
| // collect unique coordinates per dimension | ||
| let map = qube.all_unique_dim_coords(); | ||
| // only one dimension key present |
jameshawkes
approved these changes
Mar 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contributor Declaration
By opening this pull request, I affirm the following: