Skip to content

Add a universal CSV/JSON loader (Load::Universal) #7

Description

@MoonFlowww

Motivation

Nott's stated goal is to be easier than raw LibTorch. Today, users with any custom tabular dataset (CSV or JSON) must implement their own parsing, tensor construction, and train/test splitting — the same boilerplate PyTorch users write every time. The infrastructure for this is already partially in place (Data::Type::CSV, CSVParameters, column selection), but no loader actually reads a file and produces tensors.

A universal loader removes the single biggest friction point for new users trying Nott on their own data.

Current State

src/data/load/types.hpp defines:

struct CSVParameters {
    bool has_header = true;
    char delimiter = ',';
    std::vector<std::string> columns; // empty = all columns
};
struct CSV { std::string file; CSVParameters parameters{}; };

src/data/load/load.hpp has CSV parsing helpers (split_csv_line, trim_copy) but no public loader function that returns tensors. Users have no supported path to use these types.

Proposed Change

Add Load::Universal in src/data/load/load.hpp:

namespace Nott::Data::Load {

struct UniversalDescriptor {
    std::variant<Data::Type::CSV /*, Data::Type::JSON */> source;
    std::vector<std::string> input_columns;  // empty = all except target_column
    std::string              target_column;
    Data::Type::GlobalParameters split{};    // train/test fraction + shuffle
};

struct UniversalDataset {
    torch::Tensor train_inputs;
    torch::Tensor train_targets;
    torch::Tensor test_inputs;
    torch::Tensor test_targets;
};

UniversalDataset Universal(const UniversalDescriptor& descriptor);

} // namespace Nott::Data::Load

Implementation reuses the existing split_csv_line / trim_copy helpers and GlobalParameters splitting logic already used by MNIST/CIFAR loaders. JSON support can follow in a subsequent PR using the same descriptor pattern.

Usage target:

auto dataset = Nott::Data::Load::Universal({
    .source         = Nott::Data::Type::CSV{"my_data.csv"},
    .input_columns  = {"open", "high", "low", "volume"},
    .target_column  = "label",
    .split          = {.train_fraction = 0.8f}
});
model.train(dataset.train_inputs, dataset.train_targets, ...);

Acceptance Criteria

  • Load::Universal accepts a Data::Type::CSV descriptor and returns UniversalDataset
  • input_columns empty → all columns except target_column used as inputs
  • Train/test split and shuffle respect GlobalParameters
  • Missing or misnamed columns produce a clear std::invalid_argument
  • Consistent return type with existing MNIST/CIFAR loaders
  • No dependency added beyond what load.hpp already includes

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions