Hey @jonthegeek 👋 - Congrats on taking over tibblify, I'm excited to see where you take it!

I've been using tibblify extensively in production across multiple projects integrating with large, complex, nuanced, and dynamic APIs (usually also undocumented or partially/incorrectly documented). I've been leveraging a tibblify-driven system for converting deeply nested JSON API responses into tidy tibbles and wanted to share some observations, pain points, and patterns I've developed that might be useful for the package's roadmap.

Current Pain Points & Workarounds

1. `fill` Must Match `ptype_inner`, Not `ptype`

This is undocumented and tripped me up repeatedly. When using tib_scalar() with transform, the fill value must match the input type (ptype_inner), not the output type (ptype), because fill gets applied before transform:

# WRONG - fill is numeric but ptype_inner is character
tib_scalar("price", 
  ptype = double(), 
  ptype_inner = character(),
  fill = 0.0,  # will fail
  transform = as.numeric
)

# CORRECT - fill matches ptype_inner
tib_scalar("price", 
  ptype = double(), 
  ptype_inner = character(),
  fill = "0",  # works - matches ptype_inner
  transform = as.numeric
)

This becomes particularly confusing when building custom collectors for domain-specific types like durations. If I want a vctrs::new_duration() output from a "100ms" input string, the fill must still be a character string like "0", not a duration object—which is unintuitive.

Suggestion: Document this clearly, or consider applying fill after transform (matching ptype), which would be more intuitive.

2. Type Coercion & The `transform` Function Limitation

Real-world APIs often return the same field as different types depending on context (e.g., IDs as "12345" vs 12345, currency as "$250,000" vs 250000, dates as "2023-01-01" vs ISO 8601 strings). The current architecture requires you to know ptype_inner ahead of time—but the transform function cannot perform different operations based on the incoming type because tibblify has already committed to ptype_inner by that point.

What I want is a single collector that says "this should be an ID represented as a number" and handles both scenarios:

String input "12345" → parse to numeric
Numeric input 12345 → use as-is

Currently, I have to create separate wrapper collectors and know ahead of time which one to use—which defeats the purpose when the API is inconsistent:

# I want a single collector that handles both cases, but instead I need:
tib_chr_dbl <- function(key, ...) {
  tib_scalar(key, ptype = double(), ptype_inner = character(), transform = as.numeric, ...)
}

tib_dbl <- function(key, ...) {
  tibblify::tib_dbl(key, ...)
}

# And must choose the right one based on expected input type

The workaround of wrapping collectors is just putting a name to a pattern—it still requires the API to abide by the expected ptype_inner. In practice, the same logical field (e.g., a date or monetary value) might come as a string in one endpoint and a number in another, requiring different collector configurations for what is conceptually the same field.

Suggestion: Allow transform to receive the raw value before ptype_inner coercion, or provide a mechanism for ptype_inner to accept multiple types (similar to JSON Schema's type: ["string", "number", "null"]). This would allow a single tib_id() collector that checks if the input is string vs numeric and coerces accordingly, rather than depending on the API to always be consistent.

3. Dynamic Response Structures & Variant Resolution

API responses are rarely consistent. The same endpoint may return different structures based on:

Request parameters: ids_only=TRUE returns just IDs, summary=TRUE adds aggregations
Data model inconsistencies: Some records have nested entities, others don't
Missing data in source systems: Nested arrays/collections may or may not exist depending on the underlying data
Field availability: Sub-entities may have different fields populated across records

This variability applies not just at the top level, but within nested entities too. A user object might have an address sub-object in one response but not another. A transactions array might exist for some records and be completely absent for others.

tib_variant() currently just parses to a list without type enforcement—essentially a fallback for "I don't know what this is." But many APIs have predictable variants where the correct structure is determinable at runtime. This is conceptually what tspec_recursive() implies, but there's no built-in mechanism for runtime resolution.

What I Built: A "variant resolver" pattern where tib_variant acts as a placeholder in the base spec, then gets replaced with the appropriate typed collector during tidying based on inspecting the actual response:

# define component specs for different response modes
spec_full_data <- tspec_df(
  id = tib_int("id"),
  name = tib_chr("name"),
  # ... all fields for full records
)

spec_ids_only <- tspec_df(
  id = tib_int("id")  # minimal: just IDs
)

spec_summary <- tspec_row(
  total_count = tib_int("totalCount"),
  avg_value = tib_dbl("avgValue")
)

# base response spec with variant placeholders
response_spec <- tspec_object(
  status_code = tib_int("statusCode"),
  data = tib_variant("data"),      # structure varies by request mode
  summary = tib_variant("summary") # may or may not be present
)

# resolver function: inspects response and returns the correct spec
resolve_variants <- function(base_spec, resp) {
  resolved <- base_spec
  
  # resolve 'data' field based on response mode
  resolved$fields$data <- if (is_ids_only(resp)) {
    tib_df("data", spec_ids_only$fields)
  } else if (is_count_only(resp)) {
    tib_unspecified("data")  # just ignore
  } else {
    tib_df("data", spec_full_data$fields)
  }
  
  # resolve 'summary' based on presence
  resolved$fields$summary <- if (has_summary(resp)) {
    tib_row("summary", spec_summary$fields)
  } else {
    tib_unspecified("summary")
  }
  
  resolved
}

# usage: resolve variants before tibblifying
final_spec <- resolve_variants(response_spec, resp)
result <- tibblify(resp_body, spec = final_spec)

This pattern works because tspec objects are just lists—you can manipulate $fields directly. But it's manual and undocumented.

Suggestion: A hook/callback mechanism for tib_variant() to enable runtime resolution, or first-class support for conditional specs similar to JSON Schema's oneOf/anyOf. The current tspec_recursive() hints at this need but doesn't provide dynamic resolution.

4. Schema-Like Composition with `tspec_combine()` and Nested Specs

Complex APIs have deeply nested data models where the same sub-structures appear across multiple endpoints. Think of how OpenAPI/JSON Schema uses $ref to reference reusable components/schemas—you define Address, Coordinates, Owner once and reference them wherever needed.

Currently, tibblify doesn't have a clean way to mirror this pattern. I want to define component specs that match the source system's nested structure, then compose them without field extraction and !!! unsplicing:

# IDEAL: define reusable component specs that mirror the API's nested structure
spec_address <- tspec_row(
  .key = "address",  # matches JSON nesting
  street = tib_chr("street"),
  city = tib_chr("city"),
  state = tib_chr("state"),
  zip = tib_chr("zip")
)

spec_coordinates <- tspec_row(
  .key = "coordinates",
  latitude = tib_dbl("lat"),
  longitude = tib_dbl("lng")
)

spec_owner <- tspec_row(
  .key = "owner",
  name = tib_chr("name"),
  email = tib_chr("email"),
  address = spec_address  # nested component reference
)

# compose into endpoint-specific specs by nesting, not flattening
spec_entity_detail <- tspec_object(
  id = tib_int("id"),
  name = tib_chr("name"),
  owner = spec_owner,           # nested as-is
  location = spec_coordinates,  # nested as-is
  # ... endpoint-specific fields
)

# REALITY: must extract fields and manually combine
spec_entity_detail <- tspec_object(
  id = tib_int("id"),
  name = tib_chr("name"),
  !!!spec_owner$fields,         # loses nesting, requires extraction
  !!!spec_coordinates$fields,   # loses nesting, requires extraction
)

The current tspec_combine() helps merge specs at the same level, but doesn't preserve hierarchical nesting. I end up either:

Flattening everything with JSON-path vectors like c("owner", "address", "street")
Manually extracting $fields and unsplicing with !!!
Duplicating field definitions across specs

Suggestion: Support for nested spec composition that mirrors the source structure—something like tspec_ref() or allowing tspec_row()/tspec_df() as field values that get resolved during parsing. This would make tibblify specs work more like JSON Schema with $ref references.

5. Nested Entities: Unpacking vs. Path-Vector Keys

Tibblify offers two approaches for nested data:

Path-vector keys: Flatten at parse time using c("owner", "name")
Nested tib_row()/tib_df(): Parse as nested tibbles, optionally unpack() later

Each has tradeoffs that aren't well documented:

# approach 1: flatten with path vectors (loses structure)
tspec_df(
  id = tib_int("id"),
  owner_name = tib_chr(c("owner", "name")),      # flattened key
  owner_email = tib_chr(c("owner", "email")),    # must prefix to avoid collisions
  address_city = tib_chr(c("address", "city"))
)

# approach 2: preserve nesting with tib_row (keeps structure)
tspec_df(
  id = tib_int("id"),
  owner = tib_row(
    "owner",
    name = tib_chr("name"),
    email = tib_chr("email")
  ),
  address = tib_row(
    "address",
    city = tib_chr("city"),
    state = tib_chr("state")
  )
)
# then optionally: tidyr::unpack(result, c(owner, address))

Path vectors are simpler but:

Lose the original nesting structure
Require manual prefixing to avoid column name collisions
Make it harder to extract related fields as a unit
Don't map cleanly back to the source schema for round-tripping

Nested tib_row/tib_df preserves structure but:

Creates tibble columns that some downstream code doesn't expect
Requires explicit unpack() calls if you want flat output
The relationship between unpack options and tib_row() isn't clear

For database normalization workflows, I need to extract nested entities into separate tables with proper foreign keys. Nested tib_df() is ideal for this—but the documentation doesn't cover these patterns.

Suggestion: A vignette on when to use path-vector keys vs. nested specs, the unpack workflow, and extracting nested tibbles for relational database use cases.

6. Dynamic Field Filtering Based on Response Structure

Building on section 3: API responses often don't include all possible fields. Missing/NULL fields may be completely omitted from the JSON rather than sent as null. Tibblify fails if the spec expects a field that doesn't exist in the response.

This is related to variant resolution—not just top-level fields vary, but nested entities may have different fields present across records. The spec needs to adapt to what's actually in the data.

What I Built: A function that filters spec fields based on what's actually present in the response, keeping the spec as the source of truth for types while adapting to response variability:

# filter spec fields based on actual response structure
filter_spec_to_response <- function(spec, resp_data) {
  # get fields actually present in response
  available_keys <- if (is.null(names(resp_data))) {
    # array: union of all keys across items
    resp_data |> purrr::map(names) |> purrr::list_c() |> unique()
  } else {
    names(resp_data)
  }
  
  # filter spec fields to only those present (plus required fields)
  spec$fields <- purrr::keep(spec$fields, function(field) {
    if (isTRUE(field$required)) return(TRUE)  # always keep required
    if (inherits(field, "tib_variant")) return(TRUE)  # resolve separately
    
    key <- field$key
    if (is.null(key)) return(TRUE)
    top_level <- if (length(key) > 1) key[[1]] else key
    top_level %in% available_keys
  })
  
  spec
}

This could integrate with variant resolution—filter, then resolve variants, then tibblify.

Suggestion: Built-in support for optional fields that gracefully handle absence, or a strict = FALSE mode that only parses fields present in the data.

7. Empty Values Cause `NULL` Class Assignment

APIs return empty data in various forms (NULL, [], ""), and tibblify's handling causes completely invalid issues and errors—not just "unexpected behavior." When tibblify encounters these empties, it literally assigns a class of "NULL" to values, which breaks downstream operations entirely.

The root cause ties back to ptype_inner detection—tibblify sees the empty value, determines ptype_inner as "NULL", and never even reaches the transform function. This is a critical point that's difficult for humans to understand (and seemingly impossible for AI to understand from my experience): there's no point in putting length/empty checks inside transform because tibblify will detect the ptype_inner as "NULL" and never get to the transform anyway.

# This is a TRAP - transform NEVER runs when value is NULL/empty
# tibblify detects ptype_inner as "NULL" before ever calling transform
transform = function(x) {
  if (is.null(x) || length(x) == 0L) return(NA_real_)  # Never reached!
  as.numeric(x)
}

This becomes especially problematic when dealing with actual JSON serialized into R via jsonlite or httr2. The "boxing" of empty values as [] (empty arrays) in JSON gets parsed into R as empty lists, which tibblify then detects as "NULL" class for ptype_inner, causing errors. This is a fundamental mismatch between how JSON represents empties and how tibblify handles them.

Workaround: I pre-process responses with a normalize_empties() function that recursively converts empty lists/pairlists to properly-typed NA values before tibblifying. The key insight is that this function takes the tspec as an argument to assign the correct NA type based on each field's ptype—this is what tibblify should do natively:

# type-aware normalization using tspec (what tibblify should do)
normalize_empties <- function(data, tspec) {
  for (field_name in names(tspec$fields)) {
    field <- tspec$fields[[field_name]]
    if (is_empty_value(data[[field$key]])) {
      # assign correct NA type based on the field's expected ptype
      data[[field$key]] <- get_na_for_ptype(field$ptype)
    }
  }
  data
}

is_empty_value <- function(x) {
  is.null(x) || 
  (is.list(x) && length(x) == 0L) ||  # empty list from jsonlite []
  identical(x, list()) ||
  (is.character(x) && length(x) == 1L && x == "")
}

Suggestion: Tibblify should handle empty values consistently using fill or ptype—don't assign "NULL" class; invoke transform even for empties, or at minimum respect the fill value.

8. No Way to Generate Empty Prototypes from `tspec`

When iterating over paginated API responses using httr2::req_perform_iterative(), if one page fails, you need an empty tibble with the correct column types to avoid breaking vctrs::vec_rbind() or dplyr::bind_rows(). Currently there's no way to extract the expected ptype from a tspec.

This is critical for production workflows where you're pulling thousands of records across hundreds of paginated requests—a single failed page shouldn't break the entire pipeline.

What I Built:

tspec_to_ptype <- function(tspec, sort = TRUE) {
  stopifnot(inherits(tspec, "tspec"))
  
  if (tspec$type == "object") {
    # handle object specs with nested data fields
    scalar_fields <- tspec$fields |>
      purrr::discard(\(f) inherits(f, c("tib_df", "tib_row", "tib_variant"))) |>
      names()
    scalar_ptypes <- purrr::map(scalar_fields, \(f) tspec$fields[[f]]$ptype) |>
      stats::setNames(scalar_fields)
    # ... handle nested tib_df/tib_row fields separately
    return(utils::modifyList(rlang::exec(list, !!!scalar_ptypes), data_ptypes))
  }
  
  # for tspec_df/tspec_row
  cols <- purrr::map(tspec$fields, \(field) {
    if (field$type == "variant") return(list())
    field$ptype
  })
  if (sort) cols <- cols[sort(names(cols))]
  vctrs::vec_ptype(rlang::exec(tibble::tibble, !!!cols))
}

Suggestion: A tspec_ptype() or similar function to generate empty prototype tibbles from specs.

9. Custom Geometry/Spatial Types

For GIS workflows, transforming WKT strings or GeoJSON into sf geometries is common. Building custom tib_* collectors with sf's sfc vectors as ptype works but requires careful handling of empty geometries:

tib_sfc_point <- function(key, required = FALSE) {
  tib_scalar(
    key = key, required = required,
    fill = "POINT EMPTY",  # must be valid WKT string (matches ptype_inner)
    ptype_inner = character(),
    ptype = sf::st_sfc(sf::st_point(), crs = 4326),
    transform = function(x) {
      tryCatch(
        sf::st_as_sfc(x, crs = 4326),
        error = function(e) sf::st_sfc(sf::st_point(), crs = 4326)
      )
    }
  )
}

The key insight is that fill = "POINT EMPTY" must be a valid WKT string (matching ptype_inner = character()), not an sf geometry. This ties back to the fill/ptype_inner relationship discussed above.

Suggestion: A vignette on creating custom collectors for non-base types (sf, lubridate, units, etc.)

10. Untibblify & Round-Tripping

I've experimented with attaching the tspec as an attribute to the result and implementing untibblify() for round-tripping. This is useful for:

Caching tidied results but needing to reconstruct the original structure for re-submission
Testing/validation with waldo::compare()
Serializing back to JSON for API calls that accept the same structure

Suggestion: First-class support for spec preservation and round-trip untidying.

11. OpenAPI / JSON Schema → `tspec` Generation

Many APIs provide OpenAPI specs or JSON schemas. I've been working on automated tspec generation from these schemas—mapping JSON Schema types to tibblify collectors, handling $ref references, oneOf/anyOf to variants, etc.

The challenge is that real-world API specs are often incomplete or incorrect. I've found that hand-curated field mappings based on observed API behavior are more reliable than generated specs, but having a starting point from the schema is valuable.

Suggestion: This could be a companion package or built-in feature for APIs with published schemas and follow along with your work with rapid and nectar.

12. Extracting Nested 1-to-Many for Database Normalization

Many API responses contain nested arrays (e.g., orders with line items, users with roles, entities with history records) that represent 1-to-many relationships. Tibblify handles these with tib_df(), producing nested tibbles—but extracting them into proper relational tables for database persistence requires additional processing.

Pattern I've landed on:

Use tib_df() for 1-to-many nested arrays (not path vectors)
Main spec produces tibble-columns for nested entities
Separate extraction functions unnest each nested tibble with proper foreign keys

# spec preserves nested structure
spec <- tspec_object(
  id = tib_int("id"),
  transactions = tib_df(
    "transactions",
    tx_id = tib_int("txId"),
    amount = tib_dbl("amount"),
    date = tib_chr("date")
  )
)

# after tibblify: result$transactions is a nested tibble
# extract with foreign key for database insert
extract_nested <- function(result, nested_field, parent_id_field) {
  children <- result[[nested_field]]
  if (is.null(children) || nrow(children) == 0) {
    return(empty_child_ptype())
  }
  children[[parent_id_field]] <- result[[parent_id_field]]
  children
}

transactions_table <- extract_nested(result, "transactions", "id")

Suggestion: Documentation or vignettes on handling complex nested structures for database normalization workflows, including the relationship between tib_df(), unpack(), and extraction patterns.

Overarching Vision

The pattern I've landed on treats tibblify specs as runtime-resolved schemas rather than static definitions:

Base specs with variant placeholders — Define the overall structure with tib_variant() for polymorphic/conditional fields
Variant resolvers — Functions that inspect the response and replace variants with the appropriate typed spec
Response-adaptive field filtering — Dynamically filter specs to only include fields present in the actual response
Type-aware empty normalization — Pre-process responses using the tspec to convert empties to properly-typed NA values (what tibblify should do natively)
Safe wrapper with ptype fallback — Try tibblify, catch errors, return correctly-typed empty ptype on failure
Spec preservation — Attach spec to result for untidying/round-tripping
Component-based architecture — Reusable specs mirroring the source system's nested data model (like OpenAPI components/schemas with $ref)

This makes tibblify powerful for complex, real-world APIs where response shapes vary by endpoint, query parameters, or even individual records.

Enhancement Ideas Summary

Category	Suggestion
Type System	Allow `transform` to receive raw values before `ptype_inner` coercion, or support multiple `ptype_inner` types
Empty Handling	Handle `NULL`/`[]`/`""` consistently using `fill` or `ptype`—don't assign `"NULL"` class; invoke `transform` even for empties
Variant Resolution	Hook/callback for dynamic `tib_variant()` resolution; support for `oneOf`/`anyOf` like JSON Schema
Spec Composition	Allow nested spec references (like JSON Schema `$ref`); cleaner `tspec_combine()` that preserves hierarchy
Field Handling	Built-in support for optional fields; `strict = FALSE` mode that only parses present fields
Utilities	`tspec_ptype()` to generate empty prototypes from specs
Documentation	`fill`/`ptype_inner` relationship, path vectors vs nested specs, `unpack()` workflow, database normalization patterns
Round-tripping	First-class `untibblify()` and spec preservation
Schema Integration	`tspec_from_json_schema()` for automated spec generation

Happy to discuss any of these in more detail! I've got working implementations of all the above patterns if useful for reference.

Bump {tibblify} package dependency to latest stable release #92

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions