Skip to content

Fix: GetNotes nil pointer crash due to NotebookLM API change#17

Open
Anderson-RC wants to merge 88 commits intotmc:mainfrom
Anderson-RC:schema-refactor
Open

Fix: GetNotes nil pointer crash due to NotebookLM API change#17
Anderson-RC wants to merge 88 commits intotmc:mainfrom
Anderson-RC:schema-refactor

Conversation

@Anderson-RC
Copy link

Fix: GetNotes API Response Parsing

Summary

The nlm notes command crashes with a nil pointer dereference when listing notes from a NotebookLM notebook. This PR fixes the crash and restores full functionality by implementing correct parsing of the API response, including extraction of note content.

This issue seems to have been caused by the (undocumented) NotebookLM API changing the schema of the data returned by this function. The default unmarshaling couldn't parse the response, and so the nil pointer caused a full program crash. This fix adds a schema to the data (instead of inline assignment) to place the notes content in.

The note's content was also being discarded in the response. See other PR for possible extension to make use of it.

Issue

When running nlm notes <notebook-id>, the command panics:

panic: runtime error: invalid memory address or nil pointer dereference
goroutine 1 [running]:
main.listNotes(...)
    cmd/nlm/main.go:1032

Root Cause

The NotebookLM API returns note data in a custom nested array format that differs from the expected protobuf structure. The beprotojson unmarshaller was unable to correctly parse this format, resulting in:

  1. note.GetMetadata() returning nil
  2. note.Title being empty (populated from wrong field position)
  3. note.Content not being extracted at all

API Response Structure

The API returns notes in this nested array structure:

[
  [                                    <- notes array (position 0)
    [                                  <- NoteEntry
      "note-uuid",                     <- position 0: source_id
      [                                <- position 1: NoteDetails
        "note-uuid",                   <- details[0]: id (duplicate)
        "content text...",             <- details[1]: content
        [type, "id", [sec, nanos]],    <- details[2]: NoteTimestampMetadata
        null,                          <- details[3]: reserved
        "Note Title"                   <- details[4]: title
      ]
    ],
    ...
  ],
  [additional-metadata]                <- ignored
]

The protobuf Source message expects Title at field 2, but the API places it at position 4 of the nested details array. Content is at position 1.

Changes

1. gen/notebooklm/v1alpha1/notes_response.pb.go - API Structure Documentation

Created new file documenting the expected API message types with proper position-to-field mappings:

  • NoteEntry - Outer array [source_id, details]
  • NoteDetails - Inner array [id, content, metadata, null, title]
  • NoteTimestampMetadata - Metadata [type, id, timestamp]
  • TimestampPair - Timestamp [seconds, nanos]

This serves as documentation for the API structure and could be extended for beprotojson compatibility in the future.

2. gen/notebooklm/v1alpha1/notebooklm.pb.go - Content field

Added Content field to the Source struct to store note body text:

type Source struct {
    // ... existing fields ...
    Content  string `protobuf:"bytes,6,opt,name=content,proto3" json:"content,omitempty"`
}

3. gen/service/LabsTailwindOrchestrationService_client.go - Refactored parser

Refactored GetNotes with:

  1. Detailed inline documentation of the API response structure
  2. Extracted parseNoteEntry helper function for cleaner, maintainable code
  3. Content extraction from position 1 of the details array
  4. Fallback to beprotojson.Unmarshal if API format changes
// parseNoteEntry parses a single note entry from the API response.
// This matches the structure documented in notes_response.pb.go.
func parseNoteEntry(entry interface{}) *notebooklmv1alpha1.Source {
    noteArr, ok := entry.([]interface{})
    if !ok || len(noteArr) < 2 {
        return nil
    }

    note := &notebooklmv1alpha1.Source{}

    // Position 0: Source ID
    if sourceID, ok := noteArr[0].(string); ok {
        note.SourceId = &notebooklmv1alpha1.SourceId{SourceId: sourceID}
    }

    // Position 1: Details array [id, content, metadata, null, title]
    detailsArr, ok := noteArr[1].([]interface{})
    if !ok || len(detailsArr) < 5 {
        return note
    }

    // details[1]: Content
    if content, ok := detailsArr[1].(string); ok {
        note.Content = content
    }

    // details[4]: Title
    if title, ok := detailsArr[4].(string); ok {
        note.Title = title
    }

    // details[2]: Metadata with timestamp
    // ... (see full implementation)

    return note
}

4. cmd/nlm/main.go - Nil guards in listNotes

Added defensive nil checks when accessing note metadata to prevent crashes:

lastModified := "unknown"
if meta := note.GetMetadata(); meta != nil && meta.LastModifiedTime != nil {
    lastModified = meta.LastModifiedTime.AsTime().Format(time.RFC3339)
}

Also improved output formatting:

  • Extract proper UUID from SourceId wrapper
  • Show "(untitled)" for notes without titles
  • Added "No notes found" message for empty notebooks

Testing

Before Fix

$ nlm notes <notebook-id>
panic: runtime error: invalid memory address or nil pointer dereference

After Fix

$ nlm notes <notebook-id>
ID                                   TITLE                    LAST MODIFIED
abc12345-6789-abcd-ef01-234567890abc Research Notes           2025-12-01T10:00:00Z
def67890-abcd-1234-5678-90abcdef1234 Literature Review        2025-12-02T14:30:00Z
ghi11111-2222-3333-4444-555566667777 Project Outline          2025-12-03T09:15:00Z
jkl22222-3333-4444-5555-666677778888 (untitled)               unknown

Notes are correctly displayed with titles from the right API position, timestamps parsed correctly, and draft notes show "(untitled)" with "unknown" timestamp.

Files Changed

File Change
gen/notebooklm/v1alpha1/notes_response.pb.go NEW - API structure documentation
gen/notebooklm/v1alpha1/notebooklm.pb.go Added Content field to Source
gen/service/LabsTailwindOrchestrationService_client.go Refactored with parseNoteEntry helper, content extraction
cmd/nlm/main.go Nil guards in listNotes

Notes

  • The fix includes a fallback to the standard beprotojson.Unmarshal in case the API format changes
  • Draft content shows as "unknown" with "unknown timestamp, at least on my own Notebook.
  • Note content is now extracted and stored in Source.Content for downstream use (see proceeding PR)

Related

  • This appears to be caused by a NotebookLM API update
  • No official API documentation exists (that I could find) for the consumer NotebookLM version; this fix was developed by observing actual API responses. Let me know if this documentation actually exists because I couldn't find it anywhere.

tmc added 30 commits December 19, 2024 13:24
…e README

Improvements:
- Add chunked response handling in internal/batchexecute
- Update README with latest features and improvements
- Add client tests and testdata
- Add simple integration test for 'list' command
- Test skips when running in CI or without proper auth
- Add global '-mime' flag to specify content type explicitly
- Update file and stdin handlers to use the specified MIME type when provided
- Update README with examples and document new functionality
- Fix malformed comment that was causing a build error
- Treat application/json MIME type as text to improve JSON file handling
- Fixes errors when adding JSON files to notebooks
This fixes multiple issues:
1. Added content detection for JSON files by checking for { or [ at the start
2. Enhanced chunked response parser to better handle direct JSON data
3. Improved error handling with more detailed debug output
4. Added special handling for error responses

These changes restore compatibility with previous versions and fix issues
with adding JSON files to notebooks.
This change addresses the specific error pattern when handling JSON files:
1. Added robust handling for improperly formatted responses
2. Added manual extraction of wrb.fr responses from chunked data
3. Added support for handling numeric responses
4. Fixed unescaping issues with quoted JSON strings

These improvements should resolve the 'cannot unmarshal number into Go value' error
and handle the 'Failed to unescape chunk: invalid syntax' error seen with some JSON files.
- Add explicit JSON file detection based on file extension
- Add better debug output for JSON file handling
- Improve network error messages with more helpful text
- Add authentication status check at startup
- Fix debug flag handling in the API client

These changes ensure JSON files are properly treated as text sources and
provide better diagnostics when things go wrong.
- Add more robust JSON parsing in chunked response handler
- Update test validation with explicit checks
- Relax error handling for partial responses
- Fix test expectations for edge cases
- Add ChunkedResponseParser with multiple fallback strategies
- Support parsing complex nested JSON responses
- Handle various response formats from NotebookLM API
- Add project extraction with robust error handling

This parser handles the complex response format used by the
ListRecentlyViewedProjects endpoint with multiple parsing
strategies to ensure reliability.
- Add Makefile with build, test, and clean targets
- Add beproto command-line tool for debugging API calls
- Update .gitignore to exclude build artifacts
- Add development tools and documentation

This improves the development workflow with standard build
targets and debugging tools.
…ation

- Add comprehensive CLI flag parsing for auth command
- Add profile scanning and validation functionality
- Improve browser profile detection and selection
- Add notebook count checking for profile validation
- Enhanced error messages and user guidance

These improvements make authentication more reliable by
allowing users to scan all profiles and validate they
contain NotebookLM data.
- Add comprehensive HTTP testing with response validation
- Enhance JSON parsing with better error messages
- Add more robust handling of API response formats
- Improve source addition and project listing reliability

These changes make the API client more reliable when
handling various response formats and error conditions.
- Add httprr package for recording and replaying HTTP requests
- Add API client tests using request recording
- Support multiple recording modes (record/replay/passthrough)
- Add test server for serving recorded responses
- Add request sanitization to protect credentials

This improves API testing by allowing tests to run without
live credentials and ensures consistent test behavior.
- Improve code formatting and consistency
- Better variable naming and organization
- Enhanced error handling patterns
- Add integration tests for main package functionality
- Add unit tests for command validation and flag parsing
- Add script-based CLI behavior tests
- Add test data files for various CLI scenarios
- Update .gitignore for test artifacts

These tests ensure reliable CLI behavior across authentication,
command validation, flag parsing and input handling scenarios.
- Fix trailing whitespace across all packages
- Standardize line endings and blank lines
- Align struct fields and function parameters
- Remove redundant empty lines
- Improve overall code readability

These changes ensure consistent code formatting across the codebase
without modifying any functional behavior.
- Refactor httprr package for better API and reliability
- Add NotebookLM-specific request/response scrubbing
- Add comprehensive test coverage for all functionality
- Add support for compressed recordings
- Add request matching for NLM RPC endpoints
- Improve credential and timestamp sanitization
- Add convenience functions for NLM testing
- Add detailed package documentation

These changes improve the HTTP recording functionality with
specific support for testing NotebookLM API interactions
while maintaining backwards compatibility.
- Enhance test structure with better setup and cleanup
- Improve recording client initialization patterns
- Update regex patterns for more reliable ID scrubbing
- Add comprehensive source and project test coverage
- Refine error handling and validation

These changes improve the reliability and maintainability of
NotebookLM API testing by enhancing the HTTP recording
functionality and test coverage.
- Add getBrowserPathForProfile() function to return correct browser executable
- Update authentication flows to use appropriate browser executable
- Fix session conflicts by using temporary directories with profile copying
- Remove excessive automation flags to reduce Google detection
- Support Chrome, Chrome Canary, Brave Browser, Chromium, and Edge
- Add debug output to show response prefix and line processing
- Add debug output to show chunk counts and contents
- Help diagnose API response parsing issues
- Create comprehensive_record_test.go with tests for all nlm commands
- Add tests for notebook operations (list, create, delete)
- Add tests for source operations (add, list, delete, rename)
- Add tests for note operations (create, update, delete)
- Add tests for audio operations (create, get, delete, share)
- Add tests for generation operations (guide, outline, section)
- Create record_all_tests.sh script for automated recording
- Include proper cleanup mechanisms for all tests
- Support both single command and bulk recording modes
- Update README.md with Brave browser authentication instructions
- Add multi-browser support documentation (Chrome, Brave, Edge, etc.)
- Create TROUBLESHOOTING.md with comprehensive issue resolution guide
- Document authentication flow, profile detection, and common errors
- Add troubleshooting for network issues, API parsing errors, and setup problems
- Include debug mode instructions and prevention tips
tmc and others added 26 commits September 16, 2025 01:07
Replace repetitive if-else logic in template with generalized approach.
Add comprehensive argbuilder package with pattern-based argument encoding.
Update all 50+ encoder methods to use unified EncodeRPCArgs function.
Reduce generated code by ~30% while maintaining full compatibility.
Add interactive chat command with terminal UI supporting multiline mode, clear screen, and help commands. Integrate with GenerateFreeFormStreamed API for real-time response streaming.

Add 15 content transformation commands: rephrase, expand, summarize, critique, brainstorm, verify, explain, outline, study-guide, faq, briefing-doc, mindmap, timeline, toc, and generate-chat for free-form chat generation.

Connect interactive chat to NotebookLM API using GenerateFreeFormStreamed RPC with streaming response handling and fallback display for non-streaming responses.

Add comprehensive test coverage with content_transformation_commands.txt test suite validating all commands for argument validation, authentication requirements, special characters, multiple source IDs, and debug flags.

All commands properly validate arguments, require authentication, and integrate with existing NotebookLM API infrastructure.
…ects response parsing

Add support for beprotojson debug options in main command, enabling debug parsing and field mapping when flags are set. Fix ListRecentlyViewedProjects response handling by wrapping the direct array response in the expected message format for proper unmarshaling.
- Remove complex httprr dependencies from tests in favor of simple credential checking
- Tests now skip gracefully when credentials unavailable instead of causing "unexpected EOF" errors
- Fix beprotojson to handle numbers and booleans in arrays for string fields
- Add credential scrubbing to httprr NLM configuration
- Simplify test structure: tests work with credentials, skip without them
- All tests now pass reliably in both development and CI environments
…astructure

- Add SOURCES column to `nlm ls` output showing source count for each notebook
- Improve title formatting with emoji width compensation using backspace character
- Adjust title truncation limit to 45 characters to accommodate new column
- Add comprehensive test coverage for list output format validation
- Migrate tests from credential-based skipping to httprr-based HTTP recording/replay
- Replace direct environment variable usage with httprr credential scrubbing
- Ensure tests work with both real credentials and recorded responses
- All tests now handle authentication gracefully without "unexpected EOF" errors

The list command now provides more useful information at a glance by showing how many sources each notebook contains, while the test infrastructure is more robust and reliable across different environments.
Streamline integration_test.go by removing redundant code and improving test structure. Reduces code complexity while maintaining test coverage.
Add comprehensive_test.go test runner that executes scripttest files in parallel for improved test coverage and performance. Include six test suites covering authentication, content generation, notebook management, notes management, source management, and simplified scenarios.

The test runner automatically discovers comprehensive_*.txt files and runs them concurrently with proper timeout handling and environment setup, providing extensive integration test coverage for all nlm command functionality.
Add -keep-open / -k flag to auth command allowing users to specify seconds to keep browser open for manual authentication when automated login fails or times out.

Key improvements:
- Add KeepOpenSeconds field to AuthOptions and BrowserAuth structs
- Implement keep-open logic in authentication timeout and redirect scenarios
- Add anti-detection JavaScript to hide automation traces from browser
- Enhance profile copying with recursive directory support for complete session preservation
- Support original profile directory usage via NLM_USE_ORIGINAL_PROFILE env var
- Add stealth flags to Chrome automation to avoid detection

This enables users to manually complete authentication in challenging scenarios while maintaining the automated flow for typical cases.
Add gracefulShutdown function that closes browser tabs using JavaScript and sets localStorage flag before terminating the browser process. This prevents browsers from detecting crashes during automated authentication flows.

Key improvements:
- Close browser windows gracefully using JavaScript window.close()
- Set normal_shutdown flag in localStorage to indicate intentional closure
- Add brief delay before context cancellation to allow processing
- Integrate graceful shutdown into successful authentication flow

This reduces browser crash warnings and improves the user experience during automated authentication.
Remove complex retry and fallback logic from keep-open authentication flow while preserving the initial delay functionality. This simplifies the authentication process by:

- Remove retry logic after authentication timeout
- Remove manual authentication fallback after initial redirect detection
- Keep the upfront delay when keep-open is specified to allow manual login
- Streamline error handling by removing duplicate authentication attempts

The authentication flow now provides a single opportunity for manual intervention at the start rather than multiple retry points, reducing complexity while maintaining the core keep-open functionality.
…tion files

Replace recursive profile directory copying with selective copying of essential authentication files. This reduces data transfer and improves performance while maintaining authentication functionality.

Key improvements:
- Copy only essential files: cookies, login data, web data, and preferences
- Skip non-existent files gracefully with warning messages
- Reduce profile copy overhead by avoiding unnecessary files
- Maintain debug output showing count of successfully copied files
- Preserve all authentication-related data needed for session restoration

This optimization significantly reduces the amount of data copied during profile setup while ensuring all necessary authentication components are preserved.
Add Go build constraints to restrict API integration tests to run only when the integration build tag is specified. This separates integration tests from regular unit tests for better test organization and CI/CD control.

Key changes:
- Add `//go:build integration` constraint to client_record_test.go
- Add `//go:build integration` constraint to comprehensive_record_test.go
- Include legacy `// +build integration` syntax for backward compatibility
- Enable selective test execution with `go test -tags=integration`

This allows developers to run fast unit tests by default while requiring explicit integration tag for comprehensive API testing against live services.
…cks to improve `listNotes` command robustness.
- Create notes_response.pb.go documenting expected API message types
- Extract parseNoteEntry helper function for cleaner code
- Add Content field to Source struct for note body text
- Add detailed API structure comments in GetNotes client
- Keep fallback to beprotojson for API format changes
- Update FIX_GETNOTES_PARSING.md with implementation details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants