feat: Add WSL support, fix broken API calls (generate-chat, summarize, faq, etc.) and JSON parsing fix.#16
Open
Akira-AA83 wants to merge 94 commits intotmc:mainfrom
Open
feat: Add WSL support, fix broken API calls (generate-chat, summarize, faq, etc.) and JSON parsing fix.#16Akira-AA83 wants to merge 94 commits intotmc:mainfrom
Akira-AA83 wants to merge 94 commits intotmc:mainfrom
Conversation
…e README Improvements: - Add chunked response handling in internal/batchexecute - Update README with latest features and improvements - Add client tests and testdata
- Add simple integration test for 'list' command - Test skips when running in CI or without proper auth
- Add global '-mime' flag to specify content type explicitly - Update file and stdin handlers to use the specified MIME type when provided - Update README with examples and document new functionality
- Fix malformed comment that was causing a build error
- Treat application/json MIME type as text to improve JSON file handling - Fixes errors when adding JSON files to notebooks
This fixes multiple issues:
1. Added content detection for JSON files by checking for { or [ at the start
2. Enhanced chunked response parser to better handle direct JSON data
3. Improved error handling with more detailed debug output
4. Added special handling for error responses
These changes restore compatibility with previous versions and fix issues
with adding JSON files to notebooks.
This change addresses the specific error pattern when handling JSON files: 1. Added robust handling for improperly formatted responses 2. Added manual extraction of wrb.fr responses from chunked data 3. Added support for handling numeric responses 4. Fixed unescaping issues with quoted JSON strings These improvements should resolve the 'cannot unmarshal number into Go value' error and handle the 'Failed to unescape chunk: invalid syntax' error seen with some JSON files.
- Add explicit JSON file detection based on file extension - Add better debug output for JSON file handling - Improve network error messages with more helpful text - Add authentication status check at startup - Fix debug flag handling in the API client These changes ensure JSON files are properly treated as text sources and provide better diagnostics when things go wrong.
- Add more robust JSON parsing in chunked response handler - Update test validation with explicit checks - Relax error handling for partial responses - Fix test expectations for edge cases
- Add ChunkedResponseParser with multiple fallback strategies - Support parsing complex nested JSON responses - Handle various response formats from NotebookLM API - Add project extraction with robust error handling This parser handles the complex response format used by the ListRecentlyViewedProjects endpoint with multiple parsing strategies to ensure reliability.
- Add Makefile with build, test, and clean targets - Add beproto command-line tool for debugging API calls - Update .gitignore to exclude build artifacts - Add development tools and documentation This improves the development workflow with standard build targets and debugging tools.
…ation - Add comprehensive CLI flag parsing for auth command - Add profile scanning and validation functionality - Improve browser profile detection and selection - Add notebook count checking for profile validation - Enhanced error messages and user guidance These improvements make authentication more reliable by allowing users to scan all profiles and validate they contain NotebookLM data.
- Add comprehensive HTTP testing with response validation - Enhance JSON parsing with better error messages - Add more robust handling of API response formats - Improve source addition and project listing reliability These changes make the API client more reliable when handling various response formats and error conditions.
- Add httprr package for recording and replaying HTTP requests - Add API client tests using request recording - Support multiple recording modes (record/replay/passthrough) - Add test server for serving recorded responses - Add request sanitization to protect credentials This improves API testing by allowing tests to run without live credentials and ensures consistent test behavior.
- Improve code formatting and consistency - Better variable naming and organization - Enhanced error handling patterns
- Add integration tests for main package functionality - Add unit tests for command validation and flag parsing - Add script-based CLI behavior tests - Add test data files for various CLI scenarios - Update .gitignore for test artifacts These tests ensure reliable CLI behavior across authentication, command validation, flag parsing and input handling scenarios.
- Fix trailing whitespace across all packages - Standardize line endings and blank lines - Align struct fields and function parameters - Remove redundant empty lines - Improve overall code readability These changes ensure consistent code formatting across the codebase without modifying any functional behavior.
- Refactor httprr package for better API and reliability - Add NotebookLM-specific request/response scrubbing - Add comprehensive test coverage for all functionality - Add support for compressed recordings - Add request matching for NLM RPC endpoints - Improve credential and timestamp sanitization - Add convenience functions for NLM testing - Add detailed package documentation These changes improve the HTTP recording functionality with specific support for testing NotebookLM API interactions while maintaining backwards compatibility.
- Enhance test structure with better setup and cleanup - Improve recording client initialization patterns - Update regex patterns for more reliable ID scrubbing - Add comprehensive source and project test coverage - Refine error handling and validation These changes improve the reliability and maintainability of NotebookLM API testing by enhancing the HTTP recording functionality and test coverage.
- Add getBrowserPathForProfile() function to return correct browser executable - Update authentication flows to use appropriate browser executable - Fix session conflicts by using temporary directories with profile copying - Remove excessive automation flags to reduce Google detection - Support Chrome, Chrome Canary, Brave Browser, Chromium, and Edge
- Add debug output to show response prefix and line processing - Add debug output to show chunk counts and contents - Help diagnose API response parsing issues
- Create comprehensive_record_test.go with tests for all nlm commands - Add tests for notebook operations (list, create, delete) - Add tests for source operations (add, list, delete, rename) - Add tests for note operations (create, update, delete) - Add tests for audio operations (create, get, delete, share) - Add tests for generation operations (guide, outline, section) - Create record_all_tests.sh script for automated recording - Include proper cleanup mechanisms for all tests - Support both single command and bulk recording modes
- Update README.md with Brave browser authentication instructions - Add multi-browser support documentation (Chrome, Brave, Edge, etc.) - Create TROUBLESHOOTING.md with comprehensive issue resolution guide - Document authentication flow, profile detection, and common errors - Add troubleshooting for network issues, API parsing errors, and setup problems - Include debug mode instructions and prevention tips
…ects response parsing Add support for beprotojson debug options in main command, enabling debug parsing and field mapping when flags are set. Fix ListRecentlyViewedProjects response handling by wrapping the direct array response in the expected message format for proper unmarshaling.
- Remove complex httprr dependencies from tests in favor of simple credential checking - Tests now skip gracefully when credentials unavailable instead of causing "unexpected EOF" errors - Fix beprotojson to handle numbers and booleans in arrays for string fields - Add credential scrubbing to httprr NLM configuration - Simplify test structure: tests work with credentials, skip without them - All tests now pass reliably in both development and CI environments
…astructure - Add SOURCES column to `nlm ls` output showing source count for each notebook - Improve title formatting with emoji width compensation using backspace character - Adjust title truncation limit to 45 characters to accommodate new column - Add comprehensive test coverage for list output format validation - Migrate tests from credential-based skipping to httprr-based HTTP recording/replay - Replace direct environment variable usage with httprr credential scrubbing - Ensure tests work with both real credentials and recorded responses - All tests now handle authentication gracefully without "unexpected EOF" errors The list command now provides more useful information at a glance by showing how many sources each notebook contains, while the test infrastructure is more robust and reliable across different environments.
Streamline integration_test.go by removing redundant code and improving test structure. Reduces code complexity while maintaining test coverage.
Add comprehensive_test.go test runner that executes scripttest files in parallel for improved test coverage and performance. Include six test suites covering authentication, content generation, notebook management, notes management, source management, and simplified scenarios. The test runner automatically discovers comprehensive_*.txt files and runs them concurrently with proper timeout handling and environment setup, providing extensive integration test coverage for all nlm command functionality.
Add -keep-open / -k flag to auth command allowing users to specify seconds to keep browser open for manual authentication when automated login fails or times out. Key improvements: - Add KeepOpenSeconds field to AuthOptions and BrowserAuth structs - Implement keep-open logic in authentication timeout and redirect scenarios - Add anti-detection JavaScript to hide automation traces from browser - Enhance profile copying with recursive directory support for complete session preservation - Support original profile directory usage via NLM_USE_ORIGINAL_PROFILE env var - Add stealth flags to Chrome automation to avoid detection This enables users to manually complete authentication in challenging scenarios while maintaining the automated flow for typical cases.
Add gracefulShutdown function that closes browser tabs using JavaScript and sets localStorage flag before terminating the browser process. This prevents browsers from detecting crashes during automated authentication flows. Key improvements: - Close browser windows gracefully using JavaScript window.close() - Set normal_shutdown flag in localStorage to indicate intentional closure - Add brief delay before context cancellation to allow processing - Integrate graceful shutdown into successful authentication flow This reduces browser crash warnings and improves the user experience during automated authentication.
Remove complex retry and fallback logic from keep-open authentication flow while preserving the initial delay functionality. This simplifies the authentication process by: - Remove retry logic after authentication timeout - Remove manual authentication fallback after initial redirect detection - Keep the upfront delay when keep-open is specified to allow manual login - Streamline error handling by removing duplicate authentication attempts The authentication flow now provides a single opportunity for manual intervention at the start rather than multiple retry points, reducing complexity while maintaining the core keep-open functionality.
…tion files Replace recursive profile directory copying with selective copying of essential authentication files. This reduces data transfer and improves performance while maintaining authentication functionality. Key improvements: - Copy only essential files: cookies, login data, web data, and preferences - Skip non-existent files gracefully with warning messages - Reduce profile copy overhead by avoiding unnecessary files - Maintain debug output showing count of successfully copied files - Preserve all authentication-related data needed for session restoration This optimization significantly reduces the amount of data copied during profile setup while ensuring all necessary authentication components are preserved.
Add Go build constraints to restrict API integration tests to run only when the integration build tag is specified. This separates integration tests from regular unit tests for better test organization and CI/CD control. Key changes: - Add `//go:build integration` constraint to client_record_test.go - Add `//go:build integration` constraint to comprehensive_record_test.go - Include legacy `// +build integration` syntax for backward compatibility - Enable selective test execution with `go test -tags=integration` This allows developers to run fast unit tests by default while requiring explicit integration tag for comprehensive API testing against live services.
- Fix ActOnSources encoder with correct payload nesting (summarize, faq, mindmap, etc.) - Fix GenerateFreeFormStreamed to use gRPC endpoint instead of batchexecute - Add dynamic extraction of bl and f.sid parameters from NotebookLM page - Remove hardcoded API parameters - now fetched automatically - Add fallback to env vars (NLM_BUILD_VERSION, NLM_SESSION_ID) and defaults - Add NotebookID to RPC calls for proper source-path routing Fixes generate-chat, summarize, faq, and other commands that were returning "API error 3 (Unavailable)" or "400 Bad Request".
Akira-AA83
changed the title
- Add RPC o4cbdc for binary source registration - Implement Google resumable upload protocol (3-step process) - Support PDF, DOCX, images, and other binary formats - Extract SOURCE_ID from o4cbdc response - Initialize upload session and get upload URL - Upload file content with proper headers
working cli
tmc
force-pushed
the
main
branch
3 times, most recently
from
03eacf6 to
8238f17
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds WSL (Windows Subsystem for Linux) support and fixes several broken API calls that were
returning "API error 3 (Unavailable)" or "400 Bad Request".
Changes
WSL Support
NLM_USE_ORIGINAL_PROFILE=1environment variable to use Windows Chrome profile from WSLchrome_linux.godocs/WSL.mdsetup guideAPI Fixes
Fix ActOnSources encoder (
summarize,faq,mindmap,explain,study-guide, etc.)NotebookIDto RPC calls for propersource-pathroutingFix GenerateFreeFormStreamed (
generate-chat)/google.internal.labs.tailwind.orchestration.v1.LabsTailwindOrchestrationService/GenerateFree FormStreamedDynamic API Parameters
blandf.sidparametersGetAPIParams()function that extracts parameters dynamically from NotebookLM page HTMLNLM_BUILD_VERSION,NLM_SESSION_ID) → page extraction → defaultsJSON Parsing Fix
C:\Users\Desktop)sanitizeJSON()function to handle invalid escape sequences in server responsesTesting
All commands tested and working:
nlm list✅nlm sources✅nlm generate-chat✅ (was broken)nlm summarize✅ (was broken)nlm faq✅ (was broken)nlm auth✅ (with WSL)Related Issues
Fixes #14 (API error 3 - Service unavailable)
Fixes #7 (invalid character '\n' in string escape code)
May fix #5 (JSON parsing improvements)
Breaking Changes
None - all changes are backwards compatible.