Feature: Added notebook SQL round-trip conversion and paired SQL validation

[Zerraf-Badreddine]

Summary

This PR adds a Jupytext-style conversion workflow between Oracle SQL Notebook files (.isqlnb) and SQL files (.sql), plus validation and quick-fix support for notebook-formatted SQL.
The goal is to make notebook content easier to review, diff, edit with LLMs, and safely round-trip back into .isqlnb.

User-visible changes

• Added .isqlnb -> .sql conversion using a canonical paired SQL format.
• Added .sql -> .isqlnb conversion for:
  • canonical paired SQL
  • plain SQL fallback (single SQL cell)
• Preserved notebook metadata, including connectionAlias, across round-trip conversion.
• Updated Configure Connection so reconfiguring from an open notebook also updates that notebook's connectionAlias metadata, preventing converted notebooks from staying pinned to an old alias such as dev.
• Added command entry points in:
  • command palette
  • notebook toolbar
  • editor title
  • explorer context menu
• Added overwrite confirmation before replacing an existing target file.
• Added success notifications with an Open action for converted output.

SQL format (paired notebook SQL)

The SQL representation uses a Jupytext-like format:

-- oracle-sql-notebook: {"schemaVersion":1,"metadata":{"connectionAlias":"dev"}}
-- %% [sql] {"metadata":{}}
select * from dual;

-- %% [markdown] {"metadata":{}}
-- # Notes
-- Markdown cells are stored as SQL comments.

Serialization/deserialization behavior:

• notebook metadata is preserved in the header comment JSON
• cell metadata is preserved in per-cell marker JSON
• SQL cells remain executable SQL blocks
• markdown cells are emitted as SQL comment lines
• outputs/execution state/secrets are not emitted to SQL

Note on connection metadata:

• the paired SQL header may include notebook metadata such as connectionAlias
• when that SQL is converted back into .isqlnb, the notebook keeps that alias by design
• re-running Configure Connection on an open notebook now updates the notebook metadata too, so users can switch away from a hard-coded alias without manually editing the file

If a .sql file has no notebook header and no -- %% markers, it is treated as plain SQL and converted into a one-cell notebook.

Validation and conversion safety

Editor-time validation now runs for .sql files that look like paired notebook SQL.

Validation behavior:

• ordinary SQL files remain silent (no paired-format diagnostics)
• paired-format issues are surfaced in diagnostics
• deterministic issues include quick fixes
• conversion is blocked only for structurally unsafe cases

Blocking issues:

• malformed notebook header JSON
• malformed cell marker JSON
• unsupported schema version

Non-blocking issues (conversion can continue with explicit confirmation):

• missing notebook header
• stray text outside cells
• markdown lines missing -- prefix
• non-canonical [markup] marker (normalized to [markdown])
• unsupported non-SQL cell language marker
• metadata normalization cases

When only non-blocking issues exist, users can choose Convert Anyway.

Quick fixes included

• add missing notebook header
• replace malformed header JSON with canonical header
• normalize invalid header metadata payload
• replace malformed marker JSON with canonical metadata payload
• normalize invalid marker metadata payload
• prefix markdown lines with --
• normalize [markup] -> [markdown]

Implementation details

Primary files:

• src/conversion/sqlNotebookText.ts
• src/conversion/commands.ts
• src/conversion/validation.ts
• src/extension.ts
• package.json

The parser/converter was refactored into a shared analyzer model used by conversion, diagnostics, and quick fixes to keep behavior consistent.

Test coverage and verification

Added/expanded tests in:

• test/unit/sqlNotebookText.test.ts

Covered scenarios include:

• notebook -> SQL serialization
• valid paired SQL parsing
• plain SQL fallback
• connection alias preserved in notebook metadata during round-trip conversion
• missing header
• malformed header/marker JSON
• unsupported schema version
• stray non-cell text
• markdown comment-prefix enforcement
• non-canonical marker normalization
• unsupported marker languages
• quick-fix application helpers
• round-trip preservation

Local verification:

npm run check
npm run test
npm run build

Results:

• lint/typecheck passed
• tests passed (8 files, 41 tests)
• production build passed

Docs and developer workflow updates

• Updated README.md with:
  • round-trip SQL format
  • conversion command usage
  • validation behavior
• Added .vscode/launch.json for isolated Extension Development Host debugging.
• Added .vscode-devhost/ to .gitignore.
• Added converter.md as branch/session handoff notes.

Reviewer notes

• package-lock.json changed due to local dependency installation and version metadata synchronization (0.0.3 -> 0.0.4) to match package.json.

[Copilot]

Pull request overview

Adds a round-trip conversion workflow between Oracle SQL Notebook files (.isqlnb) and a canonical “paired notebook SQL” format (.sql), plus editor-time validation and quick fixes to keep paired SQL structurally safe to convert back into notebooks.

Changes:

• Introduces paired-format parsing/analysis, serialization, quick-fix application, and blocking/non-blocking issue classification.
• Adds VS Code commands and UI contributions to convert .isqlnb ⇄ .sql, with overwrite confirmation and post-conversion “Open” actions.
• Adds a SQL validation/code-action provider that surfaces paired-format diagnostics and quick fixes for .sql files.

Reviewed changes

Copilot reviewed 18 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/unit/sqlNotebookText.test.ts	Adds unit coverage for paired-format parsing/serialization, warnings, blocking issues, quick fixes, and round-trip behavior.
src/extension.ts	Wires up validation provider + registers conversion commands and document listeners.
src/conversion/validation.ts	Implements diagnostics + quick-fix code actions for paired-format SQL documents.
src/conversion/sqlNotebookText.ts	Core analyzer/parser/serializer and quick-fix range application for paired notebook SQL.
src/conversion/commands.ts	Adds command implementations for notebook→SQL and SQL→notebook conversions with prompts and notifications.
samples/converter-tests/README.md	Documents manual test cases for validation/conversion behavior in dev host.
samples/converter-tests/01_valid_paired.sql	Sample “valid paired SQL” fixture for manual testing.
samples/converter-tests/02_missing_header.sql	Sample fixture for missing-header warning + quick fix.
samples/converter-tests/03_bad_header_json_blocking.sql	Sample fixture for blocking malformed header JSON.
samples/converter-tests/04_unsupported_schema_version_blocking.sql	Sample fixture for blocking unsupported schema version.
samples/converter-tests/05_bad_marker_json_blocking.sql	Sample fixture for blocking malformed marker JSON.
samples/converter-tests/06_stray_text_outside_cells.sql	Sample fixture for stray-text warning behavior.
samples/converter-tests/07_markdown_line_needs_comment_prefix.sql	Sample fixture for markdown comment-prefix warning + quick fix.
samples/converter-tests/08_non_canonical_markup_marker.sql	Sample fixture for [markup] normalization warning + quick fix.
samples/converter-tests/09_unsupported_cell_language.sql	Sample fixture for unsupported cell language warning behavior.
samples/converter-tests/10_missing_cell_markers.sql	Sample fixture for “looks paired but missing markers” warning behavior.
package.json	Registers activation events, commands, and menu contributions for conversion entry points.
package-lock.json	Syncs lockfile version metadata (0.0.3 → 0.0.4).
.vscode/launch.json	Adds an Extension Development Host launch configuration using an isolated devhost folder.
.gitignore	Ignores .vscode-devhost/ artifacts and normalizes .env.* ignore pattern.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Abdelillah-CHADLI]

Great addition! Thank you

[Zerraf-Badreddine]

Fixed in the latest push based on the copilot review.

Changes included:

• debounced paired SQL validation on text changes
• tightened quick-fix diagnostic matching to the exact issue range
• fixed indented -- %% marker parsing consistency
• updated Configure Connection to also refresh notebook connectionAlias metadata for open notebooks, so reconverted notebooks are not stuck on an old alias

[akrambel2115]

• The current version is 0.0.4, if you intend this PR to produce a new release you should bump the version to 0.0.5

[akrambel2115]

• Sibling-file overwrite: getSiblingUri() always targets same-name.sql or same-name.isqlnb in the same folder. If that file already exists and the user confirms overwrite, you’ll replace it entirely (no backup/rename/versioning). That’s expected behavior, but it’s still a high-impact path.

• No “Save As…” option: Users can’t choose an output location/name; sibling-only can be surprising in multi-root workspaces or when converting files from read-only / generated folders.

• Potentially expensive work on UI thread: Reading whole SQL files and running analyzeSqlNotebookText() could be heavy for large .sql files. If analysis/serialization is CPU-intensive, VS Code may feel frozen (especially since this is all in a single awaited command handler without chunking/offloading).

• Large notebook JSON stringify: Writing JSON.stringify(..., null, 2) for big notebooks can be slow and creates large files (pretty-printing increases size).

• User-facing errors may be too generic: Errors are shown as Failed to convert ...: <message>, but you don’t include which source/destination URI was involved. This can make debugging harder, especially with remote/workspace FS providers.

• catch { return false; } in pathExists(): Treats any error as “doesn’t exist” (including permission errors, transient FS errors, unsupported scheme). That can lead to:

  • skipping overwrite confirmation when you actually can’t stat it,
  • then failing later on write with a less clear error.

• validateDocument is invoked on every onDidChangeTextDocument event, and it re-parses the entire SQL document (document.getText() + full analysis). For larger .sql files this can become noticeably expensive on each keystroke. Consider debouncing/throttling validation per-document (e.g., short timeout) or limiting full re-analysis to save/idle while still keeping diagnostics responsive.