Fix CLI JSON and terminal output controls

DEVELOPER_GUIDE.md

@@ -97,6 +97,12 @@ Usage: <command shape>
 
 When an error code is available, the first line is `Error [<code>]: <message>`. Use `CommandErrorWriter` for new CLI parse, validation, and filesystem preflight errors so `ProgramRunner`, `IndexCommandRunner`, and query runners keep the same format. JSON error payloads continue to use `CommandErrorJsonResult`.
 
+### CLI output encoding and terminal controls
+
+CLI JSON output must be machine-clean: redirected stdout is written as UTF-8 without a BOM, and JSON-mode commands must not emit ANSI escape sequences even when `--color=always` or `CLICOLOR_FORCE=1` would color human output. Keep JSON-safe styling suppression close to shared formatting helpers such as `ConsoleUi.ColorizeKind` so future query output paths inherit the invariant.
+
+Interactive terminal controls are allowed only when stdout is not redirected or captured, terminal capability hints are present, and the environment has not opted out. Treat `TERM=dumb`, truthy `CI`, missing Unix terminal hints, `NO_COLOR`, and `CLICOLOR=0` as reasons to suppress ANSI/progress controls unless an explicitly human-facing override is documented for that control.
+
 ### C# / .NET integration
 
 `SolutionProjectResolver` parses the plain-text `.sln` `Project(...) = "...", "...csproj"` entries and resolves C# / F# / VB project files. When exactly one `.sln` exists at the workspace root, `--project <name|path>` uses it automatically; otherwise callers can pass `--solution <path>`.

README.md

@@ -73,6 +73,8 @@ Output controls:
 |---|---|
 | Owner-only persistent stderr logs on POSIX | Global tool stderr logs are forced to `0600` permissions on every open, including existing date-stamped log files. |
 | ASCII-only terminal output | Use `--ascii`, `CDIDX_ASCII=1`, `NO_UNICODE`, `TERM=dumb`, accessibility env hints, or a non-UTF-8 locale. Spinners use pipe, slash, dash, and backslash frames; progress bars use `#` / `-`; very narrow terminals fall back to percentage-only progress. |
+| Color and terminal capability | `--color auto` emits ANSI only for capable interactive terminals; `TERM=dumb`, `CI=true`, missing Unix terminal hints, `NO_COLOR`, or `CLICOLOR=0` disable ANSI/progress control sequences. `--palette basic|256|truecolor` can override the `COLORTERM` / `TERM` color-depth detection. |
+| UTF-8 JSON pipelines | CLI `--json` output is written as UTF-8 without a BOM and never includes ANSI escape sequences, even when color is forced for human output. |
 | Script-friendly query pipelines | Use `--quiet`, `-q`, `--silent`, or `CDIDX_QUIET=1` to suppress informational stderr text while preserving errors. `--quiet` takes precedence over `--verbose`. |
 
 Use `cdidx` when a repository will be searched repeatedly from terminals,
@@ -256,6 +258,8 @@ cdidx mcp
 |---|---|
 | POSIX の persistent stderr log を owner-only にする | global tool stderr log は開くたびに `0600` 権限へ補正され、既存の日付付き log file も同じ扱いになります。 |
 | ASCII-only 端末で崩れない表示にする | `--ascii`、`CDIDX_ASCII=1`、`NO_UNICODE`、`TERM=dumb`、accessibility 系の環境変数、非 UTF-8 locale を使います。スピナーは pipe、slash、dash、backslash の frame、進捗バーは `#` / `-` になり、幅が非常に狭い端末では percentage-only になります。 |
+| color と端末 capability | `--color auto` は対応する interactive terminal でだけ ANSI を出力します。`TERM=dumb`、`CI=true`、Unix で端末 hint が無い場合、`NO_COLOR`、`CLICOLOR=0` では ANSI / progress 制御シーケンスを抑止します。`--palette basic|256|truecolor` で `COLORTERM` / `TERM` による color-depth 判定を上書きできます。 |
+| UTF-8 JSON pipeline | CLI の `--json` 出力は BOM なし UTF-8 で書き出され、human output 向けに色を強制していても ANSI escape sequence を含みません。 |
 | script 向け query pipeline の stderr を静かにする | `--quiet`、`-q`、`--silent`、`CDIDX_QUIET=1` で informational stderr を抑制し、error 行だけを残します。`--quiet` は `--verbose` より優先されます。 |
 
 ターミナル、スクリプト、CI、AI ツールから同じリポジトリを繰り返し検索する

changelog.d/unreleased/1833.fixed.md

@@ -0,0 +1,18 @@
+---
+category: fixed
+issues:
+  - 1833
+affected:
+  - src/CodeIndex/Cli/ConsoleUi.cs
+  - tests/CodeIndex.Tests/ConsoleUiTests.cs
+  - README.md
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Interactive terminal detection now honors CI and terminal opt-outs (#1833)** — `TERM=dumb`, truthy `CI`, missing Unix terminal hints, redirected stdout, and test captures suppress spinner/progress control sequences.
+
+## 日本語
+
+- **interactive terminal 判定が CI と端末 opt-out を尊重するようになりました (#1833)** — `TERM=dumb`、truthy な `CI`、Unix で端末 hint が無い場合、redirect stdout、test capture では spinner/progress 制御シーケンスを抑止します。

changelog.d/unreleased/1834.fixed.md

@@ -0,0 +1,15 @@
+---
+category: fixed
+issues:
+  - 1834
+affected:
+  - README.md
+---
+
+## English
+
+- **Documented color-depth controls for ANSI output (#1834)** — README output controls now describe `COLORTERM` / `TERM` palette detection and the `--palette basic|256|truecolor` override.
+
+## 日本語
+
+- **ANSI 出力の color-depth 制御を文書化しました (#1834)** — README の出力制御に `COLORTERM` / `TERM` による palette 判定と `--palette basic|256|truecolor` override を記載しました。

changelog.d/unreleased/1953.fixed.md

@@ -0,0 +1,17 @@
+---
+category: fixed
+issues:
+  - 1953
+affected:
+  - src/CodeIndex/Cli/ProgramRunner.cs
+  - README.md
+  - DEVELOPER_GUIDE.md
+---
+
+## English
+
+- **Redirected CLI output now uses UTF-8 without a BOM (#1953)** — JSON pipelines receive stable UTF-8 bytes instead of falling back to the host console code page.
+
+## 日本語
+
+- **redirect された CLI 出力を BOM なし UTF-8 で書くようになりました (#1953)** — JSON pipeline が host console code page にフォールバックせず、安定した UTF-8 byte を受け取れます。

changelog.d/unreleased/1956.fixed.md

@@ -0,0 +1,17 @@
+---
+category: fixed
+issues:
+  - 1956
+affected:
+  - src/CodeIndex/Cli/ConsoleUi.cs
+  - src/CodeIndex/Cli/ProgramRunner.cs
+  - tests/CodeIndex.Tests/ConsoleUiTests.cs
+---
+
+## English
+
+- **JSON output now suppresses ANSI styling even when color is forced (#1956)** — `--json` commands keep shared symbol-kind formatting machine-clean while preserving forced color for human output.
+
+## 日本語
+
+- **color を強制していても JSON 出力では ANSI styling を抑止するようになりました (#1956)** — `--json` コマンドは共有の symbol-kind 表示を machine-clean に保ちつつ、人間向け出力の強制 color は維持します。

changelog.d/unreleased/2676.fixed.md

@@ -0,0 +1,16 @@
+---
+category: fixed
+issues:
+  - 2676
+affected:
+  - src/CodeIndex/Database/DbWriter.cs
+  - tests/CodeIndex.Tests/ConcurrencyTests.cs
+---
+
+## English
+
+- **Ready-bit updates no longer start raw nested SQLite transactions (#2676)** — concurrent writers now use provider-managed immediate transactions for `PRAGMA user_version` updates.
+
+## 日本語
+
+- **ready-bit 更新で raw な nested SQLite transaction を開始しないようになりました (#2676)** — concurrent writer は `PRAGMA user_version` 更新に provider-managed immediate transaction を使います。

src/CodeIndex/Cli/ConsoleUi.cs

@@ -108,6 +108,7 @@ public static string FormatSummaryLine(string label, object? value, int labelWid
     private static TextWriter? _synchronizedOut;
     private static TextWriter? _synchronizedError;
     private static readonly string[] ByteUnits = ["bytes", "KiB", "MiB", "GiB", "TiB", "PiB"];
+    private static readonly AsyncLocal<int> JsonOutputDepth = new();
 
     private static readonly string[] DefaultBrailleSpinnerFrames =
     [
@@ -152,6 +153,15 @@ internal static void EnsureConsoleWritersSynchronized()
         }
     }
 
+    internal static IDisposable SuppressAnsiForJsonOutput(bool enabled)
+    {
+        if (!enabled)
+            return NoopDisposable.Instance;
+
+        JsonOutputDepth.Value++;
+        return new JsonOutputScope();
+    }
+
     // --- Spinner / スピナー ---
 
     public static string FormatDuration(TimeSpan duration, DurationOutputFormat format = DurationOutputFormat.Auto)
@@ -1586,7 +1596,7 @@ public static bool TryParseColorMode(string? value, out ColorMode mode)
     public static string ColorizeKind(string kind, int padWidth = 0)
     {
         var padded = padWidth > 0 ? kind.PadRight(padWidth) : kind;
-        if (ShouldUseColor())
+        if (JsonOutputDepth.Value <= 0 && ShouldUseColor())
         {
             var color = GetKindColorCode(kind, ResolveColorPalette());
             if (color.Length > 0)
@@ -1653,18 +1663,23 @@ internal static bool ShouldUseInteractiveConsole()
             Console.Out.Encoding,
             Console.Out is StringWriter,
             HasTerminalEnvironmentHint(),
+            IsTerminalEnvironmentDisabled(),
             OperatingSystem.IsWindows());
 
     internal static bool ShouldUseInteractiveConsole(
         bool isOutputRedirected,
         Encoding outputEncoding,
         bool isTextWriterCapture,
         bool hasTerminalEnvironmentHint,
+        bool isTerminalEnvironmentDisabled,
         bool isWindows)
     {
         if (isOutputRedirected)
             return false;
 
+        if (isTerminalEnvironmentDisabled)
+            return false;
+
         // StringWriter-based test capture leaves the process console attached, so
         // Console.IsOutputRedirected stays false even though interactive terminal
         // behavior would be unsafe. Detect it directly instead of inferring from
@@ -1673,7 +1688,7 @@ internal static bool ShouldUseInteractiveConsole(
         if (isTextWriterCapture)
             return false;
 
-        return true;
+        return isWindows || hasTerminalEnvironmentHint;
     }
 
     internal static bool ShouldUseAnsiOutput()
@@ -1682,6 +1697,7 @@ internal static bool ShouldUseAnsiOutput()
             Console.Out.Encoding,
             Console.Out is StringWriter,
             HasTerminalEnvironmentHint(),
+            IsTerminalEnvironmentDisabled(),
             OperatingSystem.IsWindows(),
             GetWindowsVirtualTerminalProcessingEnabled());
 
@@ -1690,10 +1706,11 @@ internal static bool ShouldUseAnsiOutput(
         Encoding outputEncoding,
         bool isTextWriterCapture,
         bool hasTerminalEnvironmentHint,
+        bool isTerminalEnvironmentDisabled,
         bool isWindows,
         bool windowsVirtualTerminalProcessingEnabled)
     {
-        if (!ShouldUseInteractiveConsole(isOutputRedirected, outputEncoding, isTextWriterCapture, hasTerminalEnvironmentHint, isWindows))
+        if (!ShouldUseInteractiveConsole(isOutputRedirected, outputEncoding, isTextWriterCapture, hasTerminalEnvironmentHint, isTerminalEnvironmentDisabled, isWindows))
             return false;
 
         if (!isWindows)
@@ -1738,6 +1755,19 @@ private static bool HasTerminalEnvironmentHint()
             && !term.Equals("dumb", StringComparison.OrdinalIgnoreCase);
     }
 
+    private static bool IsTerminalEnvironmentDisabled()
+        => IsDumbTerminal() || IsCiEnvironment();
+
+    private static bool IsCiEnvironment()
+    {
+        var ci = Environment.GetEnvironmentVariable("CI");
+        return !string.IsNullOrEmpty(ci)
+            && !ci.Equals("0", StringComparison.OrdinalIgnoreCase)
+            && !ci.Equals("false", StringComparison.OrdinalIgnoreCase)
+            && !ci.Equals("no", StringComparison.OrdinalIgnoreCase)
+            && !ci.Equals("off", StringComparison.OrdinalIgnoreCase);
+    }
+
     private static bool GetWindowsVirtualTerminalProcessingEnabled()
     {
         if (!OperatingSystem.IsWindows())
@@ -1894,4 +1924,21 @@ private static bool TryGetColumnsEnvironmentWidth(out int width)
         width = 0;
         return false;
     }
+
+    private sealed class JsonOutputScope : IDisposable
+    {
+        public void Dispose()
+        {
+            if (JsonOutputDepth.Value > 0)
+                JsonOutputDepth.Value--;
+        }
+    }
+
+    private sealed class NoopDisposable : IDisposable
+    {
+        public static readonly NoopDisposable Instance = new();
+        public void Dispose()
+        {
+        }
+    }
 }

src/CodeIndex/Cli/ProgramRunner.cs

@@ -41,6 +41,7 @@ internal static int Run(
         if (configResult.Loaded)
             GlobalToolLog.Info($"config_file_loaded path={configResult.Path}");
         jsonOptions ??= CreateDefaultJsonOptions();
+        EnsureRedirectedStdoutUsesUtf8();
 
         var quiet = TryConsumeQuietFlag(ref args) || IsTruthyEnvironmentVariable(QuietEnvironmentVariable);
         using var quietScope = quiet ? QuietStderrScope.Start() : null;
@@ -71,6 +72,7 @@ internal static int Run(
         using var metricsSession = MetricsSink.TryStart(metricsPath);
 
         TryConsumeDebugUnsafeFlag(ref args);
+        using var jsonAnsiScope = ConsoleUi.SuppressAnsiForJsonOutput(ContainsJsonOutputFlag(args));
 
         var commandStopwatch = Stopwatch.StartNew();
         var commandStartTimestamp = DateTimeOffset.UtcNow;
@@ -258,6 +260,43 @@ _ when IsProjectPathArg(commandName)
     internal static bool IsProjectPathArg(string arg) =>
         !arg.StartsWith('-') && (Directory.Exists(arg) || arg.Contains('/') || arg.Contains('\\') || arg == ".");
 
+    internal static void EnsureRedirectedStdoutUsesUtf8()
+    {
+        if (!Console.IsOutputRedirected || Console.Out is StringWriter || Console.Out.GetType().Assembly != typeof(Console).Assembly)
+            return;
+
+        var utf8NoBom = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);
+        if (Console.Out.Encoding.CodePage == utf8NoBom.CodePage)
+            return;
+
+        var writer = new StreamWriter(Console.OpenStandardOutput(), utf8NoBom)
+        {
+            AutoFlush = true
+        };
+        Console.SetOut(TextWriter.Synchronized(writer));
+    }
+
+    internal static bool ContainsJsonOutputFlag(IEnumerable<string> args)
+    {
+        var passthrough = false;
+        foreach (var arg in args)
+        {
+            if (passthrough)
+                continue;
+            if (arg == "--")
+            {
+                passthrough = true;
+                continue;
+            }
+            if (arg == "--json"
+                || arg.StartsWith("--json=", StringComparison.Ordinal)
+                || arg == JsonEnvelopeWrapper.EnvelopeFlag)
+                return true;
+        }
+
+        return false;
+    }
+
     internal static bool TryConsumeQuietFlag(ref string[] args)
     {
         if (args.Length == 0)

src/CodeIndex/Database/DbWriter.cs

@@ -340,6 +340,14 @@ private void Execute(string sql)
         cmd.ExecuteNonQuery();
     }
 
+    private void Execute(string sql, SqliteTransaction? transaction)
+    {
+        using var cmd = _conn.CreateCommand();
+        cmd.Transaction = transaction;
+        cmd.CommandText = sql;
+        cmd.ExecuteNonQuery();
+    }
+
     private void RunPassiveWalCheckpoint()
     {
         using var cmd = _conn.CreateCommand();
@@ -3170,34 +3178,44 @@ private void SetReadyBit(int flag)
         // the faster writer's flag. Wrap the read-modify-write in BEGIN IMMEDIATE so
         // SQLite's reserved write lock serialises it across processes (issue #1513).
         bool ownTransaction = !IsInTransaction();
+        bool beganTransaction = ownTransaction;
+        SqliteTransaction? transaction = null;
         if (ownTransaction)
-            Execute("BEGIN IMMEDIATE");
+            transaction = _conn.BeginTransaction(deferred: false);
+        else
+            transaction = _activeTransaction;
         try
         {
             int current;
             using (var read = _conn.CreateCommand())
             {
+                read.Transaction = transaction;
                 read.CommandText = "PRAGMA user_version";
                 var raw = read.ExecuteScalar();
                 current = raw is long l ? (int)l : (raw is int i ? i : 0);
             }
             int next = current | flag;
             if (next != current)
-                Execute($"PRAGMA user_version = {next}");
+                Execute($"PRAGMA user_version = {next}", transaction);
             if (ownTransaction)
             {
-                Execute("COMMIT");
+                transaction!.Commit();
                 ownTransaction = false;
             }
         }
         catch (Exception)
         {
             if (ownTransaction)
             {
-                try { Execute("ROLLBACK"); } catch (SqliteException) { /* best effort */ }
+                try { transaction?.Rollback(); } catch (SqliteException) { /* best effort */ }
             }
             throw;
         }
+        finally
+        {
+            if (beganTransaction)
+                transaction?.Dispose();
+        }
     }
 
     private static List<string> BuildSupportedLanguageParameters(SqliteCommand cmd, IReadOnlyCollection<string> supportedLanguages)