Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/reference/ignore-eject.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,46 @@ factlog eject report.pdf --delete-original # also delete the user's original un
factlog eject report.pdf --dry-run # show the planned changes, modify nothing
```

### How a source is named — filenames are wide, paths are narrow

| What you name | What it matches |
|---------|---------|
| stem `report` | **every** source with that stem, and their conversions |
| filename `report.html` | **every** source with that filename in any directory, and their conversions |
| path `sub/report.html`, `./report.html` | only the conversion made from the original at that path under `sources/` |
| KB-relative ref `sources/sub/report.html` | that original + the conversion made from it |
| absolute path `/kb/sources/sub/report.html` | the same (it reduces to the KB-relative ref) |

**A bare filename is not a path.** `factlog eject report.html` matches widely by
design, so it also catches `sources/sub/report.html`. To take only the top-level
one, name a path — `./report.html` for its conversion alone, or the KB-relative
ref `sources/report.html` to include the original itself (which is what makes
`--delete-original` actually delete it). The two table rows above are that
difference.

Given a path, only the conversion made from *that* path matches — conversions
mirror the original's subdirectory under `runs/sources/`, so
`factlog eject sub/report.html` deletes `runs/sources/sub/report.html.md` and
leaves `runs/sources/report.html.md`, made from a same-name original in another
directory, alone.

A path is compared **as written**: no `..` folding and no case folding, so
`sub/../report.html` and `SUB/report.html` match nothing and exit 1 — even on a
case-insensitive filesystem. Only an absolute path is resolved (symlinks
included) and reduced to a KB-relative ref; a relative path is not, so one
spelled through a symlinked directory name (`link/report.html`) matches nothing —
use the real path (`real/report.html`) or an absolute path.

To delete the original too (`--delete-original`), name it by its KB-relative ref
(`sources/sub/report.html`) or by absolute path. A sources-relative path
(`sub/report.html`) names the **conversion** made from it.

An original ingested from outside `sources/` (e.g.
`factlog ingest /elsewhere/report.html`) has no subtree to mirror, so its
conversion is written flat under `runs/sources/`. Naming that original by path
matches the flat conversion only: a conversion in a subdirectory cannot have come
from it.

### Removing a single fact (`--fact`)

When a source is fine but one extracted fact is wrong, retire just that fact —
Expand Down
37 changes: 37 additions & 0 deletions docs/reference/ignore-eject.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,43 @@ factlog eject report.pdf --delete-original # also delete the user's original un
factlog eject report.pdf --dry-run # show the planned changes, modify nothing
```

### 이름 지정 방식 — 파일명은 넓고, 경로는 좁습니다

| 지정 형태 | 매칭 범위 |
|---------|---------|
| 어간 `report` | 그 어간을 가진 **모든** 소스와 그 변환본 |
| 파일명 `report.html` | 디렉터리를 가리지 않고 그 이름을 가진 **모든** 소스와 그 변환본 |
| 경로 `sub/report.html`, `./report.html` | `sources/` 기준 그 경로의 원본에서 만들어진 변환본만 |
| KB 기준 ref `sources/sub/report.html` | 그 원본 + 그 원본에서 만들어진 변환본 |
| 절대 경로 `/kb/sources/sub/report.html` | 위와 동일(KB 기준 ref 로 환원) |

**파일명은 경로 지정이 아닙니다.** `factlog eject report.html` 은 의도적으로 넓게
매칭되므로 `sources/sub/report.html` 쪽도 함께 걸립니다. 최상위 것만 빼려면 경로로
지정하십시오 — 변환본만 지우려면 `./report.html`, 원본까지 대상에 넣으려면(즉
`--delete-original` 이 실제로 원본을 지우게 하려면) `sources/report.html` 처럼 KB
기준 ref 로 지정합니다. 위 표의 두 줄이 그 차이입니다.

경로를 주면 그 경로에서 만들어진 변환본만 매칭됩니다 — 변환본은 `runs/sources/`
아래에 원본의 서브디렉터리를 미러링하므로, `factlog eject sub/report.html` 은
`runs/sources/sub/report.html.md` 만 지우고 다른 디렉터리의 동명 원본이 만든
`runs/sources/report.html.md` 는 건드리지 않습니다.

경로는 **적힌 그대로** 비교합니다. `..` 를 접거나 대소문자를 무시하지 않으므로,
`sub/../report.html` 과 `SUB/report.html` 은 (파일시스템이 대소문자를 구분하지
않더라도) 아무것도 매칭하지 않고 종료 코드 1 로 끝납니다. 심링크가 풀리는 것은
절대 경로뿐입니다 — 상대 경로는 풀지 않으므로, 심링크 디렉터리 이름을 거쳐 적은
상대 경로(`link/report.html`)는 매칭되지 않습니다. 실제 경로(`real/report.html`)나
절대 경로로 지정하십시오.

`--delete-original` 로 원본까지 지우려면 원본의 KB 기준 ref(`sources/sub/report.html`)
나 절대 경로로 지정하십시오. `sources/` 기준 경로(`sub/report.html`)는 그 경로에서
만들어진 **변환본**을 가리킵니다.

`sources/` 밖의 원본을 적재했다면(예: `factlog ingest /elsewhere/report.html`)
미러링할 서브트리가 없어 변환본이 `runs/sources/` 바로 아래 평면으로 만들어집니다.
그 원본을 경로로 지정하면 평면 변환본만 매칭되고, 서브디렉터리의 변환본은 그
경로에서 만들어졌을 수 없으므로 절대 걸리지 않습니다.

### 사실 하나만 제거 (`--fact`)

소스 자체는 멀쩡한데 추출된 사실 하나가 잘못된 경우, 그 사실만 폐기할 수
Expand Down
145 changes: 117 additions & 28 deletions factlog/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -2317,6 +2317,11 @@ def _select_eject_sources(args, rows, disk_refs, all_refs, target, nfc):
# so a stem guess would let `eject report.docx` wrongly pull report.pptx's
# conversion; the recorded origin name disambiguates. Falls back to a stem
# match only when no header is present (a hand-made conversion).
#
# The stored value is the original's path *relative to sources/* — the
# conversion's own mirrored subdir joined with the basename read from the
# header (see the loop below). Consumers that only want the basename go
# through origin_name().
conv_origin: dict[str, str] = {}
for ref, p in disk_refs.items():
if not ref.startswith("runs/sources/"):
Expand All @@ -2336,28 +2341,109 @@ def _select_eject_sources(args, rows, disk_refs, all_refs, target, nfc):
origin = nfc(m.group(1).strip())
if origin:
# #214: the header may now record a sources/-relative path
# (sub_a/data.hwpx) rather than a bare basename. Reduce it to the
# basename so the pairing/orphan reconstruction below — which
# rebuilds sources/<subdir>/<origin> from the conversion's own
# mirrored subdir — stays correct for both header formats and no
# legacy basename header regresses.
conv_origin[ref] = PurePosixPath(origin).name

def matches(ref: str, name: str) -> bool:
name = nfc(name)
rp, np_ = Path(ref), Path(name)
if ref == name: # exact KB-relative ref
# (sub_a/data.hwpx) rather than a bare basename. Rebuild the
# original's sources-relative path from the conversion's *own*
# mirrored subdir plus the header's basename, so both header
# formats yield the same value and no legacy basename header
# regresses: ingest derives the mirrored subdir and the header
# label from one src_rel, so the header's directory component is
# never new information (and is the wrong signal to trust when a
# hand-edit makes the two disagree — the file's own location is
# what the conversion actually mirrors).
# Only PurePosixPath joining is used: it folds "//" and "./" but
# keeps ".." verbatim, so a stored origin still spells the header
# the same way a consumer would.
subdir = PurePosixPath(ref[len("runs/sources/"):]).parent
base = PurePosixPath(origin).name
# A header with no filename component ("source: /") is as
# unusable as an empty one; keep the empty-origin sentinel rather
# than inventing an original named after the subdir.
conv_origin[ref] = (subdir / base).as_posix() if base else ""

def origin_name(ref: str) -> str | None:
"""Basename of the original a conversion was made from — None when the
conversion carries no provenance header at all."""
origin = conv_origin.get(ref)
return PurePosixPath(origin).name if origin is not None else None

# resolve() reports a symlink loop as RuntimeError (not OSError) and a path
# with an embedded NUL as ValueError, and neither may reach the user as a
# traceback: an unresolvable argument simply matches nothing.
_RESOLVE_ERRORS = (OSError, RuntimeError, ValueError)

try:
troot = target.resolve()
except _RESOLVE_ERRORS:
troot = target

def selector(name: str) -> tuple[set[str], str | None, str]:
"""Canonicalise one `eject <name>` argument into what the matcher needs:

refs — the KB-relative ref(s) the argument names exactly;
src_rel — the original's path *relative to sources/*, when a path was
given; compared against conv_origin to reach the conversion
that path produced. None when the argument names no original
under sources/ (a conversion ref, or a bare name);
raw — the argument as written, for the bare filename / stem rules.
"""
raw = nfc(name)
p = Path(raw)
if p.is_absolute():
# Resolve an *absolute* argument, and the root with it: this machine
# can reach the KB through a symlink (/tmp -> /private/tmp), and an
# unresolved argument would then look like it lies outside the KB.
# A relative argument is never resolved — resolving it against the
# cwd would turn `sub/report.html`, typed inside the KB, into an
# outside-the-KB path and drop it back to basename matching, which
# is the deletion #324 is about.
try:
p = p.resolve()
except _RESOLVE_ERRORS:
pass
try:
kb_rel = nfc(p.relative_to(troot).as_posix())
except ValueError:
kb_rel = None
if kb_rel is not None and (kb_rel == "sources" or kb_rel.startswith("sources/")):
return {kb_rel}, kb_rel[len("sources/"):] or None, raw
# An original outside sources/ — anywhere else in the KB, or outside
# it entirely — has no subtree for ingest to mirror, so it always
# converts to a *flat* runs/sources/<name> whose rebuilt origin is a
# bare basename. Comparing against that basename keeps the legitimate
# case working while leaving every mirrored conversion out of reach.
return ({kb_rel} if kb_rel is not None else set()), nfc(p.name), raw
# A relative argument is read KB-relative (`sources/...`, `runs/...`) or
# sources-relative (`sub/report.html`). PurePosixPath folds "./" and "//"
# but keeps ".." verbatim: no normpath/realpath here, so the comparison
# never rewrites a path into something the recorded origin spells
# differently.
norm = PurePosixPath(raw).as_posix() if "/" in raw else raw
if norm.startswith("sources/"):
return {raw, norm}, norm[len("sources/"):] or None, raw
if norm.startswith("runs/sources/"):
return {raw, norm}, None, raw # names a conversion, not an original
return {raw, norm}, norm, raw

def matches(ref: str, sel: tuple[set[str], str | None, str]) -> bool:
refs, src_rel, name = sel
if ref in refs: # exact KB-relative ref
return True
is_conv = ref.startswith("runs/sources/")
if "/" in name:
# A path was given: the exact original is handled above; for a
# binary original also match the conversion it produced (by
# recorded origin). Same-basename files elsewhere are NOT matched.
return is_conv and conv_origin.get(ref) == np_.name
# binary original also match the conversion it produced — the one
# whose recorded origin *is* that sources-relative path. #324: a
# same-name original in another directory is NOT matched, because
# both sides keep their directory instead of collapsing to a
# basename. An *empty* src_rel (a root path like "/" names no file)
# must not compare equal either: the empty string is also the
# no-usable-origin sentinel stored for a header like "source: /".
return bool(is_conv and src_rel and conv_origin.get(ref) == src_rel)
rp, np_ = Path(ref), Path(name)
if np_.suffix: # a bare filename with an extension
if not is_conv:
return rp.name == np_.name # an original with that filename
origin = conv_origin.get(ref) # the conversion made from this original
origin = origin_name(ref) # the conversion made from this original
# Provenance is the reliable signal. A headerless conversion falls
# back to its own name minus the ingest out-suffix: since ingest now
# keeps the original's extension (report.pptx -> report.pptx.md), the
Expand All @@ -2367,7 +2453,7 @@ def matches(ref: str, name: str) -> bool:
# one (matched via its recorded origin so the source's own extension in
# the new naming — report.pptx.md — does not defeat the stem compare).
if is_conv:
origin = conv_origin.get(ref)
origin = origin_name(ref)
return Path(origin if origin else rp.name).stem == np_.stem
return rp.stem == np_.stem

Expand All @@ -2393,21 +2479,18 @@ def matches(ref: str, name: str) -> bool:
# a subtree to mirror, so the subdir is unknown) has only the
# provenance basename as an origin signal; match by basename and keep
# erring toward retention.
from pathlib import PurePosixPath

src_basenames = {Path(r).name for r in disk_refs if not r.startswith("runs/sources/")}
for ref in all_refs:
if ref.startswith("runs/sources/"):
if ref in disk_refs:
origin = conv_origin.get(ref)
# origin is not None == has a factlog provenance header
# (hand-placed conversions are kept).
# (hand-placed conversions are kept). It already carries the
# conversion's mirrored subdir, so a "/" in it *is* the
# mirrored case and sources/<origin> is the exact original.
if origin is not None:
conv_rel = ref[len("runs/sources/"):]
subdir = PurePosixPath(conv_rel).parent
if subdir.as_posix() != ".":
expected = (PurePosixPath("sources") / subdir / origin).as_posix()
paired = expected in disk_refs
if "/" in origin:
paired = f"sources/{origin}" in disk_refs
else:
paired = origin in src_basenames
if not paired:
Expand All @@ -2425,7 +2508,8 @@ def matches(ref: str, name: str) -> bool:
print(f"factlog eject (KB: {target}): orphan scan — {len(matched)} orphaned source(s)")
else:
for name in args.sources:
hits = {ref for ref in all_refs if matches(ref, name)}
sel = selector(name)
hits = {ref for ref in all_refs if matches(ref, sel)}
if hits:
matched |= hits
else:
Expand Down Expand Up @@ -2469,9 +2553,14 @@ def cmd_eject(args: argparse.Namespace) -> int:
by default (kept for audit), or removed entirely with --purge;
- optionally deletes the user's original under sources/ with
--delete-original (off by default: ingest never created it).
A source is named by its filename, stem, or KB-relative path. Naming the
binary original (e.g. report.pptx) also matches its runs/sources/<stem>
conversion; a bare stem matches every source with that stem. eject also
A source is named by its filename, stem, or path. Naming the binary original
(e.g. report.pptx) also matches its runs/sources/<stem> conversion; a bare
stem matches every source with that stem. A filename is deliberately wide —
it matches that name in every directory — while a *path* is narrow: it
selects the original at that path and the conversion made from it, never a
same-name original elsewhere (#324). Paths are read relative to sources/ (or
as a KB-relative ref / absolute path) and compared as written: no ".."
folding, no case folding, and only an absolute path is resolved. eject also
catches a source cited only in candidates.csv (an already-orphaned ref).

Orphan mode (`eject --orphans`) selects every orphaned source automatically
Expand Down
Loading