worktree: add --reflink for copy-on-write worktree creation

Creating many worktrees from the same base -- for example to run a
fleet of automated agents in parallel -- is expensive today: every
"git worktree add" materializes the entire working tree by writing
each tracked file out from the object store. The objects are shared
via the common directory, but the working tree is not: N worktrees
mean N full checkouts on disk and N times the file I/O.

Add a "--reflink" option that, on copy-on-write filesystems, populates
the new worktree by reflinking the current worktree's files and index
instead. The subsequent "git reset --hard" then only rewrites the
paths that actually differ between the current worktree and
<commit-ish>; everything else (including untracked files such as build
outputs) keeps sharing storage with the source until modified. Because
the cloned index still carries the source files' stat data, it is
refreshed against the reflinked files first so that reset recognizes
the unchanged paths as up to date and leaves them sharing extents
rather than rewriting them.

The clones are made by a new reflink_file() helper in copy.c, which
uses the FICLONE ioctl on Linux and clonefile() on macOS and reports
an error otherwise so callers fall back to a normal copy. Support is
probed up front; when unavailable -- including on filesystems without
copy-on-write and on platforms such as Windows that lack a reflink
primitive -- "--reflink" transparently falls back to an ordinary
checkout, so the worst case is no slower than today rather than a
byte-for-byte copy of the source tree. The directory walk skips the
new worktree itself when it lives inside the source one, and preserves
symlinks and modes.

The behavior can be made the default with the worktree.reflink
configuration ("true", "false" or "auto", the last suppressing the
unsupported-filesystem warning), and turned off per-invocation with
--no-reflink. A configured default degrades quietly in modes that
cannot reflink (--orphan, --no-checkout) instead of erroring, so
enabling it never breaks those commands. The checkout step continues
to honor checkout.workers, so parallel checkout composes with
--reflink for the paths that do need rewriting.

Signed-off-by: Jason Newton <nevion@gmail.com>

Author: nevion

Documentation/config/worktree.adoc

@@ -17,3 +17,13 @@
 Note that setting `worktree.useRelativePaths` to "`true`" implies enabling the
 `extensions.relativeWorktrees` config (see linkgit:git-config[1]),
 thus making it incompatible with older versions of Git.
+
+`worktree.reflink`::
+	Controls whether `git worktree add` populates new worktrees with
+	copy-on-write (reflink) clones, as if `--reflink` had been given
+	(see linkgit:git-worktree[1]). May be set to "`true`", "`false`"
+	(the default), or "`auto`". With "`true`", a filesystem that does
+	not support reflinks produces a warning before falling back to an
+	ordinary checkout; with "`auto`", the fallback is silent. An
+	explicit `--reflink` or `--no-reflink` on the command line
+	overrides this setting.

Documentation/git-worktree.adoc

@@ -10,7 +10,7 @@ SYNOPSIS
 --------
 [synopsis]
 git worktree add [-f] [--detach] [--checkout] [--lock [--reason <string>]]
-		 [--orphan] [(-b | -B) <new-branch>] <path> [<commit-ish>]
+		 [--orphan] [--reflink] [(-b | -B) <new-branch>] <path> [<commit-ish>]
 git worktree list [-v | --porcelain [-z]]
 git worktree lock [--reason <string>] <worktree>
 git worktree move <worktree> <new-path>
@@ -213,6 +213,34 @@ To remove a locked worktree, specify `--force` twice.
 	such as configuring sparse-checkout. See "Sparse checkout"
 	in linkgit:git-read-tree[1].
 
+`--reflink`::
+`--no-reflink`::
+	Populate the new worktree by creating copy-on-write (reflink)
+	clones of the current worktree's files and index instead of
+	writing every file out from the object store. The checkout that
+	follows then only has to rewrite the paths that actually differ
+	between the current worktree and _<commit-ish>_; everything else
+	(including untracked files such as build outputs) keeps sharing
+	storage with the source worktree until modified. This makes
+	`add` much faster and far cheaper on disk when creating many
+	worktrees from the same base. `--no-reflink` forces an ordinary
+	checkout, overriding the `worktree.reflink` configuration.
++
+Reflinks require a copy-on-write filesystem (for example btrfs, XFS,
+bcachefs or ZFS on Linux, or APFS on macOS). On filesystems or platforms
+that do not support reflinks, `--reflink` transparently falls back to an
+ordinary checkout. Because the source worktree's untracked and ignored
+files are cloned as well, only use `--reflink` when that is acceptable.
++
+This option cannot be combined with `--no-checkout` or `--orphan`. It can
+be enabled by default with the `worktree.reflink` configuration; see
+linkgit:git-config[1].
++
+The checkout that populates a new worktree also honors the
+`checkout.workers` configuration (see linkgit:git-config[1]), so setting it
+parallelizes the file writes and can further speed up `add`, with or
+without `--reflink`.
+
 `--guess-remote`::
 `--no-guess-remote`::
 	With `worktree add <path>`, without _<commit-ish>_, instead

builtin/worktree.c

@@ -30,7 +30,7 @@
 
 #define BUILTIN_WORKTREE_ADD_USAGE \
 	N_("git worktree add [-f] [--detach] [--checkout] [--lock [--reason <string>]]\n" \
-	   "                 [--orphan] [(-b | -B) <new-branch>] <path> [<commit-ish>]")
+	   "                 [--orphan] [--reflink] [(-b | -B) <new-branch>] <path> [<commit-ish>]")
 
 #define BUILTIN_WORKTREE_LIST_USAGE \
 	N_("git worktree list [-v | --porcelain [-z]]")
@@ -47,6 +47,11 @@
 #define BUILTIN_WORKTREE_UNLOCK_USAGE \
 	N_("git worktree unlock <worktree>")
 
+/* values for add_opts.reflink and the worktree.reflink config */
+#define REFLINK_OFF	0
+#define REFLINK_ON	1	/* warn and fall back if unsupported */
+#define REFLINK_AUTO	2	/* silently fall back if unsupported */
+
 #define WORKTREE_ADD_DWIM_ORPHAN_INFER_TEXT \
 	_("No possible source branch, inferring '--orphan'")
 
@@ -123,13 +128,15 @@ struct add_opts {
 	int checkout;
 	int orphan;
 	int relative_paths;
+	int reflink;
 	const char *keep_locked;
 };
 
 static int show_only;
 static int verbose;
 static int guess_remote;
 static int use_relative_paths;
+static int reflink_config = REFLINK_OFF;
 static timestamp_t expire;
 
 static int git_worktree_config(const char *var, const char *value,
@@ -141,6 +148,13 @@ static int git_worktree_config(const char *var, const char *value,
 	} else if (!strcmp(var, "worktree.userelativepaths")) {
 		use_relative_paths = git_config_bool(var, value);
 		return 0;
+	} else if (!strcmp(var, "worktree.reflink")) {
+		if (value && !strcmp(value, "auto"))
+			reflink_config = REFLINK_AUTO;
+		else
+			reflink_config = git_config_bool(var, value) ?
+				REFLINK_ON : REFLINK_OFF;
+		return 0;
 	}
 
 	return git_default_config(var, value, ctx, cb);
@@ -397,6 +411,182 @@ static void copy_filtered_worktree_config(const char *worktree_git_dir)
 	free(to_file);
 }
 
+/*
+ * Probe whether the filesystem backing "dir" supports reflinks. We do this
+ * once up front so that, on filesystems without copy-on-write support (or on
+ * platforms such as Windows that lack a reflink primitive entirely), we can
+ * fall back to a normal checkout instead of byte-copying the whole source
+ * working tree -- which would include untracked files and be slower than the
+ * checkout we are trying to avoid.
+ */
+static int reflink_supported(const char *dir)
+{
+	struct strbuf src = STRBUF_INIT, dst = STRBUF_INIT;
+	int fd, ok = 0;
+
+	strbuf_addf(&src, "%s/.git-reflink-probe-src", dir);
+	strbuf_addf(&dst, "%s/.git-reflink-probe-dst", dir);
+
+	fd = open(src.buf, O_WRONLY | O_CREAT | O_TRUNC, 0600);
+	if (fd >= 0) {
+		write_in_full(fd, "x", 1);
+		close(fd);
+		if (!reflink_file(dst.buf, src.buf, 0600))
+			ok = 1;
+		unlink(dst.buf);
+	}
+	unlink(src.buf);
+
+	strbuf_release(&src);
+	strbuf_release(&dst);
+	return ok;
+}
+
+/*
+ * Reflink a single regular file, falling back to a regular copy when the
+ * clone fails for this particular file (for example across mount points).
+ */
+static int reflink_or_copy(const char *dst, const char *src, int mode)
+{
+	if (!reflink_file(dst, src, mode))
+		return 0;
+	return copy_file(dst, src, mode);
+}
+
+/*
+ * Recursively copy the working-tree directory "src" into "dst" using reflinks
+ * for regular files. Directory entries that resolve to the destination
+ * worktree itself (identified by skip_dev/skip_ino, which matters when the new
+ * worktree lives inside the source one) and the top-level ".git" gitfile are
+ * skipped.
+ */
+static int reflink_tree(const char *src, const char *dst,
+			dev_t skip_dev, ino_t skip_ino, int top)
+{
+	struct strbuf s = STRBUF_INIT, d = STRBUF_INIT;
+	DIR *dir;
+	struct dirent *de;
+	int ret = 0;
+
+	dir = opendir(src);
+	if (!dir)
+		return error_errno(_("could not open directory '%s'"), src);
+
+	while (!ret && (de = readdir(dir))) {
+		struct stat st;
+
+		if (is_dot_or_dotdot(de->d_name))
+			continue;
+		if (top && !strcmp(de->d_name, ".git"))
+			continue;
+
+		strbuf_reset(&s);
+		strbuf_addf(&s, "%s/%s", src, de->d_name);
+		strbuf_reset(&d);
+		strbuf_addf(&d, "%s/%s", dst, de->d_name);
+
+		if (lstat(s.buf, &st)) {
+			ret = error_errno(_("could not stat '%s'"), s.buf);
+			break;
+		}
+
+		if (S_ISDIR(st.st_mode)) {
+			/* never recurse into the new worktree itself */
+			if (st.st_dev == skip_dev && st.st_ino == skip_ino)
+				continue;
+			if (mkdir(d.buf, st.st_mode & 07777) && errno != EEXIST) {
+				ret = error_errno(_("could not create directory '%s'"), d.buf);
+				break;
+			}
+			ret = reflink_tree(s.buf, d.buf, skip_dev, skip_ino, 0);
+		} else if (S_ISLNK(st.st_mode)) {
+			struct strbuf link = STRBUF_INIT;
+
+			if (strbuf_readlink(&link, s.buf, st.st_size))
+				ret = error_errno(_("could not read symlink '%s'"), s.buf);
+			else if (symlink(link.buf, d.buf))
+				ret = error_errno(_("could not create symlink '%s'"), d.buf);
+			strbuf_release(&link);
+		} else if (S_ISREG(st.st_mode)) {
+			if (reflink_or_copy(d.buf, s.buf, st.st_mode))
+				ret = error_errno(_("could not copy '%s' to '%s'"),
+						  s.buf, d.buf);
+		}
+		/* silently skip fifos, sockets and device nodes */
+	}
+
+	closedir(dir);
+	strbuf_release(&s);
+	strbuf_release(&d);
+	return ret;
+}
+
+/*
+ * Populate the new worktree at "path" by reflinking the current worktree's
+ * files and index. The subsequent "git reset --hard" then only has to rewrite
+ * the paths that actually differ between the source and <commit-ish>, leaving
+ * everything else sharing storage with the source. Returns 1 when reflinks are
+ * unavailable so the caller can fall back to a plain checkout.
+ */
+static int reflink_worktree(const char *path, const char *wt_git_dir,
+			    const struct add_opts *opts, struct strvec *child_env)
+{
+	const char *src_wt = the_repository->worktree;
+	char *src_index = NULL, *dst_index = NULL;
+	struct stat dst_st;
+	int ret = 0;
+
+	if (!src_wt)
+		return error(_("--reflink needs a source working tree, but this "
+			       "repository does not have one"));
+
+	if (!reflink_supported(path)) {
+		/* In auto mode the fallback is expected, so stay quiet. */
+		if (!opts->quiet && opts->reflink != REFLINK_AUTO)
+			warning(_("the filesystem at '%s' does not support reflinks; "
+				  "falling back to a regular checkout"), path);
+		return 1;
+	}
+
+	if (stat(path, &dst_st))
+		return error_errno(_("could not stat '%s'"), path);
+
+	if ((ret = reflink_tree(src_wt, path, dst_st.st_dev, dst_st.st_ino, 1)))
+		return ret;
+
+	/*
+	 * Clone the source index so the following reset sees the source's
+	 * state and only materializes the differences to <commit-ish>.
+	 */
+	src_index = repo_git_path(the_repository, "index");
+	dst_index = xstrfmt("%s/index", wt_git_dir);
+	if (!access(src_index, F_OK) &&
+	    reflink_or_copy(dst_index, src_index, 0666)) {
+		ret = error_errno(_("could not copy index to '%s'"), dst_index);
+		goto out;
+	}
+
+	/*
+	 * The cloned index still carries the source files' stat information.
+	 * Refresh it against the freshly reflinked files so that "git reset"
+	 * recognizes unchanged paths as up to date and leaves them sharing
+	 * storage instead of rewriting (and thus un-sharing) them.
+	 */
+	if (!access(dst_index, F_OK)) {
+		struct child_process cp = CHILD_PROCESS_INIT;
+		cp.git_cmd = 1;
+		strvec_pushl(&cp.args, "update-index", "-q", "--refresh", NULL);
+		strvec_pushv(&cp.env, child_env->v);
+		/* a dirty working tree is not an error here */
+		run_command(&cp);
+	}
+
+out:
+	free(src_index);
+	free(dst_index);
+	return ret;
+}
+
 static int checkout_worktree(const struct add_opts *opts,
 			     struct strvec *child_env)
 {
@@ -589,6 +779,20 @@ static int add_worktree(const char *path, const char *refname,
 	    (ret = make_worktree_orphan(refname, opts, &child_env)))
 		goto done;
 
+	/*
+	 * When --reflink is requested and the filesystem supports it, copy the
+	 * current worktree (and its index) into the new one using copy-on-write
+	 * clones. checkout_worktree() then only rewrites the paths that differ
+	 * from <commit-ish>. reflink_worktree() returns 1 when reflinks are not
+	 * available, in which case we just do an ordinary checkout below.
+	 */
+	if (opts->checkout && opts->reflink) {
+		ret = reflink_worktree(path, sb_repo.buf, opts, &child_env);
+		if (ret < 0)
+			goto done;
+		ret = 0;
+	}
+
 	if (opts->checkout &&
 	    (ret = checkout_worktree(opts, &child_env)))
 		goto done;
@@ -801,6 +1005,7 @@ static int add(int ac, const char **av, const char *prefix,
 	const char *lock_reason = NULL;
 	int keep_locked = 0;
 	int used_new_branch_options;
+	int reflink_cli = -1;
 	struct option options[] = {
 		OPT__FORCE(&opts.force,
 			   N_("checkout <branch> even if already checked out in other worktree"),
@@ -823,6 +1028,8 @@ static int add(int ac, const char **av, const char *prefix,
 			 N_("try to match the new branch name with a remote-tracking branch")),
 		OPT_BOOL(0, "relative-paths", &opts.relative_paths,
 			 N_("use relative paths for worktrees")),
+		OPT_BOOL(0, "reflink", &reflink_cli,
+			 N_("populate the worktree using copy-on-write clones when supported")),
 		OPT_END()
 	};
 	int ret;
@@ -842,6 +1049,24 @@ static int add(int ac, const char **av, const char *prefix,
 	if (opts.orphan && !opts.checkout)
 		die(_("options '%s' and '%s' cannot be used together"),
 		    "--orphan", "--no-checkout");
+
+	/*
+	 * Resolve whether to reflink: an explicit --reflink/--no-reflink on
+	 * the command line wins, otherwise fall back to the worktree.reflink
+	 * configuration (which may select the "auto" mode).
+	 */
+	opts.reflink = (reflink_cli != -1) ? reflink_cli : reflink_config;
+	if (opts.reflink && (opts.orphan || !opts.checkout)) {
+		/*
+		 * Reflinking is incompatible with these; only complain when it
+		 * was explicitly requested, otherwise quietly do a plain
+		 * checkout so a configured default does not break these modes.
+		 */
+		if (reflink_cli == REFLINK_ON)
+			die(_("options '%s' and '%s' cannot be used together"),
+			    "--reflink", opts.orphan ? "--orphan" : "--no-checkout");
+		opts.reflink = REFLINK_OFF;
+	}
 	if (opts.orphan && ac == 2)
 		die(_("option '%s' and commit-ish cannot be used together"),
 		    "--orphan");