diff: avoid segfault with freed entries (#835)

This is also going upstream as part of gitgitgadget#2027.

However, I found out about this segfault due to a reproducible error
happening in the 1JS monorepo's PR build pipelines. After trying to
uncover a repro by adding tracing to my own fork and using that version
within the pipeline, I was finally able to get a reproducer in a 1JS
Codespace. Debugging with a locally-build version of Git helped me find
the situation to fix this.

1JS has a workaround for this issue by running `git status` before the
necessary build step that runs the `git diff` command. That refresh of
the index prevents the freed diff queue items, preventing the issue.
This reduces the urgency somewhat, but I also don't know where else this
could be impacting users.

* [X] This is an early version of work already under review upstream.

I'm recommending this version on top of the `vfs-2.52.0` branch so we
can get this fixed more quickly for an upcoming release of
microsoft/git.

Author: dscho

diff.c

@@ -7053,6 +7053,7 @@ static void diffcore_skip_stat_unmatch(struct diff_options *diffopt)
 			if (!diffopt->flags.no_index)
 				diffopt->skip_stat_unmatch++;
 			diff_free_filepair(p);
+			q->queue[i] = NULL;
 		}
 	}
 	free(q->queue);
@@ -7096,6 +7097,10 @@ void diff_queued_diff_prefetch(void *repository)
 
 	for (i = 0; i < q->nr; i++) {
 		struct diff_filepair *p = q->queue[i];
+
+		if (!p)
+			continue;
+
 		diff_add_if_missing(repo, &to_fetch, p->one);
 		diff_add_if_missing(repo, &to_fetch, p->two);
 	}

t/t4067-diff-partial-clone.sh

@@ -132,6 +132,41 @@ test_expect_success 'diff with rename detection batches blobs' '
 	test_line_count = 1 done_lines
 '
 
+test_expect_success 'diff succeeds even if entries are removed from queue' '
+	test_when_finished "rm -rf server client trace" &&
+
+	test_create_repo server &&
+	for l in a c e g i p
+	do
+		echo $l >server/$l &&
+		git -C server add $l || return 1
+	done &&
+	git -C server commit -m x &&
+
+	for l in a e i
+	do
+		git -C server rm $l || return 1
+	done &&
+
+	for l in b d f i
+		do
+		echo $l$l >server/$l &&
+		git -C server add $l || return 1
+	done &&
+	git -C server commit -a -m x &&
+
+	test_config -C server uploadpack.allowfilter 1 &&
+	test_config -C server uploadpack.allowanysha1inwant 1 &&
+	git clone --filter=blob:limit=0 "file://$(pwd)/server" client &&
+
+	for file in $(ls client)
+	do
+		cat client/$file >$file &&
+		mv $file client/$file || return 1
+	done &&
+	git -C client diff --name-only --relative HEAD^
+'
+
 test_expect_success 'diff does not fetch anything if inexact rename detection is not needed' '
 	test_when_finished "rm -rf server client trace" &&