Version the training data with DVC

.dvc/.gitignore

@@ -0,0 +1,3 @@
+/config.local
+/tmp
+/cache

.dvc/config

@@ -0,0 +1,5 @@
+[core]
+    remote = routelet-r2
+['remote "routelet-r2"']
+    url = s3://aegis-routelet-samples/dvc
+    endpointurl = https://3888bcbf92cc98efac814d0108cf9358.r2.cloudflarestorage.com

.dvcignore

@@ -0,0 +1,3 @@
+# Add patterns of files dvc should ignore, which could improve
+# the performance. Learn more at
+# https://dvc.org/doc/user-guide/dvcignore

README.md

@@ -59,6 +59,18 @@ One JSON object per line:
 {"text": "play despacito on spotify", "intent": "integration"}
 ```
 
+## Data versioning
+
+The training pool under `data/` is versioned with [DVC](https://dvc.org); git
+tracks the `*.jsonl.dvc` pointers, the data lives in an R2 remote. After cloning:
+
+```
+dvc pull   # needs R2 credentials in .dvc/config.local (see dvc remote modify --local)
+```
+
+The frozen eval set, `evals/holdout.jsonl`, is committed to git directly so the
+benchmark is reproducible without DVC.
+
 ## License
 
 MIT

data/agent.jsonl.dvc

@@ -0,0 +1,5 @@
+outs:
+- md5: befb0c5056c727158bb34a9032ab5a27
+  size: 5835
+  hash: md5
+  path: agent.jsonl

data/augmented.jsonl.dvc

@@ -0,0 +1,5 @@
+outs:
+- md5: 3125afd6cc9be7bd14b1442d965baf3f
+  size: 95998
+  hash: md5
+  path: augmented.jsonl

data/chat.jsonl.dvc

@@ -0,0 +1,5 @@
+outs:
+- md5: e2e66e5449bd901c87f00ca6ef8096eb
+  size: 7968
+  hash: md5
+  path: chat.jsonl

data/find_action.jsonl.dvc

@@ -0,0 +1,5 @@
+outs:
+- md5: 14f4fae92a55c1222601a1523e423ee2
+  size: 5225
+  hash: md5
+  path: find_action.jsonl

data/integration.jsonl.dvc

@@ -0,0 +1,5 @@
+outs:
+- md5: 778f1024b240d116989879ebf14166fd
+  size: 15150
+  hash: md5
+  path: integration.jsonl

data/memory.jsonl.dvc

@@ -0,0 +1,5 @@
+outs:
+- md5: f7a7d0cd75626226c54387ca480d48b9
+  size: 3912
+  hash: md5
+  path: memory.jsonl