While testing Zingg on synthetic dataset of 30M with about 20% dupes. This was done to see how ti performs at this scale. Same trained model (301)was run in two ways
- using snowflake (completes)
- using spark(doesn't complete)
The model was trained on 52 labeled matching and 52 labeled non matching pairs.
- Snowflake
Snowflake run was completed. The full match took 12+hrs(training - 14min, match- 11hrs 43 minutes).
Driver memory: 12 GB
Input 29.3M records (30M minus 692,727 blank-name records — so not the full 30M), Null values were excluded as the blank-name records seemed to be merging into one giant cluster and stalling the clustering step. So it was removed to rule that out, This has not been confirmed yet.
The output
precision ~81%, recall ~31%
24.9M clusters
2.Spark
It read all records and got through the matching step, then got stuck in the final clustering step. It ran for about 37 hours with no output, never crashed, never finished, and was killed.
Drive Memory - 20GB
Input 30M
Methods to calculate precision and recall
Each record's ID holds the true person number, so records sharing it are genuinely the same person (a "family"). Zingg produces its own groups ("clusters").
Precision - tp / (tp + fp)
Recall - tp / (tp + fn)
Open Items
Run the same match on Snowflake with the null records included, to confirm whether the blank-name records actually caused the giant-cluster issue.
While testing Zingg on synthetic dataset of 30M with about 20% dupes. This was done to see how ti performs at this scale. Same trained model (301)was run in two ways
The model was trained on 52 labeled matching and 52 labeled non matching pairs.
Snowflake run was completed. The full match took 12+hrs(training - 14min, match- 11hrs 43 minutes).
Driver memory: 12 GB
Input 29.3M records (30M minus 692,727 blank-name records — so not the full 30M), Null values were excluded as the blank-name records seemed to be merging into one giant cluster and stalling the clustering step. So it was removed to rule that out, This has not been confirmed yet.
The output
precision ~81%, recall ~31%
24.9M clusters
2.Spark
It read all records and got through the matching step, then got stuck in the final clustering step. It ran for about 37 hours with no output, never crashed, never finished, and was killed.
Drive Memory - 20GB
Input 30M
Methods to calculate precision and recall
Each record's ID holds the true person number, so records sharing it are genuinely the same person (a "family"). Zingg produces its own groups ("clusters").
Precision - tp / (tp + fp)
Recall - tp / (tp + fn)
Open Items
Run the same match on Snowflake with the null records included, to confirm whether the blank-name records actually caused the giant-cluster issue.