Cannot replicate pre-computed syntactic distance

[letme-hj]

Hi, thank you for your work!

I wanted to ask regarding computing the syntactic distance between languages.

If I understood correctly, pre-computed syntactic distances obtained by

lang2vec.distance("syntactic", [l1, l2])

is the cosine distance between two languages, which should be properly replicated by

from scipy.spatial.distance import cosine

a = lang2vec.get_features(l1, "syntax_wals")[l1]
b = lang2vec.get_features(l2, "syntax_wals")[l2]
cosine(a, b)

And for missing features in a and b (which has -- as their values), I followed what is mentioned here: #7 (comment).

However, I find them mismatch. I also tried it with syntax_knn instead of syntax_wals, but they still mismatch.
And for some of the languages that are involved in pre-computed distances, they only have -- for all features, not actually being able to compute distances with other languages. (e.g., syntactic distance between frr, dan is provided, as shown as an example in README, but l2v.get_features("frr", "syntax_wals") gives a list of "--"s.)

Below are average Pearson correlation coefficients and pvalues between pre-computed and manually computed distances of each language.

• manually computed with syntax_wals & pre-computed : coef - 0.6325433738084123 / pvalue - 0.13051253893837714
• manually computed with syntax_knn & pre-computed : coef - 0.6257979297204636 / pvalue - 0.1392174749552544

I would really appreciate it if you could provide more details on computing the distance if I missed something here!

Thank you so much :)

[matheushjs]

I have the same question. Cannot reproduce the calculation for their pre-computed distances.

I thought that syntactic_distance might use all feature databases syntax_wals, syntax_sswl and syntax_ethnologue.
But even using all of them, I could not reproduce the results.

This is my code:

l1 = "hye"
l2 = "ell"

a_joint = []
b_joint = []

a = np.array(l2v.get_features(l1, "syntax_wals")[l1])
b = np.array(l2v.get_features(l2, "syntax_wals")[l2])

idx = np.array([ i != "--" and j != "--" for i, j in zip(a,b) ])

a = np.float64(a[idx])
b = np.float64(b[idx])

a_joint = np.concatenate([a_joint, a])
b_joint = np.concatenate([b_joint, b])

a = np.array(l2v.get_features(l1, "syntax_sswl")[l1])
b = np.array(l2v.get_features(l2, "syntax_sswl")[l2])

idx = np.array([ i != "--" and j != "--" for i, j in zip(a,b) ])

a = np.float64(a[idx])
b = np.float64(b[idx])

a_joint = np.concatenate([a_joint, a])
b_joint = np.concatenate([b_joint, b])

a = np.array(l2v.get_features(l1, "syntax_ethnologue")[l1])
b = np.array(l2v.get_features(l2, "syntax_ethnologue")[l2])

idx = np.array([ i != "--" and j != "--" for i, j in zip(a,b) ])

a = np.float64(a[idx])
b = np.float64(b[idx])

a_joint = np.concatenate([a_joint, a])
b_joint = np.concatenate([b_joint, b])

print(l2v.syntactic_distance(l1, l2))
# np.float64(0.57)

print(cosine(a_joint, b_joint))
# 0.308406270421389

print(np.mean(a_joint == b_joint))
# 0.7033898305084746

print(np.sqrt(np.sum((a_joint - b_joint)**2)))
# 5.916079783099616