Release v1.0

Author: davidanugraha

.github/workflows/python-publish.yml

@@ -0,0 +1,66 @@
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+# GitHub recommends pinning actions to a commit SHA.
+# To get a newer version, you will need to update the SHA.
+# You can also reference a tag or branch, but the action may change without warning.
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  release-build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.x"
+
+      - name: Build release distributions
+        run: |
+          # NOTE: put your own distribution build steps here.
+          python -m pip install build
+          python -m build
+
+      - name: Upload distributions
+        uses: actions/upload-artifact@v4
+        with:
+          name: release-dists
+          path: dist/
+
+  pypi-publish:
+    runs-on: ubuntu-latest
+
+    needs:
+      - release-build
+
+    permissions:
+      # IMPORTANT: this permission is mandatory for trusted publishing
+      id-token: write
+
+    # Dedicated environments with protections for publishing are strongly recommended.
+    environment:
+      name: pypi
+      # OPTIONAL: uncomment and update to include your PyPI project URL in the deployment status:
+      url: https://pypi.org/p/urielplus/
+
+    steps:
+      - name: Retrieve release distributions
+        uses: actions/download-artifact@v4
+        with:
+          name: release-dists
+          path: dist/
+
+      - name: Publish release distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

MANIFEST.in

@@ -0,0 +1,2 @@
+include urielplus/database/urielplus_csvs/*.csv
+include urielplus/database/original_uriel/*.npz

README.md

@@ -1,26 +1,161 @@
-URIEL+:Expanding Feature Coverage and Improving Usability of URIEL
-======
-URIEL+ is expanded upon the original URIEL knowledge base from the paper [(Littell et al., EACL 2017)](https://aclanthology.org/E17-2002/), focusing on describing languages distances through typological vectors, which has been cited over 200+ times. This expansion addresses previous limitaions in feature coverage and usability, particularly focusing on improving support for low-resource languages.
-
-Key Features
------------
-1. **Improved Language Coverage:** URIEL+ integrates five additional databases, including Grambank, BDPROTO, APiCS, and eWAVE, significantly enhancing its typological feature coverage and adds new orphological data for nearly 2500 languages. 
-2.   **Advanced Imputation Method:** alongside the original kNN imputation, URIEL+ provided interface for MIDASpy and SoftImpute, providing more accurate imputed data for missing values.
-3.   **Customizable Fetaure Selection:** users can choose or exclude specific features when calculating linguistic distances, allowing for tailed analyses depending on your usecase.
-4.   **Improved Usability:** Instead do precomputed distances, URIEL+ computes distances dynamically ensuring they reflect the most current data. Each calculated distance is also accompnaied by a confidence score, helping users assess the reliability of the results.
-
-Applications
-------------
-URIEL+ has evaluated across several downstream tasks, including performance prdiction(PerfPred, ProxyLM), transfer language selection (LangRank), and typological feature-driven language analysis (LinguAlchemy), where it always demonstrates a performance on par to URIEL, if not better.
-
-
-
-Installation
-------------
-The data are store in an `npz` file format, which comes out to be larger than github's size limit. Hence you will have to manually run the `update_ALL()` function in URIEL to First clone the repository and run `setup.py`.
-~~~
-git clone https://github.com/Masonshipton25/URIELPlusPlus
-cd URIELPlusPlus
-python3 setup.py install
-~~~
-Important Note: Upon cloning this repo locally, then immediately under your repository root, make sure to download glottolog v5.0 from https://zenodo.org/records/10804357
+# [URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Base](https://arxiv.org/abs/2409.18472)
+
+![knowledge base for natural language processing](./logo.png)
+
+URIEL is a knowledge base offering geographical, phylogenetic, and typological vector representations for 7970 languages. It includes distance measures between these vectors for 4005 languages, which are accessible via the lang2vec tool. Despite being frequently cited, URIEL is limited in terms of linguistic inclusion and overall usability. To tackle these challenges, we introduce URIEL+, an enhanced version of URIEL and lang2vec addressing these limitations. In addition to expanding typological feature coverage for 2898 languages, URIEL+ improves user experience with robust, customizable distance calculations to better suit the needs of the users. These upgrades also offer competitive performance on downstream tasks and provide distances that better align with linguistic distance studies.
+
+If you are interested for more information, check out our [full paper](https://arxiv.org/abs/2409.18472).
+
+## Contents
+
++ [Environment](#environment)
++ [Setup Instruction](#setup-instruction)
++ [Configuration Options Examples](#configuration-options-examples)
++ [Retrieving Loaded Features Examples](#retrieving-loaded-features-examples)
++ [Database Integration Examples](#database-integration-examples)
++ [Imputation Examples](#imputation-examples)
++ [Language Distance Calculation Examples](#language-distance-calculation-examples)
++ [Citation](#citation)
+
+## Environment
+
+Python 3.10.4 or higher. Details of dependencies are in `requirements.txt`.
+
+## Setup Instruction
+
++ To get started with URIEL+:
+    ```bash
+    pip install urielplus
+    ```
+
+    ```python
+    from urielplus.urielplus import URIELPlus
+
+    u = URIELPlus()
+    ```
+
+## Configuration Options Examples
+
++ URIEL+ offers various configurations that you can adjust:
+    - Caching: Enable or disable caching (True or False).
+    - Aggregation Method: Choose the method for aggregating data across sources ('U' for unweighted, 'A' for weighted).
+    - Fill Missing Data: Decide whether to fill missing data using parent language data (True or False).
+    - Distance Metric: Specify the distance metric to be used ("angular" or "cosine").
+
++ Changing A Configuration:
+    ```python
+    u.set_{configuration}({option})
+    ```
+
++ Checking A Configuration:
+    ```python
+    u.get_{configuration}({option})
+    ```
+
++ Replace `{configuration}` with `cache`, `aggregation`, `fill_with_base_lang`, or `distance_metric`.
++ Replace `{option}` with your desired value for the selected configuration.
++ Note: the default configurations are `cache=False`, `aggregation='U'`, `fill_with_base_lang=True`, and `distance_metric="angular"`.
+
+## Retrieving Loaded Features Examples
+
++ Retrieving A Loaded Feature:
+    ```python
+    u.get_{vector_type}_{feature_type}_array()
+    ```
++ Replace `{vector_type}` with `phylogeny`, `typological`, or `geography`.
++ Replace `{feature_type}` with `features`, `languages`, `data`, or `sources`.
+
++ Example:
+    ```python
+    u.get_typological_languages_array()
+    ```
+
+## Database Integration Examples
+
++ Integrating One Database:
+    ```python
+    u.integrate_{database}()
+    ```
++ Integrating Some Databases:
+    ```python
+    u.integrate_custom_databases({databases})
+    ```
++ Integrating All Databases:
+    ```python
+    u.integrate_databases()
+    ```
++ Set Language Codes to Glottocodes:
+    ```python
+    u.set_glottocodes()
+    ```
++ Reset all changes:
+    ```python
+    u.reset()
+    ```
+
++ Replace `{database}` with `saphon`, `bdproto`, `grambank`, `apics`, or `ewave`.
++ Replace `{databases}` with arguments `"UPDATED_SAPHON"`, `"BDPROTO"`, `"GRAMBANK"`, `"APICS"`, and/or `"EWAVE"` (e.g., `"UPDATED_SAPHON"`, `"BDPROTO"`, `"EWAVE"`).
+
+## Imputation Examples
+
++ Aggregate Typological Data:
+    ```python
+    u.set_aggregation({aggregation}) 
+    u.aggregate()
+    ```
+
++ Impute Missing Values:
+    ```python
+    u.{imputation_strategy}_imputation()
+    ```
+
++ Replace `{aggregation}` with `'U'` (union) or `'A'` (average).
++ Replace `{imputation_strategy}` with `midaspy`, `knn`, `softimpute`, or `mean`.
+
+## Language Distance Calculation Examples
+
++ Calculate a Specific Distance:
+    ```python
+    print(u.new_distance({distance_type}, {languages}))
+    ```
+
++ Calculate Distance Using Specific Features:
+    ```python
+    print(u.new_custom_distance({features}, {languages}, {source}))
+    ```
+
++ Retrieve Language Vectors:
+    ```python
+    u.get_vector({distance_type}, {languages})
+    ```
+
++ View URIEL+ Feature Coverage:
+    ```python
+    u.feature_coverage()
+    ```
+
++ Calculate Confidence Scores for Distances
+    ```python
+    print(u.confidence_score({language 1}, {language 2}, {distance_type}))
+    ```
+
++ Replace `{distance_type}` with a distance type (e.g., `"featural"`) or a list (e.g., `["syntactic"`, `"phonological"]`). Must be single distance type for retrieving language vectors.
++ Replace `{features}` with a list of features (e.g., `["F_Germanic", "S_SVO", "P_NASAL_VOWELS"]`).
++ Replace `{languages}`, `{language 1}`, and `{language 2}` with language codes (e.g., `"stan1293"`, `"hind1269"`).
++ Replace `{source}` with one database (e.g., `"WALS"`) or all databases (`'A'`).
++ Note: the default `{source}` is all databases.
+
+## Citation
+
+<u>If you use this code for your research, please cite the following work:</u>
+
+```bibtex
+@article{khan2024urielplus,
+  title={URIEL+: Enhancing Linguistic Inclusion and Usability in a Typological and Multilingual Knowledge Base},
+  author={Khan, Aditya and Shipton, Mason and Anugraha, David and Duan, Kaiyao and Hoang, Phuong H. and Khiu, Eric and Doğruöz, A. Seza and Lee, En-Shiun Annie},
+  journal={arXiv preprint arXiv:2409.18472},
+  year={2024}
+}
+```
+
+If you have any questions, you can open a [GitHub Issue](https://github.com/Masonshipton25/URIELPlus/issues) or send us an [email](mailto:masonshipton25@gmail.com).