Unit tests fail on dev branch

[sfluegel05]

For some weeks now, our unittests are failing when triggered automatically by a pull request. The first run I could identify is https://github.com/ChEB-AI/python-chebai/actions/runs/13131519464 (triggered by ad24fa7).

I don't see any major change in the code that could have caused this. @aditya0by0 could you have a closer look?

[aditya0by0]

I believe that not only the unit tests but also other GitHub Actions are failing (even in the dev branch), as seen in the tagged PR #76. While the unit tests pass successfully on my local machine, they fail in the GitHub Actions workflow. This issue might be related to recent changes in GitHub Actions or some temporary issue related with it.
I will investigate further into it.

[aditya0by0]

Unit Test Cases Failure Reason: Changes in pysmiles v2.0.0 Affecting Tokenization

The pysmiles package has been released with a direct version jump from v1.1.2 to v2.0.0 Click here to see the difference between to versions. Since we haven’t specified a version constraint, GitHub Actions is now installing the latest version by default.

One notable change in v2.0.0 is the modification of the _tokenize function, which we use in ChemDataReader and DeepChemDataReader. This appears to be causing different token generation compared to v1.1.2, as evidenced by the failure of related test cases (TestDeepChemDataReader and TestChemDataReader) on the new version, while they pass on the old version.

Options to Resolve This

1. Pin pysmiles to v1.1.2: This ensures token consistency with previous models trained on tokens from v1.1.2.
2. Upgrade to v2.0.0: This may introduce new tokenization behavior....

@sfluegel05, Please let me know how to proceed.

[sfluegel05]

For context: I had a closer look at pysmiles and it turns out that the output of _tokenize changed to a 3-tuple such as (<TokenType.ATOM: 1>, 6, 'C'),
We are using _tokenize(...)[1] as our token, which is now a running index (here: 6) and not the actual token (here: C). So the unit test caught an error that would break the tokenisation completely.

[aditya0by0]

For context: I had a closer look at pysmiles and it turns out that the output of _tokenize changed to a 3-tuple such as (<TokenType.ATOM: 1>, 6, 'C'), We are using _tokenize(...)[1] as our token, which is now a running index (here: 6) and not the actual token (here: C). So the unit test caught an error that would break the tokenisation completely.

@sfluegel05
If the tokenization process remains fundamentally unchanged, would it be possible to upgrade to the latest version of pysmiles? Since the unit test correctly identified the issue due to the updated _tokenize output format, adjusting our code to reference the correct token index should resolve the problem. This approach would help ensure compatibility with the latest version while maintaining correct tokenization behavior.

We could modify reader.py to ensure that users with an outdated version of pysmiles encounter an ImportError when running ChebAI.

import pysmiles
import pkg_resources

required_version = "2.0.0"

if pkg_resources.parse_version(pysmiles.__version__) < pkg_resources.parse_version(required_version):
    raise ImportError(
        f"Incompatible pysmiles version detected: {pysmiles.__version__}. "
        f"ChebAI requires pysmiles {required_version} or higher. "
        f"Please upgrade using: pip install --upgrade pysmiles"
    )

This ensures that existing users with an older version will be notified immediately when running ChebAI, preventing unexpected errors during execution.

Please let me know your thoughts on this.