Try loading entire ThermoML archive

[mattwthompson]

The tests in #48 are relatively minimal and might not cover the production use case of loading all ThermoML data

[lilyminium]

So, this took me about an hour, which is much faster than the Evaluator one (although not directly comparable as Evaluator downloads from the internet and I read from a pre-downloaded XML). However, where Evaluator reads ~780k data points, dimsim only reads ~416k. I'll raise a couple issues I noticed.

[mattwthompson]

Is this "the" source for the entire archive?

https://data.nist.gov/od/ds/mds2-2422/ThermoML.v2020-09-30.tgz

[lilyminium]

Yeah, that's what I've been using.

[mattwthompson]

Do you have any sort of pre-filtering from that data before running it through this code? Naively I can't get this off the ground, whereas you can; would it be possible for you to share a script or some sort of instructions for me to pick this up from where you're at? I more or less want to use this as a stress-test to figure out #65 and other issues

In [1]: import glob

In [2]: from dimsim.datasets.thermoml import ThermoMLDataSet

In [3]: ThermoMLDataSet.from_file(*glob.glob("ThermoML.v2020-09-30/10.*/*xml"))
WARNING:root:An error occurred while parsing a compound: The OpenFF Toolkit does not currently support parsing molecules with radicals. Found 2 radical electrons on molecule [Sn].
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 ThermoMLDataSet.from_file(*glob.glob("ThermoML.v2020-09-30/10.*/*xml"))

File ~/software/dimsim/dimsim/datasets/thermoml.py:2030, in ThermoMLDataSet.from_file(cls, *file_list)
   2027 counter = 0
   2029 for file in file_list:
-> 2030     data_set = cls._from_file(file)
   2032     counter += 1
   2034     if data_set is None or len(data_set) == 0:

File ~/software/dimsim/dimsim/datasets/thermoml.py:2063, in ThermoMLDataSet._from_file(cls, path)
   2061 try:
   2062     with open(path) as file:
-> 2063         return_value = ThermoMLDataSet.from_xml(file.read(), source)
   2065 except FileNotFoundError:
   2066     logging.warning(f"No ThermoML file could not be found at {path}")

File ~/software/dimsim/dimsim/datasets/thermoml.py:2134, in ThermoMLDataSet.from_xml(cls, xml, source)
   2131 properties = _PureOrMixtureData.from_xml_node(node, namespace, compounds)
   2133 if properties is None or len(properties) == 0:
-> 2134     raise Exception(
   2135         f"No properties parsed from {node=}, {namespace=}, {compounds=}. "
   2136         "This likely means the entry was skipped due to an unsupported phase / property "
   2137         "type, or a failure to parse the compound information."
   2138     )
   2140 for measured_property in properties:
   2141     unit_to_use = _property_unit_map[measured_property.type_string]

Exception: No properties parsed from node=<Element '{http://www.iupac.org/namespaces/ThermoML}PureOrMixtureData' at 0x171daee80>, namespace={'ThermoML': 'http://www.iupac.org/namespaces/ThermoML'}, compounds={1: Compound(smiles=[Au], index=1), 2: Compound(smiles=[Ag], index=2)}. This likely means the entry was skipped due to an unsupported phase / property type, or a failure to parse the compound information.

[lilyminium]

Oh, odd ... without parallelization I just try/excepted each file:

frames = []
for xml_path in tqdm.tqdm(xml_files, desc="Parsing"):
    try:
        dataset = ThermoMLDataSet.from_file(str(xml_path))
        frames.append(dataset.to_pandas())
    except Exception as e:
        print(e)
combined = pd.concat(frames, ignore_index=True)

This gets me a lot of errors like the below:

'A property with the unique id 67635509507832367918305829797918629821201385666684235294969469544637271848949 already exists.'
'A property with the unique id 47702629559491577435469797477242603826000878328689268286863987881591154458526 already exists.'
'A property with the unique id 43056396951018924135316980135232357732302473660570700944902329230313280419691 already exists.'
'A property with the unique id 42431741109325428790779706228542816406173023628211598155934435013009882544488 already exists.'
'A property with the unique id 10461007887118407316963084525143480559089411748486717899434317943061744290814 already exists.'
'A property with the unique id 109145994443468345911331129171049671198205988638099985980311280571138473640542 already exists.'
'A property with the unique id 50332888706297163912140931754468404989577624523608630718677027019317506421373 already exists.'
'A property with the unique id 39576352531898734644514311080853231142717578036778966967211618062420524553227 already exists.'
'A property with the unique id 113334287672468553961667757324634860553313082015762790542190650519026343822325 already exists.'

also the typical failures to parse a molecule:

Warning: Failed to parse name: 1-BUTYL-5-ETHYL-2-METHYLPYRIDINIUM BIS(TRIFLUOROMETHYLSULFONYL)AMIDE
Warning: Failed to parse name: 1-BUTYL-2,3,5-TRIMETHYLPYRIDINIUM BIS(TRIFLUOROMETHYLSULFONYL)AMIDE
Warning: Failed to parse name: 2,3-DIMETHYL-1-OCTYLPYRIDINIUM BIS(TRIFLUOROMETHYLSULFONYL)AMIDE
Warning: Failed to parse name: 5-ETHYL-2-METHYL-1-OCTYLPYRIDINIUM BIS(TRIFLUOROMETHYLSULFONYL)AMIDE
Warning: Failed to parse name: 2,3,5-TRIMETHYL-1-OCTYLPYRIDINIUM BIS(TRIFLUOROMETHYLSULFONYL)AMIDE
Warning: Failed to parse name: 1-BUTYL-2,3-DIMETHYLPYRIDINIUM DICYANAMIDE
Warning: Failed to parse name: 5-ETHYL-2-METHYL-1-OCTYLPYRIDINIUM DICYANAMIDE

However, nothing like your error pops up in my log.

(Note: without parallelization, it is much slower -- I multiprocessed to use 8 workers for better comparison to the Evaluator method).

[lilyminium]

Ahh I looked closer at your code. I hadn't realized that from_file actually accepts a list of files -- it looks like that's probably the difference. I was reading files one by one. A fair few of them do show up as empty for me too, either because they contain properties that dimsim can't parse or because of the identical ID error.

(having a progress bar by processing per-file also is very helpful).

[mattwthompson]

Splitting a few possible issues into sub-issues