Support for Azure storage by tramora · Pull Request #571 · KhiopsML/khiops-python

tramora · 2026-04-10T11:36:53Z

Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects)
The only supported authentication method is currently the AZURE_STORAGE_CONNECTION_STRING embedding the account name and account key

Completes #244

TODO Before Asking for a Review

Rebase your branch to the latest version of main (or main-v10)
Make sure all CI workflows are green
When adding a public feature/fix: Update the Unreleased section of CHANGELOG.md (no date)
Self-Review: Review "Files Changed" tab and fix any problems you find
API Docs (only if there are changes in docstrings, rst files or samples):
- Check the docs build without warning: see the log of the API Docs workflow
- Check that your changes render well in HTML: download the API Docs artifact and open index.html
- If there are any problems it is faster to iterate by building locally the API Docs

tramora · 2026-04-13T08:49:36Z

The automated tests on conda environments currently fail because of an outdate version of the azure-storage-file-share package in the conda-forge channel. This PR conda-forge/azure-storage-file-share-feedstock#9 should fix the issue but it is blocked for the moment.

folmos-at-orange · 2026-04-14T08:43:19Z

+                try:
+                    os.remove("/tmp/dummy")
+                except:  # pylint: disable=broad-exception-caught
+                    pass
                fs.copy_to_local(
-                    f"{proto}://{bucket_name}/khiops-cicd/samples/{file}", "/tmp/dummy"
+                    f"{proto}://{bucket_name_or_storage_prefix}/khiops-cicd/"
+                    f"samples/{file}",
+                    "/tmp/dummy",
                )
+                # We cannot use `unittest.TestCase.assertTrue` in a class method
+                assert os.path.isfile("/tmp/dummy")


I don't get why this new code was added

If it stays I'd refactor it as:

Before the loop create /tmp/khiops-dummy

Save each file to /tmp/khiops-dummy/{file}

Do not use assert as it is not a programming error, I think it should be something like if not os.path.isfile(...): raise RuntimeError

Clean /tmp/khiops-dummy afterwards
This avoids the try/except with pass in the middle.

During the development phase when fs.copy_to_local was an empty method, the test would pass even if the file download was not successful. This part had to be improved to assess the file is correctly downloaded (without any error and with the local file creation).

Following your suggestions I improved a little bit the snippet

folmos-at-orange · 2026-04-14T08:50:45Z

-    def should_skip_in_a_conda_env(self):
-        # The S3 driver is now released for conda too.
-        # No need to skip the tests any longer in a conda environment
-        return False
-
    def config_exists(self):


I think these elimination should go in a different commit.

folmos-at-orange · 2026-04-14T08:53:42Z

+class AzureStorageResourceMixin:
+    """Azure compatible Storage Resource Mixin


I think all these docs should go in the classes AzureStorage*Resource.

In the docstring of this class they should point to those, or it could be duplicated.

Also I think we should add an URL to a relevant Azure storage page.

- Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects) - The only supported authentication method is currently the `AZURE_STORAGE_CONNECTION_STRING` embedding the account name and account key

…cess.py

folmos-at-orange

I think this is an opportunity to have better docs on file system.

Currently, very little is documented, and we now have 3 different remote systems of which only one (S3) is well documented.

I'd a section for each remote system in the module's docstring. Each section contains how to configure it in each case, and a URI example.

folmos-at-orange · 2026-04-16T07:08:43Z

+
+
+class AzureStorageResourceMixin:
+    """Azure compatible Storage Resource Mixin"""


I'd add in the body, See `AzureStorageFileResource` and `AzureBlobResource` for more details.

folmos-at-orange · 2026-04-16T07:15:59Z

+        """
+        Check if the target resource exists (file or directory)
+        """


Move this to the first docstring line to keep the codebase style.

folmos-at-orange · 2026-04-16T07:17:14Z

+
+    def remove(self):
+        """
+        Remove the target resource either it is a file or a directory.


Move this to the first docstring line to keep the codebase style.

folmos-at-orange · 2026-04-16T07:17:55Z

+
+    def list_dir(self):
+        """
+        List the files of the current directory


Move this to the first docstring line to keep the codebase style.

tramora force-pushed the 244-support-azure-storage branch 3 times, most recently from be97119 to 08f91e2 Compare April 10, 2026 14:34

tramora requested review from folmos-at-orange and popescu-v April 10, 2026 15:00

tramora force-pushed the 244-support-azure-storage branch 4 times, most recently from 17fe956 to 80691c3 Compare April 12, 2026 12:59

tramora force-pushed the 244-support-azure-storage branch from 80691c3 to 701678b Compare April 13, 2026 16:32

folmos-at-orange requested changes Apr 14, 2026

View reviewed changes

Thierry RAMORASOAVINA added 2 commits April 15, 2026 18:38

Support for Azure storage

b193f7e

- Among all the Azure storages, Khiops supports only (via its specific driver) "Files" and "Blobs" (Binary Large Objects) - The only supported authentication method is currently the `AZURE_STORAGE_CONNECTION_STRING` embedding the account name and account key

Remove a useless skip condition against a conda env in test_remote_ac…

19aa231

…cess.py

tramora force-pushed the 244-support-azure-storage branch from 701678b to 19aa231 Compare April 15, 2026 16:39

tramora requested a review from folmos-at-orange April 15, 2026 16:47

folmos-at-orange approved these changes Apr 16, 2026

View reviewed changes

		class AzureStorageResourceMixin:
		"""Azure compatible Storage Resource Mixin

Conversation

tramora commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO Before Asking for a Review

Uh oh!

tramora commented Apr 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

folmos-at-orange left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tramora commented Apr 10, 2026 •

edited

Loading