Skip to content

Test image classifier flow - #52

Draft
abhi-agg wants to merge 6 commits into
mainfrom
test-image-classifier
Draft

Test image classifier flow#52
abhi-agg wants to merge 6 commits into
mainfrom
test-image-classifier

Conversation

@abhi-agg

@abhi-agg abhi-agg commented Oct 23, 2025

Copy link
Copy Markdown
Collaborator

Goal(s) of this Pull Request:

Try to reproduce and fix an issue that a client reported.

Example of how it works:

...

Acceptance criteria:

In order to approve, reviewer must be able to...

  • ...

@abhi-agg

abhi-agg commented Oct 23, 2025

Copy link
Copy Markdown
Collaborator Author

With ad95978, the flow failed. Details are as follows:

Python version: 3.11.9

Perimeter: default

Outerbound dependency versions:

$ pip freeze | grep outerbounds
metaflow-checkpoint==0.2.4
ob-metaflow==2.15.21.1
ob-metaflow-extensions==1.1.175
ob-metaflow-stubs==6.0.3.188
outerbounds==0.3.188

Command to run the flow:

WANDB_API_KEY=<my-personal-api-key> WANDB_PROJECT=example-image-classifier python image_classifier_flow.py --environment=pypi run --offline False

Console logs

Metaflow 2.15.21.1+obcheckpoint(0.2.4);ob(v1) executing ImageClassifierFlow for user:aaggarwal@mozilla.com
Run all steps on @nvct
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2025-10-23 11:27:18.716 Creating local datastore in current directory (/Users/aaggarwal/work/mozmlops/examples/image_classifier/.metaflow)
2025-10-23 11:27:19.543 Bootstrapping virtual environment(s) ...
    Micromamba ran into an error while setting up environment:
    command '/Users/aaggarwal/.metaflowconfig/micromamba/bin/micromamba create --yes --no-deps --download-only --safety-checks=disabled --no-extra-safety-checks --repodata-ttl=86400 --prefix=/var/folders/17/6m926lb52cb0x8v3bcq9kmww0000gn/T/tmpga7oldgq/prefix --quiet https://conda.anaconda.org/conda-forge/osx-arm64/bzip2-1.0.8-hd037594_8.conda https://conda.anaconda.org/conda-forge/osx-arm64/libffi-3.5.2-he5f378a_0.conda https://conda.anaconda.org/conda-forge/osx-arm64/libzlib-1.3.1-h8359307_2.conda https://conda.anaconda.org/conda-forge/osx-arm64/liblzma-5.8.1-h39f12f2_2.conda https://conda.anaconda.org/conda-forge/osx-arm64/libexpat-2.7.1-hec049ff_0.conda https://conda.anaconda.org/conda-forge/osx-arm64/ncurses-6.5-h5e97a16_3.conda https://conda.anaconda.org/conda-forge/osx-arm64/icu-75.1-hfee45f7_0.conda https://conda.anaconda.org/conda-forge/osx-arm64/tk-8.6.13-h892fb3f_2.conda https://conda.anaconda.org/conda-forge/osx-arm64/xz-tools-5.8.1-h39f12f2_2.conda https://conda.anaconda.org/conda-forge/osx-arm64/xz-gpl-tools-5.8.1-h9a6d368_2.conda https://conda.anaconda.org/conda-forge/osx-arm64/liblzma-devel-5.8.1-h39f12f2_2.conda https://conda.anaconda.org/conda-forge/osx-arm64/readline-8.2-h1d1bf99_2.conda https://conda.anaconda.org/conda-forge/osx-arm64/libsqlite-3.50.4-h4237e3c_0.conda https://conda.anaconda.org/conda-forge/osx-arm64/xz-5.8.1-h9a6d368_2.conda https://conda.anaconda.org/conda-forge/noarch/tzdata-2025b-h78e105d_0.conda https://conda.anaconda.org/conda-forge/noarch/ca-certificates-2025.10.5-hbd8a1cb_0.conda https://conda.anaconda.org/conda-forge/osx-arm64/openssl-3.5.4-h5503f6c_0.conda https://conda.anaconda.org/conda-forge/osx-arm64/python-3.11.9-h932a869_0_cpython.conda https://conda.anaconda.org/conda-forge/noarch/wheel-0.45.1-pyhd8ed1ab_1.conda https://conda.anaconda.org/conda-forge/noarch/setuptools-80.9.0-pyhff2d567_0.conda https://conda.anaconda.org/conda-forge/noarch/pip-25.2-pyh8b19718_0.conda' returned error (1)

@abhi-agg

Copy link
Copy Markdown
Collaborator Author

Re-ran With ad95978 one more time and now the flow starts but fails in the start step.

Main error seems to be: ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

Detailed console logs:

Metaflow 2.15.21.1+obcheckpoint(0.2.4);ob(v1) executing ImageClassifierFlow for user:aaggarwal@mozilla.com
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2025-10-23 12:25:44.307 Bootstrapping virtual environment(s) ...
2025-10-23 12:26:03.119 Virtual environment(s) bootstrapped!
2025-10-23 12:26:05.451 Workflow starting (run-id 1232), see it in the UI at https://ui.desertowl.obp.outerbounds.com/p/default/ImageClassifierFlow/1232
2025-10-23 12:26:20.456 [1232/start/4935 (pid 15377)] Task is starting.
2025-10-23 12:26:29.562 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Task is starting (QUEUED)...
2025-10-23 12:26:47.337 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Task is starting (RUNNING)...
2025-10-23 12:26:47.552 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Setting up task environment.
2025-10-23 12:26:51.627 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Downloading code package...
2025-10-23 12:26:54.759 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Code package downloaded.
2025-10-23 12:26:55.918 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Task is starting.
2025-10-23 12:26:59.051 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Bootstrapping virtual environment...
2025-10-23 12:28:06.797 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Environment bootstrapped.
2025-10-23 12:28:20.553 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] <flow ImageClassifierFlow step start> failed:
2025-10-23 12:28:25.414 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     Internal error
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] Traceback (most recent call last):
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/cli.py", line 631, in main
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     start(auto_envvar_prefix="METAFLOW", obj=state)
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     return self.main(args, kwargs)
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/_vendor/click/core.py", line 782, in main
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     rv = self.invoke(ctx)
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]          ^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/cli_components/utils.py", line 69, in invoke
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     return _process_result(sub_ctx.command.invoke(sub_ctx))
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.417 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     return ctx.invoke(self.callback, ctx.params)
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     return callback(args, kwargs)
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/tracing/__init__.py", line 27, in wrapper_func
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     return func(args, kwargs)
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]            ^^^^^^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     return f(get_current_context(), args, kwargs)
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/cli_components/step_cmd.py", line 167, in step
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     task.run_step(
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/task.py", line 672, in run_step
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     self._exec_step_function(step_func)
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/metaflow/task.py", line 62, in _exec_step_function
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     step_function()
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/image_classifier_flow.py", line 42, in start
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     import torchvision
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/linux-64/b245e3bc8e2126e/lib/python3.11/site-packages/torchvision/__init__.py", line 5, in <module>
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     import torch
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]   File "/home/jovyan/metaflow/linux-64/b245e3bc8e2126e/lib/python3.11/site-packages/torch/__init__.py", line 290, in <module>
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     from torch._C import   # noqa: F403
2025-10-23 12:34:45.884 [1232/start/4935 (pid 15377)] Nvct task failed:
2025-10-23 12:34:45.887 1 task is running: start (1 running; 0 done).
2025-10-23 12:34:45.887 No tasks are waiting in the queue.
2025-10-23 12:34:45.887 4 steps have not started: end, train, evaluate, upload_model_to_gcs.
2025-10-23 12:34:46.543 [1232/start/4935 (pid 15377)] Task failed with status: ERRORED. This could be a transient error. Use @retry to retry.
2025-10-23 12:34:46.543 [1232/start/4935 (pid 15377)]
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]     ^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc] ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory
2025-10-23 12:28:25.418 [1232/start/4935 (pid 15377)] [ee049b15-73d9-418b-943e-6bc515e2b6fc]
2025-10-23 12:34:50.611 [1232/start/4935 (pid 15377)] Task failed.
2025-10-23 12:34:53.282 Workflow failed.
2025-10-23 12:34:53.283 Terminating 0 active tasks...
2025-10-23 12:34:53.283 Flushing logs...
    Step failure:
    Step start (task-id 4935) failed.

…kage

 - use wandb package explicitly in flow where needed
 - disabled uploading model to gcs
 - install outerbounds package instead of mozmlops via requirements file
@abhi-agg

abhi-agg commented Oct 23, 2025

Copy link
Copy Markdown
Collaborator Author

With 7807e30, I did following:

  1. Stop using mozmlops dependency in the flow
  2. Install outerbounds[gcp] latest version on my machine instead of install mozmlops

to see whether mozmlops package is an issue (I will not be able to upload artifacts to gcp via the flow but that is fine).

The flow failed.

Details are as follows:

Local python version: 3.11.9

Perimeter: default

Locally installed outerbounds/metaflow dependencies:

$ pip freeze | grep "metaflow\|outerbounds"
metaflow-checkpoint==0.2.8
ob-metaflow==2.18.13.1
ob-metaflow-extensions==1.4.25
ob-metaflow-stubs==6.0.10.19
outerbounds==0.10.19

Flow run:

$ WANDB_API_KEY=<my-key-here> WANDB_PROJECT=example-image-classifier python image_classifier_flow.py --environment=pypi run --offline False
Metaflow 2.18.13.1+obcheckpoint(0.2.8);ob(v1) executing ImageClassifierFlow for user:aaggarwal@mozilla.com
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2025-10-23 13:23:46.694 Bootstrapping virtual environment(s) ...
    Pip ran into an error while setting up environment:
    Could not determine Python version from conda environment

@abhi-agg

Copy link
Copy Markdown
Collaborator Author

With 6e79000, I updated torch and torchvision dependency to latest versions.

The flow failed.

Main error: ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

Details are as follows:

$ WANDB_API_KEY=<my-key> WANDB_PROJECT=example-image-classifier python image_classifier_flow.py --environment=pypi run --offline False
Metaflow 2.18.13.1+obcheckpoint(0.2.8);ob(v1) executing ImageClassifierFlow for user:aaggarwal@mozilla.com
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2025-10-23 13:31:30.034 Bootstrapping virtual environment(s) ...
2025-10-23 13:52:22.289 Virtual environment(s) bootstrapped!
2025-10-23 13:52:23.636 Workflow starting (run-id 1237), see it in the UI at https://ui.desertowl.obp.outerbounds.com/p/default/ImageClassifierFlow/1237
2025-10-23 13:52:40.510 [1237/start/4958 (pid 23627)] Task is starting.
2025-10-23 13:52:49.772 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Task is starting (QUEUED)...
2025-10-23 13:54:35.371 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Task is starting (RUNNING)...
2025-10-23 13:53:09.840 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Setting up task environment.
2025-10-23 13:53:13.574 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Downloading code package...
2025-10-23 13:53:17.002 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Code package downloaded.
2025-10-23 13:53:17.901 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Task is starting.
2025-10-23 13:53:21.219 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Bootstrapping virtual environment...
2025-10-23 13:54:34.179 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Environment bootstrapped.
2025-10-23 13:54:52.913 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] <flow ImageClassifierFlow step start> failed:
2025-10-23 13:55:00.805 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     Internal error
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] Traceback (most recent call last):
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/cli.py", line 658, in main
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     start(auto_envvar_prefix="METAFLOW", obj=state)
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/_vendor/click/core.py", line 829, in __call__
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     return self.main(args, kwargs)
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/_vendor/click/core.py", line 782, in main
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     rv = self.invoke(ctx)
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]          ^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/cli_components/utils.py", line 69, in invoke
2025-10-23 13:55:00.809 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     return _process_result(sub_ctx.command.invoke(sub_ctx))
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/_vendor/click/core.py", line 1066, in invoke
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     return ctx.invoke(self.callback, ctx.params)
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/_vendor/click/core.py", line 610, in invoke
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     return callback(args, kwargs)
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]            ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/tracing/__init__.py", line 27, in wrapper_func
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     return func(args, kwargs)
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]            ^^^^^^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/_vendor/click/decorators.py", line 21, in new_func
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     return f(get_current_context(), args, kwargs)
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/cli_components/step_cmd.py", line 167, in step
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     task.run_step(
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/task.py", line 852, in run_step
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     self._exec_step_function(step_func, orig_step_func)
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/task.py", line 142, in _exec_step_function
2025-10-23 13:55:00.810 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     raise raised_exception
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/.mf_code/metaflow/task.py", line 108, in _exec_step_function
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     step_function()
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/image_classifier_flow.py", line 42, in start
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     import torchvision
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/linux-64/84b6a96f95ea5ff/lib/python3.11/site-packages/torchvision/__init__.py", line 5, in <module>
2025-10-23 13:59:18.225 [1237/start/4958 (pid 23627)] Nvct task failed:
2025-10-23 13:59:18.228 1 task is running: start (1 running; 0 done).
2025-10-23 13:59:18.228 No tasks are waiting in the queue.
2025-10-23 13:59:18.228 4 steps have not started: train, upload_model_to_gcs, evaluate, end.
2025-10-23 13:59:18.898 [1237/start/4958 (pid 23627)] Task failed with status: ERRORED. This could be a transient error. Use @retry to retry.
2025-10-23 13:59:18.899 [1237/start/4958 (pid 23627)]
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     import torch
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]   File "/home/jovyan/metaflow/linux-64/84b6a96f95ea5ff/lib/python3.11/site-packages/torch/__init__.py", line 427, in <module>
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     from torch._C import   # noqa: F403
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]     ^^^^^^^^^^^^^^^^^^^^^^
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421] ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory
2025-10-23 13:55:00.811 [1237/start/4958 (pid 23627)] [33f9b249-47d5-4d1c-8568-f1ff06965421]
2025-10-23 13:59:21.752 [1237/start/4958 (pid 23627)] Task failed.
2025-10-23 13:59:24.454 Workflow failed.
2025-10-23 13:59:24.454 Terminating 0 active tasks...
2025-10-23 13:59:24.454 Flushing logs...
    Step failure:
    Step start (task-id 4958) failed.

@abhi-agg

Copy link
Copy Markdown
Collaborator Author

Switched from @pypi to @conda decorator via 0034215 and the flow is still failing (running into different failure now).

Detailed logs are:

Metaflow 2.18.13.1+obcheckpoint(0.2.8);ob(v1) executing ImageClassifierFlow for user:aaggarwal@mozilla.com
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2025-10-24 14:49:42.382 Bootstrapping virtual environment(s) ...
    Micromamba ran into an error while setting up environment:
    command '/Users/aaggarwal/.metaflowconfig/micromamba/bin/micromamba create --yes --quiet --dry-run --no-extra-safety-checks --repodata-ttl=86400 --safety-checks=disabled --retry-clean-cache --prefix=/var/folders/17/6m926lb52cb0x8v3bcq9kmww0000gn/T/tmplbxbv4wg/prefix --channel=conda-forge requests>=2.21.0 google-cloud-storage>=2.5.0 google-auth>=2.11.0 google-cloud-secret-manager>=2.10.0 simple-gcp-object-downloader>=0.1.0 pytorch==2.8.0 torchvision==0.20.1 wandb==0.22.2 python==3.11.9' returned error (1)
    nothing provides __cuda needed by pytorch-2.5.1-cuda118_py310h8b36b8a_300

    Please set the environment variable CONDA_OVERRIDE_CUDA to a specific version of cuda.

    Here is an example of supplying environment variables through the command line
    CONDA_OVERRIDE_CUDA=<cuda-version> python flow.py <args>

@abhi-agg

Copy link
Copy Markdown
Collaborator Author

Taking inspiration from a working flow, switched to python 3.10(via 08b3b98) (previously was 3.11.9) and the last error still persists.

Detailed logs are:

$ METAFLOW_DEBUG_CONDA=1 WANDB_API_KEY=<my-key> WANDB_PROJECT=example-image-classifier python image_classifier_flow.py --environment=conda run --offline False
Metaflow 2.18.13.1+obcheckpoint(0.2.8);ob(v1) executing ImageClassifierFlow for user:aaggarwal@mozilla.com
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2025-10-24 15:27:33.680 Creating local datastore in current directory (/Users/aaggarwal/work/mozmlops/examples/image_classifier/.metaflow)
2025-10-24 15:27:36.795 Bootstrapping virtual environment(s) ...
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:218]: Solving packages for Conda environments..
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:106]: Solving packages for conda environment f87f4c65e15af88
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:106]: Solving packages for conda environment 2294e58e6bb1264
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:106]: Solving packages for conda environment 13d57
Switch to python version "3.10" in the flow
e29b283f19
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:106]: Solving packages for conda environment 1d82100d8dc80b8
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:106]: Solving packages for conda environment faa78f12fd4c415
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:234]: Downloading packages for Conda environment f87f4c65e15af88
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:172]: Downloading packages for conda environment f87f4c65e15af88
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:127]: Caching packages for conda environments ['f87f4c65e15af88']
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:234]: Downloading packages for Conda environment faa78f12fd4c415
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:184]: Caching 87 new packages to the datastore for conda environment ['f87f4c65e15af88']
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:172]: Downloading packages for conda environment faa78f12fd4c415
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:127]: Caching packages for conda environments ['faa78f12fd4c415']
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:234]: Downloading packages for Conda environment 13d57e29b283f19
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:184]: Caching 87 new packages to the datastore for conda environment ['faa78f12fd4c415']
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/micromamba.py:172]: Downloading packages for conda environment 13d57e29b283f19
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:127]: Caching packages for conda environments ['13d57e29b283f19']
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:184]: Caching 139 new packages to the datastore for conda environment ['13d57e29b283f19']
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:201]: Finished caching packages.
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:201]: Finished caching packages.
debug[conda /Users/aaggarwal/work/mozmlops/examples/image_classifier/env_example_imageclassifier/lib/python3.11/site-packages/metaflow/plugins/pypi/conda_environment.py:201]: Finished caching packages.
    Micromamba ran into an error while setting up environment:
    command '/Users/aaggarwal/.metaflowconfig/micromamba/bin/micromamba create --yes --quiet --dry-run --no-extra-safety-checks --repodata-ttl=86400 --safety-checks=disabled --retry-clean-cache --prefix=/var/folders/17/6m926lb52cb0x8v3bcq9kmww0000gn/T/tmpb4465pv1/prefix --channel=conda-forge requests>=2.21.0 google-cloud-storage>=2.5.0 google-auth>=2.11.0 google-cloud-secret-manager>=2.10.0 simple-gcp-object-downloader>=0.1.0 pytorch==2.8.0 torchvision==0.20.1 python==3.10' returned error (1)
    nothing provides __cuda needed by pytorch-2.5.1-cuda118_py310h8b36b8a_300

    Please set the environment variable CONDA_OVERRIDE_CUDA to a specific version of cuda.

    Here is an example of supplying environment variables through the command line
    CONDA_OVERRIDE_CUDA=<cuda-version> python flow.py <args>

@abhi-agg

Copy link
Copy Markdown
Collaborator Author

Taking inspiration from the same working flow, stopped specifying each step's dependent package versions (via 94d4a38) and the flow ran successfully.

This is the run https://ui.desertowl.obp.outerbounds.com/dashboard/runs/p/default/ImageClassifierFlow/1250

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant