[Workflow Interface] Issues observed in running Workflow API tutorials with Python 3.12

[scngupta-dsp]

Description
Issues observed while running a Workflow Interface tutorial with virtual environment created with Python 3.12

To Reproduce
Steps to reproduce the behavior:

1. Create a virtual environment with Python 3.12
2. Run the Workflow Interface Tutorial (for e.g. openfl-tutorials/experimental/workflow/101_MNIST.ipynb)
3. The tutorial fails with errors

Expected behavior
The tutorial should run successfully without issues

Logs and Snapshots:

Installation of Workflow API pre-requisites
!pip install -r workflow_interface_requirements.txt

ERROR: Could not find a version that satisfies the requirement ray==2.9.2 (from versions: 2.31.0, 2.32.0rc0, 2.32.0, 2.33.0, 2.34.0, 2.35.0, 2.36.0, 2.36.1, 2.37.0, 2.38.0, 2.39.0, 2.40.0, 2.41.0, 2.42.0, 2.42.1, 2.43.0)
ERROR: No matching distribution found for ray==2.9.2

After upgrading ray

---

ModuleNotFoundError Traceback (most recent call last)
Cell In[4], line 3
1 from copy import deepcopy
----> 3 from openfl.experimental.workflow.interface import FLSpec, Aggregator, Collaborator
4 from openfl.experimental.workflow.runtime import LocalRuntime
5 from openfl.experimental.workflow.placement import aggregator, collaborator

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/openfl/experimental/workflow/interface/init.py:7
1 # Copyright 2020-2024 Intel Corporation
2 # SPDX-License-Identifier: Apache-2.0
5 """openfl.experimental.workflow.interface package."""
----> 7 from openfl.experimental.workflow.interface.fl_spec import FLSpec
8 from openfl.experimental.workflow.interface.participants import Aggregator, Collaborator

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/openfl/experimental/workflow/interface/fl_spec.py:16
13 if TYPE_CHECKING:
14 from openfl.experimental.workflow.runtime import FederatedRuntime, LocalRuntime, Runtime
---> 16 from openfl.experimental.workflow.utilities import (
17 MetaflowInterface,
18 SerializationError,
19 aggregator_to_collaborator,
20 checkpoint,
21 collaborator_to_aggregator,
22 filter_attributes,
23 generate_artifacts,
24 should_transfer,
25 )
28 class FLSpec:
29 """FLSpec Class
30
31 A class representing a Federated Learning Specification. It manages clones,
(...)
39 _runtime (RuntimeType): The runtime of the flow.
40 """

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/openfl/experimental/workflow/utilities/init.py:12
5 """openfl.experimental.workflow.utilities package."""
7 from openfl.experimental.workflow.utilities.exceptions import (
8 ResourcesAllocationError,
9 ResourcesNotAvailableError,
10 SerializationError,
11 )
---> 12 from openfl.experimental.workflow.utilities.metaflow_utils import MetaflowInterface
13 from openfl.experimental.workflow.utilities.resources import get_number_of_gpus
14 from openfl.experimental.workflow.utilities.runtime_utils import (
15 check_resource_allocation,
16 checkpoint,
(...)
19 parse_attrs,
20 )

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/openfl/experimental/workflow/utilities/metaflow_utils.py:23
21 from dill.source import getsource # nosec
22 # from metaflow.datastore import DATASTORES, FlowDataStore
---> 23 from metaflow.datastore import FlowDataStore
24 from metaflow.plugins.datastores.local_storage import LocalStorage
25 from metaflow.plugins import DATASTORES

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/metaflow/init.py:114
111 from .datatools import S3
113 # includefile
--> 114 from .includefile import IncludeFile
116 # Decorators
117 from .decorators import step, _import_plugin_decorators

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/metaflow/includefile.py:50
48 # From here on out, this is the IncludeFile implementation.
49 from .datatools import Local, S3
---> 50 from metaflow.plugins.azure.includefile_support import Azure
51 from metaflow.plugins.gcp.includefile_support import GS
53 DATACLIENTS = {
54 "local": Local,
55 "s3": S3,
56 "azure": Azure,
57 "gs": GS,
58 }

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/metaflow/plugins/init.py:115
113 from .retry_decorator import RetryDecorator
114 from .resources_decorator import ResourcesDecorator
--> 115 from .aws.batch.batch_decorator import BatchDecorator
116 from .kubernetes.kubernetes_decorator import KubernetesDecorator
117 from .argo.argo_workflows_decorator import ArgoWorkflowsInternalDecorator

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/metaflow/plugins/aws/batch/batch_decorator.py:8
5 import time
7 from metaflow import util
----> 8 from metaflow import R, current
10 from metaflow.decorators import StepDecorator
11 from metaflow.plugins import ResourcesDecorator

File ~/miniforge-pypy3/envs/env_openfl_latest_3.12/lib/python3.12/site-packages/metaflow/R.py:2
1 import os
----> 2 import imp
3 from tempfile import NamedTemporaryFile
5 from .util import to_bytes

ModuleNotFoundError: No module named 'imp'

'

Desktop (please complete the following information):

• OS: Ubuntu (WSL)
• OS Version: 22.04.3 LTS
• Python: 3.12

Additional context
Further issues observed after updating metaflow version

[teoparvanov]

If I understand correctly, the main issue is this:

ERROR: Could not find a version that satisfies the requirement ray==2.9.2 (from versions: 2.31.0, 2.32.0rc0, 2.32.0, 2.33.0, 2.34.0, 2.35.0, 2.36.0, 2.36.1, 2.37.0, 2.38.0, 2.39.0, 2.40.0, 2.41.0, 2.42.0, 2.42.1, 2.43.0)
ERROR: No matching distribution found for ray==2.9.2

Have you tried upgrading pip in your python 3.12 virtual environment?

pip install --upgrade pip

[scngupta-dsp]

Upgrading pip does not help with this particular issue. My understanding is that ray version 2.9.2 (openfl-tutorials/experimental/workflow/workflow_interface_requirements.txt) is not compatible with Python 3.12

Environment: Python 3.10: ray version 2.9.2 is available

(env_ray_debug_python3.10) scngupta@soc-9TTR9K3:~$ python --version
Python 3.10.16
(env_ray_debug_python3.10) scngupta@soc-9TTR9K3:~$ pip index versions ray
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
ray (2.43.0)
Available versions: 2.43.0, 2.42.1, 2.42.0, 2.41.0, 2.40.0, 2.39.0, 2.38.0, 2.37.0, 2.36.1, 2.36.0, 2.35.0, 2.34.0, 2.33.0, 2.32.0, 2.31.0, 2.30.0, 2.24.0, 2.23.0, 2.22.0, 2.21.0, 2.20.0, 2.12.0, 2.11.0, 2.10.0, 2.9.3, 2.9.2, 2.9.1, 2.9.0, 2.8.1, 2.8.0, 2.7.2, 2.7.1, 2.7.0, 2.6.3, 2.6.2, 2.6.1, 2.6.0, 2.5.1, 2.5.0, 2.4.0, 2.3.1, 2.3.0, 2.2.0, 2.1.0, 2.0.1, 2.0.0, 1.13.0
(env_ray_debug_python3.10) scngupta@soc-9TTR9K3:~$

Environment: Python 3.11: ray version 2.9.2 is available

(env_ray_debug_python3.11) scngupta@soc-9TTR9K3:~$ python --version
Python 3.11.11
(env_ray_debug_python3.11) scngupta@soc-9TTR9K3:~$ pip index versions ray
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
ray (2.43.0)
Available versions: 2.43.0, 2.42.1, 2.42.0, 2.41.0, 2.40.0, 2.39.0, 2.38.0, 2.37.0, 2.36.1, 2.36.0, 2.35.0, 2.34.0, 2.33.0, 2.32.0, 2.31.0, 2.30.0, 2.24.0, 2.23.0, 2.22.0, 2.21.0, 2.20.0, 2.12.0, 2.11.0, 2.10.0, 2.9.3, 2.9.2, 2.9.1, 2.9.0, 2.8.1, 2.8.0, 2.7.2, 2.7.1, 2.7.0, 2.6.3, 2.6.2, 2.6.1, 2.6.0, 2.5.1, 2.5.0, 2.4.0, 2.3.1, 2.3.0
(env_ray_debug_python3.11) scngupta@soc-9TTR9K3:~$

Environment: Python 3.12: ray version 2.9.2 is NOT available

(env_ray_debug) scngupta@soc-9TTR9K3:~$ python --version
Python 3.12.9
(env_ray_debug) scngupta@soc-9TTR9K3:~$ pip index versions ray
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
ray (2.43.0)
Available versions: 2.43.0, 2.42.1, 2.42.0, 2.41.0, 2.40.0, 2.39.0, 2.38.0, 2.37.0, 2.36.1, 2.36.0, 2.35.0, 2.34.0, 2.33.0, 2.32.0, 2.31.0
(env_ray_debug) scngupta@soc-9TTR9K3:~$

This issue can be solved by upgrading ray to latest version (2.43.0). However this is not the main issue

As mentioned in issue description, after upgrading ray - there is another compatibility issue observed with metaflow version (with Python 3.12). This issue is caused since imp module has been removed in Python 3.12 (Ref: https://docs.python.org/3/whatsnew/3.12.html#imp)

Upgrading to latest versions of metaflow is not a simple task due to modifications in openfl/experimental/workflow/utilities/metaflow_utils.py