Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Workflow Interface]: Prints on running a flow with ray backend are coming in a previous notebook cell #1411

Open
scngupta-dsp opened this issue Feb 28, 2025 · 2 comments

Comments

@scngupta-dsp
Copy link
Contributor

Issue Description:
When using backend=ray, the prints on executing flflow.run are appearing in a previous notebook cell. This is quite strange and inconvenient to user

To Reproduce:
Run openfl-tutorials/experimental/workflow/102_Aggregator_Validation.ipynb

Logs: Refer snapshot below

Image

Desktop:

  • OS: Ubuntu (WSL)
  • OS Version: 22.04.3 LTS
  • Python: 3.11

Versions:

aiosignal 1.3.2
ansicolors 1.1.8
astroid 3.3.8
asttokens 3.0.0
astunparse 1.6.3
attrs 25.1.0
bcrypt 4.3.0
boto3 1.37.3
botocore 1.37.3
certifi 2025.1.31
cffi 1.17.1
chardet 5.2.0
charset-normalizer 3.4.1
click 8.1.8
cloudpickle 3.1.1
comm 0.2.2
contourpy 1.3.1
cryptography 44.0.1
cycler 0.12.1
debugpy 1.8.12
decorator 5.2.1
defusedxml 0.7.1
dill 0.3.6
docker 7.1.0
dynaconf 3.2.10
entrypoints 0.4
exceptiongroup 1.2.2
execnb 0.1.11
executing 2.2.0
fastcore 1.7.29
fastjsonschema 2.21.1
filelock 3.17.0
flatten-json 0.1.14
fonttools 4.56.0
fpdf 1.7.2
frozenlist 1.5.0
fsspec 2025.2.0
ghapi 1.0.6
grpcio 1.65.5
idna 3.10
importlib_metadata 8.6.1
iniconfig 2.0.0
ipykernel 6.29.5
ipython 8.32.0
ipywidgets 8.1.5
isort 6.0.1
jedi 0.19.2
Jinja2 3.1.5
jmespath 1.0.1
joblib 1.4.2
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
jupyter_client 8.6.3
jupyter_core 5.7.2
jupyterlab_widgets 3.0.13
kiwisolver 1.4.8
lxml 5.3.1
markdown-it-py 3.0.0
MarkupSafe 3.0.2
matplotlib 3.10.0
matplotlib-inline 0.1.7
mccabe 0.7.0
mdurl 0.1.2
metaflow 2.7.15
mpmath 1.3.0
msgpack 1.1.0
nbclient 0.10.2
nbdev 2.3.12
nbformat 5.10.4
nest_asyncio 1.6.0
networkx 3.4.2
numpy 2.2.3
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.8.61
nvidia-nvtx-cu12 12.1.105
openfl 1.8.0.dev0
packaging 24.2
pandas 2.2.3
papermill 2.6.0
paramiko 3.5.1
parso 0.8.4
pexpect 4.9.0
pickleshare 0.7.5
pillow 11.1.0
pip 25.0.1
platformdirs 4.3.6
pluggy 1.5.0
prompt_toolkit 3.0.50
protobuf 5.29.3
psutil 7.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
pycparser 2.22
pycryptodome 3.21.0
Pygments 2.19.1
pylint 3.3.4
PyNaCl 1.5.0
pyparsing 3.2.1
pytest 8.3.4
pytest-asyncio 0.25.3
pytest-mock 3.14.0
python-dateutil 2.9.0.post0
pytz 2025.1
PyYAML 6.0.2
pyzmq 26.2.1
ray 2.9.2
referencing 0.36.2
requests 2.32.3
rich 13.9.4
rpds-py 0.23.1
s3transfer 0.11.3
scikit-learn 1.6.1
scipy 1.15.2
setuptools 75.8.2
six 1.17.0
stack_data 0.6.3
sympy 1.13.3
tabulate 0.9.0
tenacity 9.0.0
tensorboardX 2.6.2.2
threadpoolctl 3.5.0
tomlkit 0.13.2
torch 2.3.1
torchvision 0.18.1
tornado 6.4.2
tqdm 4.67.1
traitlets 5.14.3
triton 2.3.1
typing_extensions 4.12.2
tzdata 2025.1
urllib3 2.3.0
watchdog 6.0.0
wcwidth 0.2.13
wheel 0.45.1
widgetsnbextension 4.0.13
zipp 3.21.0

@scngupta-dsp
Copy link
Contributor Author

Able to reproduce this issue with a minimal example - logs below. From this snapshot we can see

  • The logs from a ray future are being printed in the cell where ray was initialized
  • This is the same behavior observed in Workflow Interface tutorial where the logs are printed in the cell where LocalRuntime is instantiated

Image

@scngupta-dsp
Copy link
Contributor Author

Additional Information:
Issue is not observed after downgrading ipykernel to 6.26.0. Reference issue logged: ipython/ipykernel#1370

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants