Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Utility functions to core #2406

Draft
wants to merge 9 commits into
base: develop
Choose a base branch
from

Conversation

egede
Copy link
Member

@egede egede commented Mar 13, 2025

No description provided.

@egede egede temporarily deployed to Integrate Pull Request March 13, 2025 00:21 — with GitHub Actions Inactive
@egede egede marked this pull request as draft March 13, 2025 00:22
@egede
Copy link
Member Author

egede commented Mar 13, 2025

@Ishaan-Chandak I just created a PR from your branch as it makes it easier to comment on the code.

Copy link
Member Author

@egede egede left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try to implement the changes suggested here. If that then removes the warnings, then the entire code base should be changed to use the new import location.

It would be good to write test cases for all the different utility functions as well. If you decide to do this, the tests should utilise the mock system to avoid actually submitting and running any jobs as part of the tests.

@egede
Copy link
Member Author

egede commented Mar 13, 2025

Another comment is that we should make sure that the old import location still works but emits a deprecation warning. While the code is not formally part of the Ganga API, it will have a number of external users.

@Ishaan-Chandak Ishaan-Chandak temporarily deployed to Integrate Pull Request March 13, 2025 04:56 — with GitHub Actions Inactive
@@ -10,6 +10,7 @@
import GangaCore.Core
from GangaCore.Core.GangaRepository import getRegistry
from GangaCore.Core.InternalServices.ShutdownManager import _protected_ganga_exitfuncs
from ganga.GangaCore.Utility.Config.Config import ConfigError
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is ConfigError ever used? If not, we can delete this line.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no it was not being used,
I have deleted this

@Ishaan-Chandak Ishaan-Chandak requested a deployment to Integrate Pull Request March 13, 2025 09:02 — with GitHub Actions Waiting
@egede
Copy link
Member Author

egede commented Mar 13, 2025

OK, so I guess the final thing that is required is to change all the imports in the testing code. I give a list below. I think a global search and replace with the new location should work.

GangaCore/old_test/Bugs/Savannah13462.gpi:from GangaTest.Framework.utils import failureException
GangaCore/old_test/Bugs/Savannah17638.gpi:from GangaTest.Framework.utils import sleep_until_completed,failureException
GangaCore/old_test/Bugs/Savannah18178/Savannah18178.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/Bugs/Savannah24452/Savannah24452.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/Bugs/Savannah27039.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/Bugs/Savannah28290.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/Bugs/Savannah32201.gpi:from GangaTest.Framework.utils import failureException
GangaCore/old_test/Bugs/Savannah36651.gpi:from GangaTest.Framework.utils import sleep_until_state,file_contains,failureException
GangaCore/old_test/Bugs/Savannah86537.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/GPI/AbortApp/AbortApp.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/GPI/BackendApplicationMatrix.gpim:from GangaTest.Framework.utils import sleep_until_state,failureException
GangaCore/old_test/GPI/JobKill.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/Mergers/TestRootMerger.py:from GangaTest.Framework.utils import sleep_until_completed, write_file
GangaCore/old_test/GPI/Mergers/TestTextMerger.py:from GangaTest.Framework.utils import (file_contains, sleep_until_completed,
GangaCore/old_test/GPI/Mergers/TestTextMergerGzip.py:from GangaTest.Framework.utils import sleep_until_completed, write_file
GangaCore/old_test/GPI/OutputFiles/OutputFilesBatch.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesBatchMultipleFilesForLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesBatchMultipleFilesForMassStorageAndLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesBatchMultipleFilesForMassStorageUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesCompressedBatch.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesCompressedLocal.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesInteractive.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesInteractiveMultipleCompressedFilesForMassStorageAndLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesInteractiveMultipleFilesForLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesInteractiveMultipleFilesForMassStorageAndLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesInteractiveMultipleFilesForMassStorageUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesInteractiveMultipleFilesForPattern.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLCG.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLCGMultipleCompressedFilesForMassStorageAndLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLocal.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLocalMultipleCompressedFilesForMassStorageAndLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLocalMultipleFilesForLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLocalMultipleFilesForMassStorageAndLCGUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLocalMultipleFilesForMassStorageUpload.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesLocalMultipleFilesForPattern.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/OutputFiles/OutputFilesMixedBatch.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/PreparedState.gpi:#from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/Resubmit.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/ResubmitBackend.gpi:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/old_test/GPI/ResubmitSubjobs/ResubmitSubjobs.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/GPI/Sandbox/InputOnly.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/GPI/Sandbox/InputOutput.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/GPI/Sandbox/OutputOnly.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/GPI/Templates.gpi:from GangaTest.Framework.utils import assert_cannot_submit
GangaCore/old_test/GPI/TestSplitting/SubjobRTprepareFail.gpi:from GangaTest.Framework.utils import assert_cannot_submit
GangaCore/old_test/GPI/TestSplitting/SubjobSubmitFail.gpi:from GangaTest.Framework.utils import assert_cannot_submit
GangaCore/old_test/GPI/auto_resubmit/auto_resubmit.gpi:from GangaTest.Framework.utils import sleep_until_state
GangaCore/old_test/Internals/TestTransitions.py:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Bugs/TestSavannah15630.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Bugs/TestSavannah19059.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Bugs/TestSavannah19123.py:        from GangaTest.Framework.utils import sleep_until_state
GangaCore/test/GPI/Bugs/TestSavannah44116.py:        from GangaTest.Framework.utils import sleep_until_state
GangaCore/test/GPI/Bugs/TestSavannah47814.py:        from GangaTest.Framework.utils import sleep_until_state, file_contains
GangaCore/test/GPI/Bugs/TestSavannah77962.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Bugs/TestSavannah8009.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Checkers/TestCustomChecker.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/FileTests/TestLocalFileClient.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/FileTests/TestMassStorageClient.py:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/FileTests/TestMassStorageInput.py:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/FileTests/TestMassStorageWN.py:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/FileTests/TestSJIDName.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/FileTests/TestSJIDName.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/FileTests/TestSJIDName.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Loading/TestLazyLoadingSubjobs.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Mergers/TestCustomMerger.py:from GangaTest.Framework.utils import write_file
GangaCore/test/GPI/Mergers/TestMergeFailures.py:from GangaTest.Framework.utils import sleep_until_state
GangaCore/test/GPI/Mergers/TestSmartMerger.py:from GangaTest.Framework.utils import file_contains, write_file
GangaCore/test/GPI/Mergers/TestStdOut.py:        from GangaTest.Framework.utils import write_file
GangaCore/test/GPI/Mergers/TestStructure.py:        from GangaTest.Framework.utils import write_file
GangaCore/test/GPI/Notebook/Test_Notebook.py:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/Parser/TestSelect.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestArgSplitter.py:        from GangaTest.Framework.utils import sleep_until_completed, file_contains
GangaCore/test/GPI/TestReport.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestSJSubmit.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestSJSubmit.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestSJSubmit.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestSJSubmit.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestSJSubmit.py:        from GangaTest.Framework.utils import (sleep_until_completed,
GangaCore/test/GPI/TestSplitter.py:    from GangaTest.Framework.utils import assert_cannot_submit
GangaCore/test/GPI/TestSplitter.py:    from GangaTest.Framework.utils import assert_cannot_submit
GangaCore/test/GPI/TestSubjobs.py:        from GangaTest.Framework.utils import sleep_until_state
GangaCore/test/GPI/TestSubjobs.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestTimeStamps.py:from GangaTest.Framework.utils import sleep_until_completed, sleep_until_state
GangaCore/test/GPI/TestTimeStamps.py:    from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestTimeStamps.py:    from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestTimeStamps.py:    from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/newXMLTest/TestSJXMLGenAndLoad.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/newXMLTest/TestXMLGenAndLoad.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/newXMLTest/TestXMLGenAndLoad.py:                from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestInteractive.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/GPI/TestLocalCleanenv.py:        from GangaTest.Framework.utils import sleep_until_completed, file_contains
GangaCore/test/Unit/Virtualization/TestVirtualizationDocker.py:from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/gangaDB/db_testing/TestGangaDBGenAndLoad.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaCore/test/gangaDB/db_testing/TestGangaDBSJGenAndLoad.py:        from GangaTest.Framework.utils import sleep_until_completed
GangaDirac/old_test/GPI/Files/TestDiracFile.py:# GangaTest.Framework.utils defines some utility methods
GangaDirac/old_test/GPI/Files/TestDiracFile.py:from GangaTest.Framework.utils import sleep_until_completed, sleep_until_state
GangaDirac/old_test/Lib/RTHandlers/TestExeDiracRTHandler.py:# GangaTest.Framework.utils defines some utility methods
GangaDirac/old_test/Lib/RTHandlers/TestExeDiracRTHandler.py:#from GangaTest.Framework.utils import sleep_until_completed,sleep_until_state
GangaDirac/old_test/Lib/Server/TestWorkerThreadPool.py:# GangaTest.Framework.utils defines some utility methods
GangaDirac/old_test/Lib/Server/TestWorkerThreadPool.py:# from GangaTest.Framework.utils import file_contains#,
GangaTest/Framework/tests.py:from GangaTest.Framework.utils import is_job_completed
GangaTest/TestGPIP/TestGPIP.gpip:        from GangaTest.Framework.utils import is_job_finished

Comment on lines 4 to 9
warnings.warn(
"The module 'ganga.GangaTest.Framework.util' is deprecated and will be removed in a future release. "
"Please use 'ganga.GangaCore.Utility.job_monitoring' instead.",
DeprecationWarning,
stacklevel=2
)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you check that this is actually working

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I did test this. I tested the read_file functionality. I created a python script and a dummy read file and checked by importing the read_file function from GangaTest.framework.utils.
It did work correctly.

@Ishaan-Chandak Ishaan-Chandak deployed to Integrate Pull Request March 14, 2025 15:03 — with GitHub Actions Active
@Ishaan-Chandak Ishaan-Chandak requested a deployment to Integrate Pull Request March 20, 2025 06:34 — with GitHub Actions Waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants