fix(trainer): use writable generated script paths

[danish9039]

What this PR does / why we need it:

Writes non-MPI CustomTrainer generated Python files and package-installation logs under /tmp instead of the image working directory.

Restricted non-root containers can have a root-owned working directory, which prevents the SDK from creating its generated files. The generated script restores the image working directory as the first Python import root while preserving existing import paths, including Python safe-path mode. MPI continues to use DEFAULT_MPI_USER_HOME.

This is required by kubeflow/trainer#3702, which intentionally preserves each image's configured working directory.

Which issue(s) this PR fixes:

Follow-up to kubeflow/trainer#3702.

Validation:

• 127 focused Kubernetes backend tests
• Read-only working-directory execution in normal and safe-path modes
• Working-directory precedence over a same-named /tmp module
• Existing PYTHONPATH preservation
• Concurrent behavioral execution
• MPI path preservation
• Ruff lint and formatting checks
• git diff --check

Checklist:

• Documentation included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[lntutor]

Reviewed the generated-script path change and its behavioral coverage. The non-MPI path keeps the image working directory first for imports while writing generated files and install logs under /tmp; MPI retains its existing user-home behavior. The read-only, safe-path, shadowing, PYTHONPATH, concurrent, and MPI cases cover the relevant execution paths. Approved.

[google-oss-prow]

@lntutor: changing LGTM is restricted to collaborators

Details

In response to this:

Reviewed the generated-script path change and its behavioral coverage. The non-MPI path keeps the image working directory first for imports while writing generated files and install logs under /tmp; MPI retains its existing user-home behavior. The read-only, safe-path, shadowing, PYTHONPATH, concurrent, and MPI cases cover the relevant execution paths. Approved.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.