Skip to content

Add support for gpu selection for users with multiple gpus#231

Merged
Glat0s merged 1 commit into
VisoMasterFusion:devfrom
RngDevil:dev_v3_gpu_select
May 12, 2026
Merged

Add support for gpu selection for users with multiple gpus#231
Glat0s merged 1 commit into
VisoMasterFusion:devfrom
RngDevil:dev_v3_gpu_select

Conversation

@RngDevil
Copy link
Copy Markdown
Contributor

@RngDevil RngDevil commented May 9, 2026

Purpose

For users that have more than one GPU (e.g. Laptop with RTX 4060 8GB - gpu-id 0 and eGPU-RTX 5060 Ti 16GB - gpu-id 1) and would like to select the more performant GPU (e.g. gpu-id 1)

Changes

  1. Add optional command line parameter --gpu-id <GPU_ID> to select the GPU. Defaults to 0 for backwards compatibility
  2. Changes hardcoding of device_id=0 to device_type=device_type and device id=gpu_id where appropriate
  3. Tested on 4060 and 5060 using TensorRT

@RngDevil
Copy link
Copy Markdown
Contributor Author

RngDevil commented May 9, 2026

Note, when changing to different GPUs, the cached tensorrt-engines may need to be deleted or at least renamed

@Glat0s
Copy link
Copy Markdown
Contributor

Glat0s commented May 10, 2026

Review against current upstream/dev for merge safety:

This PR is not safe to merge into dev as-is. Please repost to dev branch.

  1. Merge conflict against dev

    The PR is currently based on main, and GitHub reports it clean for that base, but merging the PR head into current upstream/dev conflicts in:

    • app/processors/workers/frame_worker.py

    Repro:

    git fetch upstream dev pull/231/head:review-pr-231
    git merge-tree upstream/dev review-pr-231

    Result includes:

    CONFLICT (content): Merge conflict in app/processors/workers/frame_worker.py
    

    Fix: rebase/retarget this PR onto the latest dev branch and resolve the frame_worker.py conflict, then rerun the processor tests.

  2. CPU provider can receive an invalid I/O binding device id when --gpu-id is nonzero

    ModelsProcessor.switch_providers_priority() keeps self.gpu_id unchanged when switching to CPU, while many ONNX Runtime I/O bindings now pass device_type=self.models_processor.device_type together with device_id=self.models_processor.gpu_id. Example paths include app/processors/face_masks.py where CPU mode would bind device_type=cpu and device_id=<gpu id>.

    That is risky because CPU I/O bindings should use device id 0; running with --gpu-id 1 and then selecting CPU can pass device_id=1 to CPU bindings.

    Fix: centralize the binding device id, for example:

    @property
    def binding_device_id(self) -> int:
        return self.gpu_id if self.device_type != cpu else 0

    Then use self.models_processor.binding_device_id for all ORT bind_input() / bind_output() calls. Alternatively, update gpu_id-based binding sites to use 0 whenever device_type == cpu.

Validation performed:

@RngDevil RngDevil changed the base branch from main to dev May 11, 2026 07:17
@RngDevil RngDevil force-pushed the dev_v3_gpu_select branch from 5e85195 to 10c0c00 Compare May 11, 2026 08:26
@RngDevil
Copy link
Copy Markdown
Contributor Author

Changes have been rebased to dev branch, resolved the conflict and resolved the CPU provider can receive an invalid I/O binding device id when --gpu-id is nonzero issue. I've also tested this locally on my system to ensure it is still functional

@Glat0s Glat0s added the minor New feature, backwards compatible label May 12, 2026
@Glat0s
Copy link
Copy Markdown
Contributor

Glat0s commented May 12, 2026

Would be nice to have the gpu selection options in the ui also. We can do this later. I'll merge for now.

@Glat0s Glat0s merged commit 0098691 into VisoMasterFusion:dev May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

minor New feature, backwards compatible

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants