Support custom schedulers

[saurav-agarwalla]

Description

What problem are you trying to solve?
Feature request for aws/karpenter-provider-aws#7265.

Observed Behavior: we have deployed a custom scheduler in our EKS cluster that keeps in view of some extra paramaters while scheduling pod on a node, it would stop scheduling on a node if there are specific number of pods already in a RUNNING state on that node. so it may happen that a node has required capacity to schedule a pod but the custom-scheduler won't schedule on it.

eg: karpenter giving this log on a pending pod, Pod should schedule on: nodeclaim/shaarif-nodepool-fccsj, node/ip-10-0-253-116.ap-southeast-1.compute.internal this is the status of the pending pod

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-22T11:31:06Z"
    message: '0/33 nodes are available: 32 node(s) didn''t match Pod''s node affinity/selector.
      preemption: 0/33 nodes are available: 1 No preemption victims found for incoming
      pod, 32 Preemption is not helpful for scheduling.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: BestEffort

since the node has the capacity to schedule a pod but the custom scheduler filters out that node from scheduling as per the custom logic. as a consequence the pod stays in pending state for eternity. Expected Behavior: karpenter should spin up a node for a pod if it is marked unschedulable by any scheduler.

Reproduction Steps (Please include YAML): "Deploy a custom scheduler with a scheduler extender to filter out potential nodes, considering additional constraints."

Versions: karpenter version : 0.34.0 k8 release: 1.29.8 custom-kube-scheduler image: v1.25.12

* Chart Version:

* Kubernetes Version (`kubectl version`): v1.29.8-eks-a737599


* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request

* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

* If you are interested in working on this issue or have submitted a pull request, please leave a comment

How important is this feature to you?

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[saurav-agarwalla]

/triage needs-investigation