As Kubernetes continues to dominate container orchestration, understanding how its scheduler works becomes crucial for any DevOps engineer.This hands-on lab demonstrates the inner workings of Kubernetes scheduling and walks through creating a custom scheduler from scratch.
Understanding the Kubernetes Scheduler
Before diving into the terminal, it's important to understand the component at play. The Kube-Scheduler serves as the brain behind pod placement in Kubernetes. It follows a three-phase process:
Filtering - Eliminates nodes that don't meet requirements
Scoring - Ranks suitable nodes based on various factors
Binding - Creates the connection between pod and chosen node
The scheduler considers CPU, memory, node affinity, taints, tolerations, and various other constraints to make intelligent placement decisions.
Lab Setup: Creating the First Scheduled Pod
The lab begins by templating an nginx pod that will later be targeted with a custom scheduler:
kubectl run nginx --image=nginx -o yaml --dry-run=client | tee nginx_scheduler.yaml
This command generates a pod manifest without actually creating the pod. The --dry-run=client flag proves perfect for templating purposes.
Exploring the schedulerName Option
Kubernetes provides a powerful but lesser-known feature: the ability to specify which scheduler should handle a pod. The lab explores this capability:
kubectl explain pod.spec | more
Among the various pod specification options, schedulerName appears - a string field that allows targeting a specific scheduler. This becomes the gateway to custom scheduling logic.
Configuring a Custom Scheduler
The next step involves updating the nginx pod manifest to use a custom scheduler called my-scheduler:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: nginx
name: nginx
spec:
schedulerName: my-scheduler
containers:
- image: nginx name: nginx resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} EOF
The key addition here is schedulerName: my-scheduler under the spec section.
Figure 2: The nginx_scheduler.yaml file with schedulerName specified
Applying the Configuration
The lab proceeds with applying this configuration:
kubectl apply -f nginx_scheduler.yaml
Checking the pod status reveals an interesting result:
kubectl get pods -o wide
Observation: The pod remains stuck in Pending state. This occurs because there's no scheduler called my-scheduler running in the cluster. The pod waits patiently for its designated scheduler to pick it up.
Figure 1: The nginx pod remains in Pending state as it waits for the my-scheduler
Building the Custom Scheduler
While production schedulers are typically written in Golang and deeply integrated with the Kubernetes API, this lab uses a bash script for educational purposes. This approach helps demonstrate the scheduler's workflow without getting lost in complex code.
First, the necessary tools are installed:
apt update && apt install -y git jq
Next, the example scheduler repository is cloned:
git clone https://github.com/spurin/simple-kubernetes-scheduler-example.git
The lab then moves into the directory and examines the scheduler script:
cd simple-kubernetes-scheduler-example
more my-scheduler.sh
Understanding the Scheduler Logic
The script performs several key operations:
Queries available nodes using jsonpath to extract node names
Identifies pods waiting for my-scheduler
Selects a node (randomly in this simple example)
Makes a binding request to the API server
To understand how the JSON traversal works, the raw node data can be examined:
kubectl get nodes -o json
This reveals the structure the script navigates through - items as a list, each with metadata and a name field.
Running the Custom Scheduler
The exciting moment arrives when the custom scheduler is executed:
./my-scheduler.sh
The script picks up the pending pod and binds it to a node. After pressing Ctrl+C, the pod status is checked again:
kubectl get pods -o wide
Success! The pod is now running on a specific node, scheduled by the custom scheduler.
Figure 3: The my-scheduler.sh script in action, binding the pod to a node
Figure 4: The nginx pod now shows as Running with a node assignment
The Power of nodeName: Bypassing the Scheduler
Kubernetes also provides a way to bypass scheduling altogether using the nodeName field. This directly assigns a pod to a specific node without any scheduler involvement.
The lab first returns to the previous directory:
cd ..
The options for nodeName under pod.spec can be reviewed:
kubectl explain pod.spec | more
Direct Node Assignment with nodeName
The yaml file is then updated to directly specify worker-2 as the target node:
cat < nginx_scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: nginx
name: nginx
spec:
nodeName: worker-2
containers:
- image: nginx name: nginx resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} EOF
The configuration is applied:
kubectl apply -f nginx_scheduler.yaml
Checking the pod status confirms it has been scheduled directly to worker-2:
kubectl get pods -o wide
Result: The pod appears on worker-2 immediately, bypassing the scheduler entirely. This demonstrates how nodeName provides a direct path from pod specification to node assignment.
Figure 5: The pod scheduled directly to worker-2 using nodeName specification
The pod is removed for the next demonstration:
kubectl delete pod/nginx --now
nodeSelector: Label-Based Scheduling (Further Study)
Another approach to pod placement involves using nodeSelector, which leverages node labels for targeting specific nodes. This method provides more flexibility than nodeName while still maintaining control over placement.
Examining Node Labels
The worker-1 node is examined to view its labels:
kubectl describe node/worker-1 | more
The labels section at the top reveals various metadata, including kubernetes.io/hostname=worker-1. This label becomes the selector for targeting this specific node.
Figure 6: Examining the labels on worker-1 node
Figure 7: The kubernetes.io/hostname label that will be used in nodeSelector
Configuring nodeSelector
The yaml file is updated to use nodeSelector with the hostname label:
cat < nginx_scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: nginx
name: nginx
spec:
nodeSelector:
kubernetes.io/hostname: worker-1
containers:
- image: nginx name: nginx resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} EOF
The update is applied:
kubectl apply -f nginx_scheduler.yaml
Verification shows the pod has been scheduled to worker-1:
kubectl get pods -o wide
Observation: Unlike nodeName, nodeSelector still goes through the scheduler but constrains the scheduling decision to nodes matching the specified labels. This approach offers more flexibility and is considered a best practice for most use cases.
Figure 8: The nodeSelector configuration using the hostname label
Cleanup
Finally, all lab resources are cleaned up:
kubectl delete pod/nginx --now
rm -rf simple-kubernetes-scheduler-example
rm -rf nginx_scheduler.yaml
Key Takeaways
Through this lab, several important concepts are demonstrated:
Scheduler workflow: Filtering → Scoring → Binding
Custom schedulers: How to create and target custom scheduling logic
The binding process: How pods get assigned to nodes
Direct node assignment: Using nodeName to bypass scheduling entirely
Label-based scheduling: Using nodeSelector for flexible, constraint-based placement
Scheduling Methods Compared
The lab explored three distinct approaches to pod scheduling:
Custom Scheduler (schedulerName): Provides complete control over scheduling logic, ideal for specialized placement algorithms
Direct Assignment (nodeName): Bypasses the scheduler entirely, useful for debugging or specific fixed placements
Label-Based Selection (nodeSelector): Combines scheduler intelligence with constraint-based node selection, offering the best balance for most production scenarios
When Would One Use This?
Custom schedulers prove valuable when:
Specialized placement logic is needed for specific workloads
Default scheduler constraints don't meet requirements
Advanced placement strategies are being implemented (GPU scheduling, network topology awareness, etc.)
Conclusion
Understanding Kubernetes scheduling unlocks a deeper level of cluster control. While the default scheduler handles most scenarios effectively, knowing how to implement custom scheduling provides the flexibility to optimize for specific use cases.
The ability to bypass or customize scheduling demonstrates Kubernetes' modular design philosophy - providing sensible defaults while allowing deep customization when needed.
This lab successfully demonstrates that the scheduler's "magic" is actually a well-designed, understandable process that can be customized to meet unique requirements.
What's Next?
Having mastered Kubernetes scheduling mechanisms, the next topic in this series will explore Kubernetes Storage - another critical component for running stateful workloads in production environments. 🚀
For those interested in diving deeper into Kubernetes internals, this hands-on approach provides an excellent foundation for understanding how the platform orchestrates containerized workloads at scale.
