I'm a Toronto Raptors fan, and one thing that's always bothered me about basketball highlights on TikTok, Reels and Shorts is how often the crop loses the action. The ball goes toward the basket, but the video ends up showing empty floor, the crowd or a player on the bench.
I wanted to see whether I could do better.
I recently added basketball support to Clips Kitty, my free, open-source Python application for turning long videos into short-form clips. The new basketball pipeline is available in version 2.1.
Demo: https://youtu.be/g1dMqrq-YaI
Source code: https://github.com/ColinGPT9/clips-studio
The problem with cropping basketball broadcasts
A standard basketball broadcast is 16:9, while a full-height vertical video is 9:16. You lose a large portion of the original frame, and the important objects are constantly moving: the ball, the players and the rim.
My first version used the generic COCO YOLOv8n model to detect the basketball. It wasn't good enough for this task.
I sampled 256 frames from 32 clips in the 2026 NBA Finals highlights package. Of those, 172 had a visible ball. I had the frames labelled by Claude vision agents so I could compare the output against the actual ball and rim positions.
In the rendered vertical clips, the first version kept the ball inside the crop in just 85 of 172 evaluated frames: 49%.
What I changed
Instead of building a detector from scratch, I integrated E-BARD's basketball-trained YOLOv8n detection model. The weights are used unmodified under CC BY 4.0.
That improved basketball detection, but raw detections alone weren't enough. At a confidence threshold of 0.10, false positives were still a problem.
The pipeline combines several steps:
- Scoreboard OCR: Reads changes in the broadcast scoreboard to identify scoring moments.
- Basketball detection: Uses E-BARD to detect the ball, players and hoop.
-
Camera-motion compensation: Uses OpenCV's
cv2.phaseCorrelateto estimate camera movement before linking ball candidates into tracks. - False-positive filtering and tracking: Filters candidate detections using size, shape and position, then links plausible ball trajectories within each camera shot.
- Crop planning: Plans a crop path for the entire clip, accounting for passes, shots, the ball's flight and the rim. It pans rather than continuously zooming in and out.
The important lesson was that improving the detector and improving the crop planner solve different parts of the problem.
The code for the framing implementation is here:
https://github.com/ColinGPT9/clips-studio/tree/main/sports/basketball/crop
Measuring the finished video
I measured the rendered 9:16 clips, rather than relying only on the detector's raw output.
The results from the first version compared with the current version were:
- Ball inside the crop: 85 of 172 frames (49%) in the first version, compared with 162 of 172 (94%) now.
- Rim visible as the ball arrived: 20 of 47 moments (43%) in the first version, compared with 46 of 47 (98%) now.
- Backboard's outer edge visible: 15 of 47 moments (32%) in the first version, compared with 45 of 47 (96%) now.
These results use the same clip windows and the same full-screen 9:16 aspect ratio for both versions.
The detection benchmark also showed that E-BARD's input resolution matters. On the same test, ball presence reached 87.8% at 1280 px, compared with 86.0% at 960 px. At 1920 px, it dropped to 64.0%.
For the 960 px detection test, a ball counted as found when a box at confidence 0.10 landed sufficiently close to the labelled ball. This is a presence-based metric, not the same as the E-BARD model card's stricter recall measurement.
What still doesn't work
This is a small evaluation: 256 frames from one broadcaster's Finals highlights package. It isn't a general benchmark across basketball footage.
The pipeline still struggles with some close-ups, free throws, replays and broadcasts with closer camera angles. In a small test of 35 labelled balls from a closer team-network broadcast, ball detection was 54% at 960 px and 80% at 640 px.
I added logic to use a smaller input size when the camera is closer, but there's more work to do.
The project is open source, so I'd especially appreciate feedback from people working on sports computer vision, object tracking and camera-aware video cropping.
How would you handle false ball detections in the stands or keep a crop stable through rapid camera pans and cuts?
Try it
Clips Kitty is free, runs locally on Windows and is available under the AGPL-3.0 licence.
- Source code: https://github.com/ColinGPT9/clips-studio
- Basketball framing code: https://github.com/ColinGPT9/clips-studio/tree/main/sports/basketball/crop
- E-BARD detector: https://github.com/GabrieleGiudic/E-BARD
I'm not affiliated with or endorsed by the Toronto Raptors or the NBA.
AI disclosure: AI assistance was used in preparing this post. The implementation, benchmark decisions and reported measurements are described from my development notes.
