How AI Tracks Players in Real-Time: Computer Vision in Sports Broadcasting

The next time you watch a football match on TV and see a player’s heat map, a speed readout, or a passing accuracy stat appear in real time — that is computer vision at work. Until recently, this technology required a team of engineers and a rack of servers. Today, a single NVIDIA Jetson board can do it all at the edge. Here is how it works.

What does AI see when it watches a sports match?

A well-trained computer vision model watching a sports match can simultaneously track:

Every player’s position — bounding boxes updated 60 times per second
Body keypoints — 17 points per player including wrists, elbows, knees, ankles — used to classify actions
The ball — sub-pixel trajectory tracking using a Kalman filter to smooth movement between frames
Ball speed — calculated from trajectory using court/field homography to convert pixel movement to real metres per second
Shot/action classification — using the player’s body pose, the AI classifies whether they are serving, spiking, blocking, or passing
Team identification — jersey colour clustering assigns players to teams automatically

The technology stack

A complete sports AI system built on NVIDIA Jetson uses four key technologies:

1. YOLOv8-pose for player detection and keypoints

YOLOv8-pose is a specialised variant that detects both the bounding box and the 17-point body skeleton for every person in frame — simultaneously, in a single neural network pass. On Jetson Orin with TensorRT it achieves 40–60 FPS on a full 1080p frame.

from ultralytics import YOLO
model = YOLO("yolov8n-pose.engine")   # pose estimation model

results = model(frame)
for person in results[0].keypoints.data:
    # 17 keypoints: nose, eyes, ears, shoulders, elbows, wrists, hips, knees, ankles
    keypoints = person.cpu().numpy()  # shape: [17, 3] — x, y, confidence per point

2. Custom ball detection model

Sports balls are small, fast-moving, and often partially occluded. A custom YOLOv8 model trained specifically on ball images — combined with a Kalman filter that predicts the ball’s position between frames — achieves robust tracking even at high speeds.

from filterpy.kalman import KalmanFilter
import numpy as np

kf = KalmanFilter(dim_x=4, dim_z=2)
kf.F = np.array([[1,0,1,0],[0,1,0,1],[0,0,1,0],[0,0,0,1]])  # state transition
kf.H = np.array([[1,0,0,0],[0,1,0,0]])                       # measurement

def track_ball(detection, trail):
    if detection:
        kf.update(detection)
    kf.predict()
    pos = kf.x[:2].flatten()
    trail.append(tuple(pos.astype(int)))
    return pos, trail[-20:]  # keep last 20 positions for trail

3. Court homography for real-world measurements

To report speed in km/h rather than pixels/frame, you need to map pixel coordinates to real-world court coordinates. This is done with a homography matrix — computed once at setup by clicking on known court markings.

import cv2, numpy as np

# Click these 4 points on the court in the camera view
pixel_pts = np.float32([[120,450],[1200,450],[1200,680],[120,680]])

# Actual court dimensions in metres (e.g. volleyball: 9m x 18m)
real_pts   = np.float32([[0,0],[9,0],[9,18],[0,18]])

H, _ = cv2.findHomography(pixel_pts, real_pts)

def pixel_to_metres(px, py):
    pt = np.array([[[px, py]]], dtype=np.float32)
    real = cv2.perspectiveTransform(pt, H)
    return real[0][0]  # [x_metres, y_metres]

4. Action classification with ResNet

Once you have body keypoints per player, a lightweight ResNet-18 classifier can determine what action they are performing — spike, serve, block, dig — by analysing the angles and positions of joints. This runs as a second model in the DeepStream pipeline alongside the detection model.

Streaming results to YouTube Live

Once the AI has annotated each frame with boxes, keypoints, ball trail, and scoreboard, the final step is encoding and pushing to YouTube Live via RTMP using Jetson hardware H.264 encoding:

import cv2

STREAM_KEY = "YOUR_YOUTUBE_STREAM_KEY"
RTMP_URL   = f"rtmp://a.rtmp.youtube.com/live2/{STREAM_KEY}"
pipeline   = (
    "appsrc ! videoconvert ! nvv4l2h264enc bitrate=6000000 ! "
    "h264parse ! flvmux streamable=true ! "
    f"rtmpsink location={RTMP_URL}"
)
out = cv2.VideoWriter(pipeline, cv2.CAP_GSTREAMER, 0, 30, (1920,1080))

Real-world applications

Volleyball, basketball, football: player tracking, shot classification, formation analysis
Athletics: sprint timing, technique analysis, injury risk detection from body mechanics
Tennis: ball speed, serve placement, court coverage heatmaps
Hockey and ice hockey: puck tracking, player clustering, penalty detection
Cricket: bowling action analysis, fielding placement optimisation

Want to build your own sports AI system? The HemiHex Volleyball AI Streaming guide gives you a complete step-by-step build, and the Jetson Inspection Kit provides the hardware foundation. Get started →