How do I make a robust and performant Animated Flights app using HoloViz?

Inspired by this great discussion, I decided to build an Animated Flights app! :airplane:

I’ve put together a basic proof of concept (POC) that works, but it currently doesn’t:

  1. Scale well to many flights (50+).
  2. Handle many simultaneous users (50+).
  3. Use hv.Curve or other HoloViews elements — I went with hv.Scatter since hv.Curve didn’t connect the lines as expected, and Points raised exceptions when the buffer was empty.

If anyone’s interested, I think developing a robust, production-ready example would be a fantastic addition to the HoloViews ecosystem. Most of the existing examples I’ve found are great for exploration but typically only scale to a single user.

"""The purpose of this HoloViz Panel App is to showcase how to create a Live Global Flight Tracker using Panel, hvPlot and the rest of the HoloViz ecosystem

The data is dummy data, but the code is structured such that it is easy to plugin real data if you have access too it.

This example was inspired by a similar example using Bokeh https://discourse.bokeh.org/t/animated-flights/12629.

Key Requirements:
- Non-blocking animation updates: The flight tracker updates should be asynchronous to prevent
  blocking the UI and allow for smooth interactions with other dashboard components.
- Extensibility: The code structure should make it easy to add additional interactive components
  (e.g., filters, statistics panels, control buttons) without interfering with the animation.
- Performance: Updates should be efficient to maintain smooth animations even with many flights.
- Scale to +500 concurrent users
- Scale to +100 flight paths simultaneously
- Server should be responsive

Architecture & Techniques:
The live streaming flight tracker leverages several key HoloViews/Panel patterns:

1. hv.streams.Buffer - Acts as a circular buffer to efficiently manage streaming data for each
   flight (trail, fade, head positions). The buffer automatically handles memory management by
   keeping only the most recent N data points, discarding older ones.

2. hv.DynamicMap - Creates dynamic, automatically-updating visualizations that respond to stream
   updates. When the Buffer receives new data, DynamicMap triggers a re-render of only the
   affected elements, not the entire plot, ensuring smooth performance.

3. HoloViews Overlays - Multiple flight paths are composed as HoloViews elements (Curves for
   trails, Points for airplane heads) and combined into overlays. Each flight consists of
   multiple layers (glow, main trail, fading tail, head) that are composited together.

4. pn.state.add_periodic_callback - Schedules the animation update function to run at regular
   intervals (e.g., 33ms for ~30fps). This is the traditional approach for driving animations.

5. pn.io.unlocked - Critical context manager that temporarily unlocks the Bokeh document lock
   during buffer updates, preventing UI deadlocks and ensuring the interface remains responsive
   during rapid data updates.

6. Web Mercator Projection - All coordinates are converted from WGS84 (lat/lon) to Web Mercator
   for proper alignment with map tiles and efficient rendering in the browser.

Implementation Approach:
- Consider using async generator functions to yield data and visualization updates, which provides
  more readable and modern code compared to traditional periodic callbacks. Async generators make
  the flow of data updates easier to follow and maintain.

Resources:
- Streaming Timeseries Example: https://examples.holoviz.org/gallery/streaming_timeseries/streaming_timeseries.html
  Demonstrates using hv.streams.Buffer, hv.DynamicMap, pn.state.add_periodic_callback, and pn.io.unlocked
  for real-time data streaming without blocking the UI.
  
Note: This implementation does not use the Streamz package as it is no longer actively maintained.
"""

# ==============================================================================
# IMPORTS
# ==============================================================================
import panel as pn
import holoviews as hv
from holoviews.streams import Buffer
import numpy as np
import pandas as pd
import random
from datetime import datetime

pn.extension()

# ==============================================================================
# CONSTANTS & CONFIGURATION
# ==============================================================================
# Major global cities with (longitude, latitude) coordinates
CITIES = {
    "New York": (-74.006, 40.7128),
    "Los Angeles": (-118.2437, 34.0522),
    "London": (-0.1276, 51.5074),
    "Paris": (2.3522, 48.8566),
    "Tokyo": (139.6917, 35.6895),
    "Dubai": (55.2708, 25.2048),
    "Singapore": (103.8198, 1.3521),
    "Sydney": (151.2093, -33.8688),
    "São Paulo": (-46.6333, -23.5505),
    "Mumbai": (72.8777, 19.0760),
    "Beijing": (116.4074, 39.9042),
    "Moscow": (37.6173, 55.7558),
    "Istanbul": (28.9784, 41.0082),
    "Bangkok": (100.5018, 13.7563),
    "Toronto": (-79.3832, 43.6532),
    "Hong Kong": (114.1694, 22.3193),
    "Shanghai": (121.4737, 31.2304),
    "Seoul": (126.9780, 37.5665),
    "Madrid": (-3.7038, 40.4168),
    "Rome": (12.4964, 41.9028),
    "Amsterdam": (4.9041, 52.3676),
    "Frankfurt": (8.6821, 50.1109),
    "Chicago": (-87.6298, 41.8781),
    "San Francisco": (-122.4194, 37.7749),
    "Miami": (-80.1918, 25.7617),
    "Delhi": (77.1025, 28.7041),
    "Mexico City": (-99.1332, 19.4326),
    "Cairo": (31.2357, 30.0444),
    "Johannesburg": (28.0473, -26.2041),
    "Melbourne": (144.9631, -37.8136)
}

# Flight animation parameters
N_FLIGHTS = 2
TRAIL_LENGTH = 20
UPDATE_INTERVAL_MS = 500  # ~30fps
BUFFER_LENGTH = 100

# Flight color palette (trail_color, glow_color)
FLIGHT_COLORS = [
    ("#00d4ff", "#0066ff"),
    ("#ff0080", "#ff6b00"),
    ("#00ff88", "#00cc66"),
    ("#ffdd00", "#ff8800"),
    ("#ff00ff", "#8800ff"),
    ("#00ffff", "#00aaff"),
]

# Map tile source (dark theme)
DARK_MAP_URL = "https://basemaps.cartocdn.com/dark_all/{Z}/{X}/{Y}.png"


# ==============================================================================
# HELPER FUNCTIONS
# ==============================================================================
def wgs84_to_web_mercator(lon, lat):
    """
    Convert WGS84 coordinates to Web Mercator projection.
    Vectorized for efficient numpy operations.
    
    Args:
        lon: Longitude(s) in degrees
        lat: Latitude(s) in degrees
    
    Returns:
        tuple: (x, y) coordinates in Web Mercator
    """
    k = 6378137
    x = lon * (k * np.pi / 180.0)
    y = np.log(np.tan((90 + lat) * np.pi / 360.0)) * k
    return x, y


def generate_flight_arc(lon1, lat1, lon2, lat2, steps=200):
    """
    Generate great circle arc between two cities with altitude curve.
    
    Args:
        lon1, lat1: Departure coordinates (longitude, latitude)
        lon2, lat2: Arrival coordinates (longitude, latitude)
        steps: Number of points along the arc
    
    Returns:
        tuple: (x_arc, y_arc) arrays in Web Mercator coordinates
    """
    t = np.linspace(0, 1, steps)
    
    # Vectorized great circle interpolation
    lon_arc = lon1 + (lon2 - lon1) * t
    lat_arc = lat1 + (lat2 - lat1) * t
    
    # Altitude curve for realistic flight path
    distance = np.sqrt((lon2 - lon1)**2 + (lat2 - lat1)**2)
    altitude_factor = min(distance * 0.7, 18)
    lat_arc += altitude_factor * np.sin(np.pi * t) * (1 - 0.3 * t)
    
    return wgs84_to_web_mercator(lon_arc, lat_arc)


def create_flight_data():
    """
    Generate random flight configuration.
    
    Returns:
        dict: Flight configuration with arc data, colors, and speed
    """
    city_list = list(CITIES.values())
    
    # Random route
    departure = random.choice(city_list)
    arrival = random.choice(city_list)
    while arrival == departure:
        arrival = random.choice(city_list)
    
    lon1, lat1 = departure
    lon2, lat2 = arrival
    
    x_arc, y_arc = generate_flight_arc(lon1, lat1, lon2, lat2)
    
    # Color scheme
    trail_color, glow_color = random.choice(FLIGHT_COLORS)
    
    return {
        "x_arc": x_arc,
        "y_arc": y_arc,
        "i": random.randint(0, len(x_arc) // 4),
        "speed": random.uniform(2.0, 4.0),
        "trail_color": trail_color,
        "glow_color": glow_color
    }


# ==============================================================================
# DATA STRUCTURES
# ==============================================================================
class FlightState:
    """
    Track state for a single flight including position, data sources, and styling.
    """
    def __init__(self, flight_data):
        self.x_arc = flight_data["x_arc"]
        self.y_arc = flight_data["y_arc"]
        self.i = flight_data["i"]
        self.speed = flight_data["speed"]
        self.trail_color = flight_data["trail_color"]
        self.glow_color = flight_data["glow_color"]
        
        # Create Buffer streams for each flight component
        self.trail_buffer = Buffer(data=pd.DataFrame({'x': [], 'y': []}), length=BUFFER_LENGTH)
        self.fade_buffer = Buffer(data=pd.DataFrame({'x': [], 'y': []}), length=BUFFER_LENGTH)
        self.head_buffer = Buffer(data=pd.DataFrame({'x': [], 'y': []}), length=10)
    
    def update_position(self):
        """Update flight position and return whether flight is still active."""
        i = int(self.i)
        
        if i < len(self.x_arc):
            # Calculate speed factor based on flight progress
            progress = i / len(self.x_arc)
            if progress < 0.15:  # Takeoff
                speed_factor = 0.6 + (progress / 0.15) * 0.4
            elif progress > 0.85:  # Landing
                speed_factor = 0.6 + ((1 - progress) / 0.15) * 0.4
            else:  # Cruise
                speed_factor = 1.0
            
            # Update buffers with new data
            start = max(0, i - TRAIL_LENGTH)
            fade_start = max(0, i - TRAIL_LENGTH * 2)
            # with pn.io.unlocked():
            # Main trail
            self.trail_buffer.send(pd.DataFrame({
                'x': self.x_arc[start:i],
                'y': self.y_arc[start:i]
            }))
            
            # Fading tail
            self.fade_buffer.send(pd.DataFrame({
                'x': self.x_arc[fade_start:start],
                'y': self.y_arc[fade_start:start]
            }))
            
            # Airplane head
            self.head_buffer.send(pd.DataFrame({
                'x': [self.x_arc[i]],
                'y': [self.y_arc[i]]
            }))
            
            self.i += self.speed * speed_factor
            return True
        else:
            return False
    
    def reset_route(self):
        """Generate a new random route for this flight."""
        print("Resetting flight route...")
        new_data = create_flight_data()
        self.x_arc = new_data["x_arc"]
        self.y_arc = new_data["y_arc"]
        self.i = 0
        self.speed = new_data["speed"]
        
        # 30% chance to change color
        if random.random() < 0.3:
            self.trail_color = new_data["trail_color"]
            self.glow_color = new_data["glow_color"]

        self.trail_buffer.clear()
        self.fade_buffer.clear()
        self.head_buffer.clear()


# ==============================================================================
# VISUALIZATION COMPONENTS
# ==============================================================================
def create_base_map():
    """
    Initialize base map with Web Mercator projection and dark tiles.
    
    Returns:
        hv.Tiles: Base map with styling
    """
    # Web Mercator bounds for the whole world
    # Longitude: -180 to 180, Latitude: approximately -85 to 85 (Web Mercator limit)
    x_range = (-20000000, 20000000)  # Covers roughly -180° to 180° longitude
    y_range = (-10000000, 10000000)  # Covers roughly -85° to 85° latitude
    
    tiles = hv.Tiles(DARK_MAP_URL).opts(
        width=1400,
        height=800,
        xlim=x_range,
        ylim=y_range,
        xaxis=None,
        yaxis=None,
        bgcolor='#0a0a0a',
        active_tools=['wheel_zoom', 'pan']
    )
    return tiles


def create_flight_layer(flight_state):
    """
    Create visualization layers for a single flight.
    
    Args:
        flight_state: FlightState instance
    
    Returns:
        hv.Overlay: Composite of all flight visualization layers
    """
    # Outer glow
    glow = hv.DynamicMap(
        lambda data: hv.Scatter(data, 'x', 'y').opts(
            line_width=5,
            color="blue", # flight_state.glow_color,
            alpha=0.18
        ),
        streams=[flight_state.trail_buffer]
    )
    
    # Core trail
    trail = hv.DynamicMap(
        lambda data: hv.Scatter(data, 'x', 'y').opts(
            line_width=2,
            color="red", # flight_state.trail_color,
            alpha=0.85
        ),
        streams=[flight_state.trail_buffer]
    )
    
    # Fade tail
    fade = hv.DynamicMap(
        lambda data: hv.Scatter(data, 'x', 'y').opts(
            line_width=1.5,
            color="orange", # flight_state.fade_color,
            alpha=0.25
        ),
        streams=[flight_state.fade_buffer]
    )
    
    # Airplane head - outer glow
    head_glow = hv.DynamicMap(
        lambda data: hv.Scatter(data, 'x', 'y').opts(
            size=18,
            color=flight_state.glow_color,
            alpha=0.2
        ),
        streams=[flight_state.head_buffer]
    )
    
    # Airplane head - main marker
    head_main = hv.DynamicMap(
        lambda data: hv.Scatter(data, 'x', 'y')
        .opts(
            size=8,
            color=flight_state.trail_color,
            alpha=1.0
        ),
        streams=[flight_state.head_buffer]
    )
    
    # Airplane head - center dot
    head_center = hv.DynamicMap(
        lambda data: hv.Scatter(data, 'x', 'y')
        .opts(
            size=3,
            color='white',
            alpha=1.0
        ),
        streams=[flight_state.head_buffer]
    )
    return glow * trail * fade * head_glow * head_main * head_center


def update_flight_positions(flights):
    """
    Update all flight positions and reset completed flights.
    
    Args:
        flights: List of FlightState instances
    """
    with pn.io.unlocked():
        for flight in flights:
            if not flight.update_position():
                # Flight completed, generate new route
                flight.reset_route()


# ==============================================================================
# PANEL DASHBOARD
# ==============================================================================
def create_dashboard():
    """
    Create the main Panel dashboard with flight tracker visualization.
    
    Returns:
        pn.Column: Complete dashboard layout
    """
    # Initialize flights
    flights = [FlightState(create_flight_data()) for _ in range(N_FLIGHTS)]
    
    # Create base map
    base_map = create_base_map()
    
    # Create all flight layers
    flight_layers = [create_flight_layer(flight) for flight in flights]
    
    # Combine all layers into a single visualization
    visualization = base_map
    for layer in flight_layers:
        visualization = visualization * layer
    
    # Set up periodic callback for animation (only when document is available)
    def update():
        print(datetime.now(), "Updating flight positions...")
        update_flight_positions(flights)
    
    if pn.state.curdoc:
        pn.state.add_periodic_callback(update, period=UPDATE_INTERVAL_MS)
    
    # Create dashboard layout with title styling
    dashboard = pn.Column(
        pn.pane.Markdown("# Live Global Flight Tracker", 
                        styles={'color': '#E0E0E0', 'text-align': 'center'}),
        pn.pane.HoloViews(visualization, sizing_mode='stretch_width', min_height=800),
        sizing_mode='stretch_width'
    )
    return dashboard


# ==============================================================================
# MAIN EXECUTION
# ==============================================================================
# Create and serve the dashboard
app = create_dashboard()

if pn.state.served:
    app.servable()

1 Like