Pandas, hvplot, datashader, and legends

jorwoods · March 9, 2020, 5:10pm

If I produce a scatter plot using df.hvplot.scatter(x='x', y='y', by='color', datashade=True, groupby='group') is there an easy way for me to get it to produce a legend even though the datashader option is on?

Example dataframe:

df = pd.DataFrame([
    {'x': 1, 'y': 2, 'color':'b', 'group':'a'},
    {'x': 2, 'y': 3, 'color':'r', 'group':'b'},
    {'x': 3, 'y': 1, 'color':'r', 'group':'a'},
    {'x': 4, 'y': 0, 'color':'b', 'group':'b'},
    {'x': 5, 'y': 5, 'color':'b', 'group':'a'},
    {'x': 6, 'y': 3, 'color':'r', 'group':'b'},
    {'x': 7, 'y': 2, 'color':'r', 'group':'a'},
])

ToucheSir · May 20, 2021, 3:13am

Ran into this today as well. At the very least, it would be great to have a note on Customization — hvPlot 0.7.0 documentation about which options are mutually exclusive or ignored when datashade is set.

philippjfr · May 20, 2021, 8:25am

Once I merge Add automatic categorical legend for datashaded plots by philippjfr · Pull Request #4806 · holoviz/holoviews · GitHub this will become possible. Unfortunately until then the only thing I can suggest is overlaying a set of points.

domerich · September 20, 2023, 8:22am

is there any way to preserve the colors when I set rasterize=True? See

magic-lantern · November 28, 2023, 7:25pm

I’m still seeing similar problems with datashade=True not showing legends. Any one know if/how to make a legend show up?

As a code example:

# load hvplot and default bokeh extension
import hvplot.pandas
# load some sample data
import hvplot.sample_data

df = hvplot.sample_data.airline_flights.read()

df['dayofweek'] = df.dayofweek.astype('category')

df[df.carrier=='OO'].hvplot.scatter(
    x='arr_time',
    y='dep_time',
    by='dayofweek'
) + df[df.carrier=='OO'].hvplot.scatter(x='arr_time',
                                        y='dep_time',
                                        by='dayofweek',
                                        datashade=True
                                       )

First plot (on left) has a nice legend, second plot (on right) has no legend.

magic-lantern · November 28, 2023, 8:27pm

Continuing my previous example, this seems like a messy way to get a legend. Seems like this should be part of the library rather than creating a second plot and overlaying the first.

arr = []
for i in range(1, len(df.dayofweek.unique()) + 1):
    arr.append(df[(df.carrier=='OO') & (df.dayofweek == i)].head(1))
minidf = pd.concat(arr)

minidf.hvplot.scatter(
    x='arr_time',
    y='dep_time',
    by='dayofweek',
    s=1
) * df[df.carrier=='OO'].hvplot.scatter(x='arr_time',
                                        y='dep_time',
                                        groupby='dayofweek',
                                        datashade=True
                                       )

ahuang11 · November 29, 2023, 7:04pm

Maybe you can try rasterize=True and set .opts(color_levels=7)

magic-lantern · November 30, 2023, 1:31am

@ahuang11 - I’m not seeing color_levels as an option available at Customization — hvPlot 0.9.0 documentation

I do see that Holoviews does have an option with that name:

import holoviews as hv
hv.help(hv.Raster)

Output includes

color_levels:            Number of discrete colors to use when colormapping or a set of color
                         intervals defining the range of values to map each color to.

However, it’s not clear to me how to access that feature from hvplot. Can you provide a code example or link to documentation?

ahuang11 · November 30, 2023, 1:48am

hvPlot is built on top of HoloViews so you can do:
your_rasterized_hvplot_obj.opts(color_levels=7)

And thus should also work
hv.help(your_rasterized_hvplot_obj)

Can you also submit an issue in hvplot to mention that you can use opts too in customization because it’s not intuitive for users?

magic-lantern · December 1, 2023, 8:34pm

@ahuang11 - Thanks for the additional guidance. I’ll open an issue.

Also, it seems that the color_levels option isn’t available for all plotting objects.

by_plot = df[df.carrier=='OO'].hvplot.scatter(x='arr_time',
                                              y='dep_time',
                                              by='dayofweek',
                                              rasterize=True
                                             )
by_plot

Generates this rasterized plot

Adding .opts(color_levels=7) to my rasterized plot object results in an error:

by_plot.opts(color_levels=7)

ValueError: Unexpected option ‘color_levels’ for NdOverlay type across all extensions. No similar options found.

The Holoviews help doesn’t seem to give any indication about color_levels

hv.help(by_plot)

Parameters of ‘DynamicMap’ instance

Parameters changed from their default values are marked in red.
Soft bound values are marked in cyan.
C/V= Constant/Variable, RO/RW = ReadOnly/ReadWrite, AN=Allow None
…

(continues on with a lot of items but color_levels is not included)

When printing my plot object this is what it tells me.

print(by_plot)

:DynamicMap
:NdOverlay [dayofweek]
:Image [arr_time,dep_time] (arr_time_dep_time Count)

I’m starting to see that I need to understand Holoviews in order to actually use hvPlot outside the default functionality.

updated: Realized I had an incorrect import that was breaking my hv.help() call!

magic-lantern · December 1, 2023, 8:58pm

If I do a groupby plot, the color_levels option is available. However, this is not what I want as it’s different than the by plot.

groupby_plot = df[df.carrier=='OO'].hvplot.scatter(x='arr_time',
                                              y='dep_time',
                                              groupby='dayofweek',
                                              rasterize=True,
                                             )
groupby_plot.opts(color_levels=7)

ahuang11 · December 1, 2023, 9:29pm

Try: y_plot.opts(“Image”, color_levels=7)

magic-lantern · December 1, 2023, 9:48pm

Closer - at least it doesn’t throw an error. However, the resulting plot is unchanged from before the options.

ahuang11 · December 1, 2023, 9:54pm

Are you sure that’s rasterize=True? That looks like datashade=True?

magic-lantern · December 2, 2023, 3:39am

Totally possible that this whole issue is just a PEBCAK!

Here’s the whole code in one notebook:

gist.github.com

https://gist.github.com/magic-lantern/7cc4bd51425b74f66a021665ede0a688#file-hvplot_example-ipynb

environment.yml

name: hvplot_example
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - hvplot
  - intake
  - intake-parquet
  - intake-xarray

This file has been truncated. show original

hvplot_datashade_legend.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3a23a06-adc3-4a7a-a9f9-d52052fe6a2b",
   "metadata": {},
   "source": [
    "# Trying to get datashaded/rasterized hvPlots to show a legend\n",
    "\n",
    "This example uses an hvplot sample data set. As this sample dataset is actually quite small (~ 1 million records) datashading/rasterizing is not very beneficial. HOWEVER, in the case that you have 100 million or more datapoints, datashade/raster is required to generate a plot without crashing the browser (at least in my experience)."

This file has been truncated. show original

hvplot_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f5dbf693-e787-461e-8990-b5a980f3877b",
   "metadata": {},
   "source": [
    "## Example of customizing subplots with hvPlot"
   ]
  },

This file has been truncated. show original

This binder link should work also, though can be slow to start:

magic-lantern · December 4, 2023, 10:59pm

@philippjfr - Perhaps my recent comments are just due to lack of understanding of how to use new Holoviews features in hvPlot as mentioned in your Holoviews issue “Add automatic categorical legend for datashaded plots”

github.com/holoviz/holoviews

Add automatic categorical legend for datashaded plots

holoviz:master ← holoviz:datashader_categorical_legend

opened 12:04PM - 28 Jan 21 UTC

philippjfr

+215 -56

Inspects the pipeline of an RGB element and discovers the shade and rasterize op…erations used to generate an RGB to then automatically add a legend. <img width="1215" alt="Screen Shot 2021-01-28 at 1 04 12 PM" src="https://user-images.githubusercontent.com/1550771/106136282-5e827380-6169-11eb-8e1f-4f84fea2fefb.png"> - [x] Clean up operation and factor out operations to traverse pipeline - [x] Factor out code to render legend plot and use across all interfaces - [ ] Update legend when needed

It was merged into main/master back in May of 2022.

Do you have any recommendations on how to take advantage of your code in hvPlot?

ahuang11 · December 6, 2023, 1:06am

I haven’t taken a deep look, but maybe this will help you
https://examples.holoviz.org/census/census.html

magic-lantern · December 6, 2023, 2:43pm

That’s a great example of what can be done with Datashader. However, in my opinion, it’s also an example of one of the limitations of Datashader: none of the graphics have a legend. The only way to understand the visualizations is to read the paragraphs of text.

I think, in the case of a single color, a carefully worded plot title/description could explain what is being shown. For those with multiple colors, the color_key should be a part of every plot instead of only in text/code:

if background == "black":
      color_key = {'w':'aqua', 'b':'lime',  'a':'red', 'h':'fuchsia', 'o':'yellow' }
else: color_key = {'w':'blue', 'b':'green', 'a':'red', 'h':'orange',  'o':'saddlebrown'}