Custom Discrete Colormaps

[WesleyTheGeolien]

Hello,

I am looking to create custom colormap steps. eg what .opts(color_levels=levels) does pretty much but rather than splitting on the levels I want each level to be equal.

For example I might have a colorbar that is equally spaced every 200m between 0 and 1000 then spaced every 1000 between 1000 and 6000.

I found this website that explains how to create a custom map which I have been able to apply with satisfactory results HOWEVER my colorbar is still normally scaled is there a way to “fix” this? Also I fancy something could be passed to holoviews.plotting.util.color_intervals to allow this sort of thing to be done automatically.

Basic result using color_levels:

Current Result using code from link above:

Desired Result (notice colorbar):

[philippjfr]

I don’t think this is possible right now. I’d suggest you chime in here.

[AurelienSciarra]

I may have a workaround but I don’t know if it will work with your plot.

    import pandas as pd
    import numpy as np
    import holoviews as hv
    from holoviews import opts
    from bokeh.palettes import all_palettes
    from bokeh.models import FuncTickFormatter
    from bokeh.plotting import show

    df = pd.DataFrame(columns=['x','y','val','bin'])
    df['x'] = list(range(10))
    df['y'] = list(range(10))
    df['val'] = np.random.randint(110,size=10)
    df['bin'] = (df.val/10).astype(int)
    plot = hv.Scatter(df,kdims=['x'],vdims=['y','bin']).opts(color='bin',colorbar=True,tools=['hover'],color_levels=11)
    plot

    def custom_colorbar(plot,palette,labels=None):
        plot = plot.opts(color_levels=10)
        min_ = 0
        nb_color = len(palette)
        max_=2*nb_color
        rend = hv.render(plot)
        cmapper = rend.right[0].color_mapper
        ticker = rend.right[0].ticker
        cmapper.high = nb_color
        cmapper.palette = palette
        ticker.desired_num_ticks = nb_color+1
        ticker.num_minor_ticks = 0
        step = int((max_-min_)/nb_color)
        ticker.base=2
        ticker.mantissas = list(range(min_,max_+step,step))
        if labels is not None and len(labels) == nb_color+1:
            Xlabel = dict(zip(list(range(nb_color+1)),labels))
            formatter = FuncTickFormatter(code="""
                        var labels = %s;
                        return labels[tick] || tick;
                        """ % Xlabel)
            rend.right[0].formatter = formatter
        return rend

    labels = ['0', '200', '400', '600','800', '1000', '2000','3000','4000','5000','6000']
    new_rend = custom_colorbar(plot,all_palettes['BrBG'][10],labels)
    show(new_rend)

Before

After

The plot generated by the function is no longer a Holoview plot, now it’s a bokeh figure.
And before using this function you need to add a column in your datasource in which you will map your value with the color levels (0 to 10 in your case).

[ahuang11]

You could use hooks to keep it as holoviews object.

[StuckDuckF]

How would you do that Mr. @ahuang11 . I am trying to achieve the same thing but with predefined thresholds for an Xarray Dataarray. Trying to replicate the following

I managed to get the colors almost right using but I am missing the black region (500+)

colors = ['#cccccc', '#999999', '#7489a8','#00ccff','#0000fe','#ffff00','#dbc903','#fa8484','#990001','#000000']
thresholds = [2, 5, 10, 20, 40, 60, 80, 100, 200, 500]

but I can not enforce the data to be binned according to the thresholds.

Mr. @ahuang11 is answering on this GitHub issue and even opened a feature reuqest

raqdps_ds = fstd2nc.Buffer(
    RAQDPS_FSTS,
    vars=["AF", "AC", "O3", "N2"],
    rpnstd_metadata=True,
    opdict=True,
    forecast_axis=True,
).to_xarray()


crs_in = RotatedPole(
    pole_latitude=raqdps_ds.rotated_pole.attrs.get("grid_north_pole_latitude"),
    pole_longitude=raqdps_ds.rotated_pole.attrs.get("grid_north_pole_longitude"),
    central_rotated_longitude=0.0,
    globe=Globe(
        ellipse=None,
        semimajor_axis=raqdps_ds.rotated_pole.attrs.get("earth_radius"),
        semiminor_axis=None,
    ),
)

raqdps_ds.AF.where(raqdps_ds.AF > 2.0).hvplot.quadmesh(
    x="rlon",
    y="rlat",
    rasterize=True,
    geo=True,
    crs=crs_in,
    projection=crs_in,
    project=True,
).opts(
    cmap=colors,
    colorbar_opts={
0.        "ticker": FixedTicker(ticks=[2, 5, 10, 20, 40, 60, 80, 100, 200, 500, 2000])
    },
    color_levels=[2, 5, 10, 20, 40, 60, 80, 100, 200, 500, 2000],
    clim=(2, 500),
) * (
    coastline * borders * ocean * lakes * rivers
).opts(
    projection=crs_in
) * HOTSPOTS_HVPLOT
)

the output of which looks like

and raqdps_ds looks like

I am still missing the 500+ range in black and I am not sure what I am doing wrong and computing the max to pass to clim is not really an option for me.

[ahuang11]

What if you set clim=(2, 2000) and just hide the ticker?

import xarray as xr
import hvplot.xarray
from bokeh.models import FixedTicker

ds = xr.tutorial.open_dataset('air_temperature')

ds.hvplot.quadmesh("lon", "lat").opts(
    cmap=["green", "yellow", "orange", "red"],
    colorbar_opts={"ticker": FixedTicker(ticks=[230, 250, 270, 300])},
    color_levels=[230, 250, 270, 300, 320],
    clim=(230, 320),

)

[StuckDuckF]

I actually tried this but I am not sure how to hide the ticker and it seems it breaks the ranges Mr. @ahuang11, in your example could you have say 230-250 be green, 250-270 be yellow, 270-300 be orange and 300+ be red with your suggestion?

[ahuang11]

Yes just remove one ticker value (the code above).

[StuckDuckF]

That fixes it for me, by the way is it possible to have the color bar be of equal height as the non linear scale makes some ticks disappear. Mr. @ahuang11 is awesome

[ahuang11]

Not entirely sure; you might have to play around with the Bokeh tickers; it might be categorical
tickers — Bokeh 3.1.0 Documentation

You can also just hack it by reducing the 2000 to 1000

[StuckDuckF]

Hahaha I “hacked” but I will post here if I find out the official solution.

[kun126]

Hi @StuckDuckF, thanks for sharing your code.

I noticed that, to have colorbar_opts callable, you keep geo features (coastline, borders, ocean…) outside the quadmesh function. I tried something similar on my side, but it took quite long to get the map.
Is there something else I can do to speed it up, or to set features=['borders','coastline'] inside with callable colorbar_opts?

[ahuang11]

by the way is it possible to have the color bar be of equal height as the non linear scale

I was trying to do the same and was reading:

How to custom scale of ColorBar?

However, I don’t think it’s possible with holoviews alone

Is there something else I can do to speed it up, or to set features=['borders','coastline'] inside with callable colorbar_opts ?

You can still call colorbar_opts with features like .opts("QuadMesh", colorbar_opts={...}) However I don’t think that will affect the speed at all. With no additional info, I can only propose trying rasterize=True or converting your dataset into zarr / kerchunk.

[ahuang11]

Just kidding, found a way, but extremely verbose and tedious.

The idea is to first map the colors first.

Then artificially add a custom colorbar as a hook.

Let me see if I can reproduce with a map…

import holoviews as hv
import pandas as pd
from bokeh.models import (
    CategoricalColorMapper,
    ColorBar,
)

hv.extension("bokeh")


def cbar_hook(hv_plot, element):
    plot = hv_plot.handles["plot"]
    cdict = {
        "#465a55": "x < -5000",
        "#75968f": "-5000 < x < -2500",
        "#a5bab7": "-2500 < x < -1000",
        "#c9d9d3": "-1000 < x < -500",
        "#e2e2e2": "-500 < x < 0",
        "#dfccce": " 0 < x < 500",
        "#ddb7b1": "500 < x < 1000",
        "#cc7878": "1000 < x < 2500",
        "#933b41": "2500 < x < 5000",
        "#550b1d": "x > 5000",
    }
    # mapper = LinearColorMapper(palette=colors, low=min(data_heatmap), high=max(data_heatmap))
    mapper = CategoricalColorMapper(
        palette=list(cdict.keys()), factors=list(cdict.values())
    )

    color_bar = ColorBar(
        color_mapper=mapper,
        major_label_text_font_size="15px",
        # ticker=BasicTicker(desired_num_ticks=len(colors)),
        # formatter=PrintfTickFormatter(format="%d"),
        label_standoff=6,
        border_line_color=None,
    )
    plot.add_layout(color_bar, "right")


data_heatmap = [-647, 25756, -7600, -1235, -1345]
data_x = [0, 1, 2, 3, 4]
data_y = [4, 2, 3, 1, 5]
df = pd.DataFrame(data={"heatmap": data_heatmap, "x": data_x, "y": data_y})

bins = [
    min(data_heatmap) - 1,
    -5000,
    -2500,
    -1000,
    -500,
    0,
    500,
    1000,
    2500,
    5000,
    max(data_heatmap) + 1,
]  # define your bins (need to -1 from min and +1 to max to get all inclusive)
palette = [
    "#465a55",
    "#75968f",
    "#a5bab7",
    "#c9d9d3",
    "#e2e2e2",
    "#dfccce",
    "#ddb7b1",
    "#cc7878",
    "#933b41",
    "#550b1d",
]  # define your colors (length should be = len(bins)-1)
ldict = {
    bins[i]: x for i, x in enumerate(palette)
}  # makes a dictionary mapping the left bin to a color
# do the binning
df["b"] = pd.cut(df["heatmap"], bins)
# get left bin for colormapping
df["b"] = [x.left for x in df["b"]]
# use ldict to get color now
df["color"] = df["b"].map(ldict)

hv.Points(df, kdims=["x", "y"], vdims=["heatmap", "color"]).opts(
    color="color",
    cmap=palette,
    width=500,
    height=500,
    size=10,
    colorbar=True,
    hooks=[cbar_hook],
)

[ahuang11]

I surprise myself; it’s actually pretty clean

import xarray as xr
import hvplot.xarray
from bokeh.models import CategoricalColorMapper, ColorBar

COLORS = ["green", "yellow", "orange", "red"]
BOUNDS = [230, 250, 270, 300, 320]


def cbar_hook(hv_plot, _):
    plot = hv_plot.handles["plot"]
    factors = [f"{BOUNDS[i]} - {BOUNDS[i + 1]}" for i in range(len(COLORS))]
    mapper = CategoricalColorMapper(
        palette=COLORS,
        factors=factors,
    )
    color_bar = ColorBar(color_mapper=mapper)
    plot.right[0] = color_bar


ds = xr.tutorial.open_dataset("air_temperature")
ds.hvplot.quadmesh("lon", "lat").opts(
    cmap=COLORS,
    color_levels=BOUNDS,
    clim=(BOUNDS[0], BOUNDS[-1]),
    hooks=[cbar_hook],
)

[JulianGiles]

Hi all, I have been following this topic for a while now and just came back to try to do a plot with a discrete colorbar. Thanks @ahuang11 for a very nice way of adding the colorbar with equally-sized colors!

However, I’ve found that there are still “minor” errors in how hvplot pairs values with colors, which are independent of the colorbar that is added manually with a hook.

I have reproduced your exact example @ahuang11 and even though it looks good at first glance, there are pixels which are incorrectly colored. Example:

This pixel at the bottom with value 300.100 should be colored red, however it is orange. This goes under the radar in a plot like this, but comes up notoriously in more extreme cases (like the plots with my data that involve non-regular intervals with 2 decimal positions between 0.7 and 1, several pixels are incorrectly colored).

[JulianGiles]

(had to continue in a new message since I am new and I cannot post more than 1 image per reply)

Another example, if I change COLORS and BOUNDS to catch extreme values under different colors,

COLORS = ["magenta", "green", "yellow", "orange", "red", "purple"]
BOUNDS = [0, 230, 250, 270, 300, 320, 400]

Then I get this plot

where it is evident that several values on the verge of a color limit have changed.

This is independent of the hook to make the nice colorbar, which basically overwrites the default colorbar. I understand that hvplot of linear interpolation between colors that is not a piecewise interpolation between the provided color limits, see the Github discussion mentioned above.

Any ideas on how to work around this issue?

[ahuang11]

Wow thanks for noticing that! I don’t have a solution at the moment, but if I were to approach this later, I would manually recreate the dataset with explicit values for control, and experiment from there.

It seems like a rounding issue.

Also, maybe file this as an issue on HoloViews?

I suspect there might be something up here in holoviews.plotting.util; I see round(), but I think it is needed to use as index.

def color_intervals(colors, levels, clip=None, N=255):
    """
    Maps the supplied colors into bins defined by the supplied levels.
    If a clip tuple is defined the bins are clipped to the defined
    range otherwise the range is computed from the levels and returned.

    Arguments
    ---------
    colors: list
      List of colors (usually hex string or named colors)
    levels: list or array_like
      Levels specifying the bins to map the colors to
    clip: tuple (optional)
      Lower and upper limits of the color range
    N: int
      Number of discrete colors to map the range onto

    Returns
    -------
    cmap: list
      List of colors
    clip: tuple
      Lower and upper bounds of the color range
    """
    if len(colors) != len(levels)-1:
        raise ValueError('The number of colors in the colormap '
                         'must match the intervals defined in the '
                         'color_levels, expected %d colors found %d.'
                         % (N, len(colors)))
    intervals = np.diff(levels)
    cmin, cmax = min(levels), max(levels)
    interval = cmax-cmin
    cmap = []
    for intv, c in zip(intervals, colors):
        cmap += [c]*int(round(N*(intv/interval)))
    if clip is not None:
        clmin, clmax = clip
        lidx = int(round(N*((clmin-cmin)/interval)))
        uidx = int(round(N*((cmax-clmax)/interval)))
        uidx = N-uidx
        if lidx == uidx:
            uidx = lidx+1
        cmap = cmap[lidx:uidx]
        if clmin == clmax:
            idx = np.argmin(np.abs(np.array(levels)-clmin))
            clip = levels[idx: idx+2] if len(levels) > idx+2 else levels[idx-1: idx+1]
    return cmap, clip

[JulianGiles]

Yes, it may be an issue related to some rounding of the color ranges. I can file an issue on HoloViews.

So far the only workaround that I found is to apply .where() to my xarray dataset recursively, selecting the values for each color separately, and plotting each selection with a unique color of the colorbar, then stacking the QuadMeshes by multiplying them. A final transparent layer with all values has to be added on top to retain the values in the plot to be able to see them when hovering the mouse over the pixels. Something like this:

COLORS = ["magenta", "green", "yellow", "orange", "red", "purple"]
BOUNDS = [0, 230, 250, 270, 300, 320, 400]

def cbar_hook(hv_plot, _):
    plot = hv_plot.handles["plot"]
    factors = [f"{BOUNDS[i]} - {BOUNDS[i + 1]}" for i in range(len(COLORS))]
    mapper = CategoricalColorMapper(
        palette=COLORS,
        factors=factors,
    )
    color_bar = ColorBar(color_mapper=mapper)
    plot.right[0] = color_bar


ds = xr.tutorial.open_dataset("air_temperature").isel(time=0)#-273.15

plots = []
for n,cc in enumerate(COLORS):
    plots.append(ds.where(ds>=BOUNDS[n]).where(ds<BOUNDS[n+1]).hvplot.quadmesh("lon", "lat").opts(
        cmap=[COLORS[n]],
        color_levels=BOUNDS[n:n+2],
        clim=(BOUNDS[n], BOUNDS[n+1]),
        # hooks=[cbar_hook],
        width=800, height=400,
    ))

layout = pn.Column( reduce((lambda x, y: x * y), plots) *
    ds.hvplot.quadmesh("lon", "lat").opts(
        cmap=['#ffffff00'], # add a transparent layer only to recover the values when hovering with mouse
        color_levels=BOUNDS[0]+BOUNDS[-1],
        clim=(BOUNDS[0], BOUNDS[-1]),
        hooks=[cbar_hook],
        width=800, height=400,
    )

)

This solution is not only inconvenient, but since the result is an Overlay of several QuadMesh elements, it gets big very fast. It works fine for a simple example like the one above, but with more complex data it gets very slow and saving the result to a file takes a lot of space. The main issue being that Overlay object contains a lot of QuadMesh elements:

:Overlay
   .QuadMesh.I   :QuadMesh   [lon,lat]   (air)
   .QuadMesh.II  :QuadMesh   [lon,lat]   (air)
   .QuadMesh.III :QuadMesh   [lon,lat]   (air)
   .QuadMesh.IV  :QuadMesh   [lon,lat]   (air)
   .QuadMesh.V   :QuadMesh   [lon,lat]   (air)
   .QuadMesh.VI  :QuadMesh   [lon,lat]   (air)

Is there any way to reduce the Overlay to a single QuadMesh that is the result of all the stacked elements? Or, is there a way of pre-rendering the image so that only the visible data of the bokeh figure is saved?

[ahuang11]

Not sure if there’s a solution for the workaround; it probably is easier to contribute a fix to color_intervals (if that’s the function that is wrong).

You could try providing the exact color_levels and colors to color_intervals and see if it lines up with your expectation.

I commented on the issue here Plotting with a discrete nonlinear colorbar assigns values to colors incorrectly · Issue #5966 · holoviz/holoviews · GitHub