Holoview + Apache Druid backend

ccbeloy · September 2, 2022, 3:47pm

Hi,

I found below example on using Heavy.AI as database backend for Holoview plots.

Quansight/omnisci-examples/blob/master/notebooks/workshop/Connecting_Holoviz_Visualization_Ecosystem_to_OmniSci_part_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OmniSci - Ibis - Holoviews - hvPlot\n",
    "\n",
    "![](https://avatars1.githubusercontent.com/u/7553829?s=200&v=4)    ![](https://avatars1.githubusercontent.com/u/34879953?s=200&v=4) \n",
    "\n",
    "### Bio\n",
    "\n",
    "Tony Fast is a Developer Advocate at Quansight with a passion for literate programing and a contributor to Jupyter and HoloViz.\n",
    "\n",
    "Kim Pevey is a Software Engineer at Quansight with an interest for analyzing big data with visualization tools in python. She is also a contributor to HoloViz.\n",
    "\n",
    "### Abstract\n",
    "This notebook demonstrates visualizations of OmniSci Database data using   \n",
    "the Holoviz tools Holoviews and hvPlot. The connection between the OmniSci  \n",
    "Database and Holoviz is an OmniSci backend in Ibis"

This file has been truncated. show original

I wanted to achieve the same using Apache Druid database. I don’t think there is Ibis database driver for Druid. There is an existing PyDruid library not sure if it integrates with Ibis interfaces. Another option could also be Ibis + SqlAchemy, if this is available?

I would really appreciate if someone could provide some guidance please? Thanks.

Marc · September 3, 2022, 5:14am

Hi @ccbeloy

One easy way to use Apache Druid could be to use hvplot .bind and .interactive.

If you try it out and get stuck then try to post a new question including simple steps to get a druid server with data up and running and some example queries you would like to use. That would make it relatively easy for the community to try to put an example together.

ccbeloy · September 3, 2022, 1:09pm

Hi @Marc

I’m looking for more elegant solution like what holoviews + Ibis offers where initial load of data and further transformations like filters/aggregation (and whatnot), all automatically converted to backend database queries.

The .bind and interactive I think would definitely work since you can always fallback to manually querying the database, but I think is more difficult and more coding involve and really requires knowledge in the transformation pipeline of holoview just like having a link selection across different plots like link_selection on both datashade points map and histogram.

Also, I’m not able to find any reference implementation using this approach, perhaps if someone could share a sample implementation please?

Hope this clarifies what I’m trying to achieve . Thanks.

Marc · September 3, 2022, 5:41pm

@ccbeloy . You can find the Ibis implementation here.

github.com

holoviz/holoviews/blob/master/holoviews/core/data/ibis.py

import sys
import numpy
from packaging.version import Version

try:
    from collections.abc import Iterable
except ImportError:
    from collections import Iterable

from .. import util
from ..element import Element
from ..ndmapping import NdMapping, item_check, sorted_context
from .interface import Interface
from . import pandas
from .util import cached


try:
    import ibis
    ibis_version =  Version(ibis.__version__)

This file has been truncated. show original

Its probably the best reference