I am following this example of building a GUI to control a small experiment.
I am new to GUIs in general and Holoviews and Panel in particular, and am learning as I go along.
I have a few questions regarding the “correct” way to do things.
Right now, the program runs in a Jupyter notebook, measures data, and saves it in a hv.streams.Buffer. That data is then plotted using hv.DynamicMap(plot, streams=[buffer]) (as in the above example)
where plot is hv.Curve for example
After the measurement finishes, everything is saved in a pandas dataframe, and I fit the data using linear regression.
Then I plot the fit on top of the original data by combining both using the asterisk (e.g. plot*fit, instead of just plot)
This works fine, but only after I rerun the Panel.Row() that contains the DynamicMap.
What is the proper way to display the fit on top of the the plot as soon as the measurement finishes?
If there are several ways to do this, which one is better? (e.g. can it be done with hvplot only, or is holoviews necessary)
Are there important differences if I run this in a Jupyter notebook versus “standalone”?
Extra question: What is the best way to create a pandas dataframe from the Buffer of my data after the measurement is finished?
Thank you @Marc
I initially left out the MWE because I think my problems lie more with the general understanding of how things are done. Anyway, I included it below and would be grateful for any pointers.
import time
import asyncio
import numpy as np
from scipy.optimize import curve_fit
import pandas as pd
import holoviews as hv
import panel as pn
from holoviews.streams import Buffer
import hvplot.pandas
class FakeInstrument(object):
def __init__(self, offset=0.0):
self.offset = offset
def set_offset(self, value):
self.offset = value
def read_data(self):
return np.random.random() + self.offset
instrument = FakeInstrument()
def fit_to_data(x, m, c):
return m * x + c
def make_df(data_frequency=1.0, data_phaseshift=0.0):
return pd.DataFrame(data={'Frequency (Hz)': data_frequency, 'Phase shift (deg)': data_phaseshift}, index=[0])
empty_reference_df = pd.DataFrame(columns=make_df().columns)
df = pd.DataFrame(columns=make_df().columns)
buffer_length = 100
buffer = Buffer(empty_reference_df, length=buffer_length, index=False)
plot = hv.DynamicMap(hv.Curve, streams=[buffer]).opts(padding=0.1, xlim=(0, None))#This shows the data as it is acquired in real time
LABEL_START = 'Start measurement'
LABEL_STOP = 'Stop measurement'
button_run_experiment = pn.widgets.Button(name=LABEL_START)
acquisition_task = None
fit = None#what would be the correct way the initialize an "empty" plot?
async def acquire_data():
#def acquire_data():
global df
global fit
offset = 0.2
interval = 0.1
t0 = time.time()
time_elapsed = 0
while time_elapsed < 2:#the experiment finishes automatically after some time
instrument.set_offset(offset)
time_elapsed = time.time() - t0
value = instrument.read_data()
b = make_df(time_elapsed, value)
buffer.send(b)#datapoint is sent to the buffer, to be plotted
df = pd.concat([df, b])#Additionally, it is added to a dataframe, for fitting and saving to file. Can this be done simpler? (Directly from buffer -> dataframe?)
time_spent_buffering = time.time() - t0 - time_elapsed
if interval > time_spent_buffering:
await asyncio.sleep(interval - time_spent_buffering)
popt, pcov = curve_fit(fit_to_data, df.iloc[:,0], df.iloc[:,1])
df['Fit'] = fit_to_data(df.iloc[:,0], *popt)#I add the fitted/calculated datapoints to the dataframe, to plot later.
fit = df.hvplot.line(x='Frequency (Hz)', y='Fit')#This is supposed to show a fit to the data after the experiment is run
button_run_experiment.clicks +=1
return
def run_experiment(*events):
global df
global acquisition_task
if button_run_experiment.name == LABEL_START:
button_run_experiment.name = LABEL_STOP
button_run_experiment.button_type = "danger"
buffer.clear()
df = pd.DataFrame(columns=make_df().columns)
acquisition_task = asyncio.gather(acquire_data())
else:
acquisition_task.cancel()
button_run_experiment.name = LABEL_START
button_run_experiment.button_type = "default"
button_run_experiment.on_click(run_experiment)
pn.Column(plot, button_run_experiment)#This is run initially.
#pn.Column(plot*fit, button_run_experiment)#After having run the experiment (via the button), the data has been fitted, and now this can be run to show bot data and fit
Thanks for the MWE. It is always interesting to see how people organize things and it gives context and a support for discussion.
How many samples are we talking about? Less than 100000? If so everything should fit in memory and easily tractable for bokeh.
I ask that because all your data is in the plot object. Indeed, it is important to understand that HoloViews is not doing any graphical rendering (Bokeh is in charge of this) ; it is a library to easily specify, transform, annotate the data to get what you want in terms of visualisation and interactivity. Thus the data remains accessible within the HoloViews object.
→ Have a look at plot.dframe(), you’ll have a nice surprise . You don’t need to create the dataframe. Maybe you can clean up your code with that new fact then we can move on to your other questions.
Okay, I removed the superfluous dataframe, and added an empty “fit” plot from the beginning. using plot*fit. However, upon (re)calculating the fit, the graphical output does not update.
Do I use @pn.depends for this? Or maybe a function that upon finishing returns the hole Column of interface elements?
This is the part I don’t quite understand. For the plot to be updated, does the figure (here “plot” or “fit”) have to be recalculated, or the UI element (here pn.Column(plot*fit)).
From my trials, it seems the first is not enough, but the second seems wasteful, though I may very well be mistaken.
Again, thank you for your time, any hints are very much appreciated.
import time
import asyncio
import numpy as np
from scipy.optimize import curve_fit
import pandas as pd
import holoviews as hv
import panel as pn
from holoviews.streams import Buffer
import hvplot.pandas
class FakeInstrument(object):
def __init__(self, offset=0.0):
self.offset = offset
def set_offset(self, value):
self.offset = value
def read_data(self):
return np.random.random() + self.offset
instrument = FakeInstrument()
def fit_to_data(x, m, c):
return m * x + c
def make_df(data_frequency=1.0, data_phaseshift=0.0):
return pd.DataFrame(data={'Frequency (Hz)': data_frequency, 'Phase shift (deg)': data_phaseshift}, index=[0])
empty_reference_df = pd.DataFrame(columns=make_df().columns)
buffer_length = 100
buffer = Buffer(empty_reference_df, length=buffer_length, index=False)
plot = hv.DynamicMap(hv.Curve, streams=[buffer]).opts(padding=0.1, xlim=(0, None))#This shows the data as it is acquired in real time
fit = hv.Curve([])#an empty plot since no data to fit exists yet
LABEL_START = 'Start measurement'
LABEL_STOP = 'Stop measurement'
button_run_experiment = pn.widgets.Button(name=LABEL_START)
acquisition_task = None
async def acquire_data():
#def acquire_data():
global fit
offset = 0.2
interval = 0.1
t0 = time.time()
time_elapsed = 0
while time_elapsed < 2:#the experiment finishes automatically after some time
instrument.set_offset(offset)
time_elapsed = time.time() - t0
value = instrument.read_data()
b = make_df(time_elapsed, value)
buffer.send(b)#datapoint is sent to the buffer, to be plotted
time_spent_buffering = time.time() - t0 - time_elapsed
if interval > time_spent_buffering:
await asyncio.sleep(interval - time_spent_buffering)
popt, pcov = curve_fit(fit_to_data, plot.dframe().iloc[:,0], plot.dframe().iloc[:,1])
fit = hv.Curve([(i, fit_to_data(i, *popt)) for i in plot.dframe().iloc[:,0]])#This is a fit to the current data, calculated after the experiment finishes
button_run_experiment.clicks +=1
return
def run_experiment(*events):
global acquisition_task
if button_run_experiment.name == LABEL_START:
button_run_experiment.name = LABEL_STOP
button_run_experiment.button_type = "danger"
buffer.clear()
acquisition_task = asyncio.gather(acquire_data())
else:
acquisition_task.cancel()
button_run_experiment.name = LABEL_START
button_run_experiment.button_type = "default"
button_run_experiment.on_click(run_experiment)
pn.Column(plot*fit, button_run_experiment)#To update the displayed data, this has to be rerun
In fact you were halfway there. You can employ the same logic for fit as you did for plot. That is to say, define it as DynamicMap placeholder of which you’ll update the data once available.
Since you have small data, you don’t have to use Buffer and can rather use Pipe (see the documentation).
I also updated a bit the fitting part since you can use linear regression rather than non linear regression with a linear function.
import time
import asyncio
import numpy as np
import pandas as pd
import holoviews as hv
import panel as pn
import hvplot.pandas
################################
import scipy.stats as stats
from holoviews.streams import Buffer, Pipe
################################
class FakeInstrument(object):
def __init__(self, offset=0.0):
self.offset = offset
def set_offset(self, value):
self.offset = value
def read_data(self):
return np.random.random() + self.offset
instrument = FakeInstrument()
def make_df(data_frequency=1.0, data_phaseshift=0.0):
return pd.DataFrame(data={'Frequency (Hz)': data_frequency, 'Phase shift (deg)': data_phaseshift}, index=[0])
empty_reference_df = pd.DataFrame(columns=make_df().columns)
buffer_length = 100
buffer = Buffer(empty_reference_df, length=buffer_length, index=False)
plot = hv.DynamicMap(hv.Curve, streams=[buffer]).opts(padding=0.1, xlim=(0, None))#This shows the data as it is acquired in real time
################### Placeholder for the fit
fitpipe = Pipe(data=[])
fit = hv.DynamicMap(hv.Curve, streams = [fitpipe])
################################
LABEL_START = 'Start measurement'
LABEL_STOP = 'Stop measurement'
button_run_experiment = pn.widgets.Button(name=LABEL_START)
acquisition_task = None
async def acquire_data():
#def acquire_data():
global fit
offset = 0.2
interval = 0.1
t0 = time.time()
time_elapsed = 0
while time_elapsed < 2:#the experiment finishes automatically after some time
instrument.set_offset(offset)
time_elapsed = time.time() - t0
value = instrument.read_data()
b = make_df(time_elapsed, value)
buffer.send(b)#datapoint is sent to the buffer, to be plotted
time_spent_buffering = time.time() - t0 - time_elapsed
if interval > time_spent_buffering:
await asyncio.sleep(interval - time_spent_buffering)
######################## Linear regression
xs, ys = plot.dframe()['Frequency (Hz)'], plot.dframe()['Phase shift (deg)']
model = stats.linregress(xs,ys)
datafit = pd.DataFrame({'Frequency (Hz)':xs,'Phase shift (deg)':model.slope*xs+model.intercept})
fitpipe.send(datafit)
################################
button_run_experiment.clicks +=1
return
def run_experiment(*events):
global acquisition_task
if button_run_experiment.name == LABEL_START:
button_run_experiment.name = LABEL_STOP
button_run_experiment.button_type = "danger"
##################### Empty the pipe
fitpipe.send([])
################################
buffer.clear()
acquisition_task = asyncio.gather(acquire_data())
else:
acquisition_task.cancel()
button_run_experiment.name = LABEL_START
button_run_experiment.button_type = "default"
button_run_experiment.on_click(run_experiment)
pn.Column(plot*fit, button_run_experiment)
. You don’t need to create the dataframe. Maybe you can clean up your code with that new fact then we can move on to your other questions.