Distributed fiber optic sensors: panel multi app with flask

Hi all,

In this post I want to show the app I am building for my research work. Before continue talking about the webapp, I am going to give some context. We are a photonics research group working in custom fiber optic distributed sensors (https://en.wikipedia.org/wiki/Fiber-optic_sensor). These sensors measure the vibration and temperature signal along 100 km of oil pipeline in real time with a resolution less than one meter. Analyzing the vibration signals and its spectral response with some classical probabilities and convolutional networks with LSTM we can predict if there is some risk for the pipeline and where it is happenning in real time, for example we can detect a big excavator digging next to the pipeline or some intrusor trying to thief the oil. In these cases, the people controlling the pipeline can send security personal to avoid the risk.

After one year playing with panel holoviz I am going to show the app I am building. It is noteworthy to mention that one and a half year ago I only used labview. The webapp is a multipage app with panel and flask. The login is done with ldap and active directory (ldap3 library), the users and the routing is managed by flask while the connection with the measurement equipments, the database and the data is controlled by panel.

The layout consist in an icon bar in the left (https://www.w3schools.com/howto/howto_css_icon_bar.asp) for the routing of the 6 different panel app served with pn.serve(dict_apps). The apps use some of the panel templates and are iframed due to not being able to embed the components of panel in custom templates in flask as is requested in this bokeh feature request (https://github.com/bokeh/bokeh/issues/8499). The webapp allows the monitoring of the pipeline vibration and temperature signal, the configuration of zones to configure different settings and allowed events, the
inquiry to a sql database (pyodbc) with the detected events in the past, and the monitoring of all the vibration activity during the day.

Here you have some recording of the first draft of the app, as you can see the panel apps are simple as compared to the ones shown in the discourse forum.

I want to thanks to all the panel people and mainly to @philippjfr and @Marc by their work.

Best regards,
Néstor.

4 Likes

Thanks so much for sharing! Looks great. We should definitely work on integrating multi-page navigation somehow.

2 Likes

Congratulations @nghenzi2019.

Looks Great. Thanks for sharing.

Which table widget are you using? I’m asking because it has pagination.

What things would you be adding in the future?

1 Like

Hello,
I think it is simply Datatable. An example is shown here. Correct me if you already knew it.
https://panel.holoviz.org/gallery/external/DataTable.html#external-gallery-datatable

I saw this type of navigation in grafana, but it is in Visual studio code, youtube in the smart tv and so on. It is really comfortable to be able to switch across the different apps. I use iframes as can be seen below. It would be nice to have the same functionality that Visual Studio code, i.e. when you press the icon corresponding to the active app, the sidebar of the main app is shown or hidden, and when you press the icon corresponding to an inactive app you go to that app as it is shown in the example below.

For someone trying to dome the same stuff, here is a not so short example but fully functional. In each app function (app1,app2,app3,app4,app5) you can replace by the desired functionality

from functools import partial

import numpy as np
import panel as pn

from bokeh.models import ColumnDataSource
from bokeh.plotting import figure

template = """
{% extends base %}

{% block postamble %}
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">

<style>
    html, body {margin: 0; 
            height: 100%; 
            overflow-y: auto;
            background-color: white;}	

    .icon-bar { width: 80px;
            background-color: #555;
            height: 100%;}

    .icon-bar .icon {display: block;
            text-align: center;
            padding: 8px;
            transition: all 0.3s ease;
            color: white;
            font-size: 28px;
            text-size-adjust: none;
            height: 54px; }

    .icon-bar .icon:hover {background-color: #000;
            width: 120 px;}

    .left{width:80px;
            float:left;
            height:100%;}

    .right{margin-left:80px;
    height:100%;}
</style>

{% endblock %}

{% block contents %}

<div class="main_div" style="height:100%">

  <div class="left">
    <div class="icon-bar" style="height:100%">
        <div class= 'icon' ><span></span></div>
        <div class= 'icon' ><span></span></div>
        <div class='icon' onclick="Redirect('app1')">  <i class="fa fa-trash"></i>  </div>
        <div class='icon' onclick="Redirect('app2')">  <i class="fa fa-globe"></i>  </div>
        <div class='icon' onclick="Redirect('app3')">  <i class="fa fa-minus-circle"></i>  </div>
        <div class='icon' onclick="Redirect('app4')">  <i class="fa fa-envelope"></i>  </div>
        <div class='icon' onclick="Redirect('app5')">  <i class="fa fa-search"></i>  </div>
    </div>
  </div>

  <div class="right">
      <iframe id="iframe" style="border: None;  
          width: 100%; height: 100%; overflow: hidden;"  
          src="http://localhost:5006/app1" title="W3Schools">
      </iframe>  
  </div>

  <script>
    function Redirect(app){
        document.getElementById('iframe').src = 'http://localhost:5006/'+app;
    }
  </script>

</div>
{% endblock %}
"""

def navigator():
    return pn.Template(template)


def update(source):
    data = np.random.randint(0, 2 ** 31, 10)
    source.data.update({"y": data})


def app1():
    source = ColumnDataSource({"x": range(10), "y": range(10)})
    p = figure()
    p.line(x="x", y="y", source=source)
    vanilla = pn.template.VanillaTemplate(title='app1: Vanilla')
    vanilla.main.append(pn.pane.Bokeh(p,sizing_mode='stretch_both'))
    cb = pn.state.add_periodic_callback(partial(update, source), 200, timeout=5000)
    return vanilla

def app2():
    source = ColumnDataSource({"x": range(10), "y": range(10)})
    p = figure()
    p.line(x="x", y="y", source=source)
    golden = pn.template.GoldenTemplate(title='app2: Golden')
    golden.main.append(pn.pane.Bokeh(p,sizing_mode='stretch_both'))
    cb = pn.state.add_periodic_callback(partial(update, source), 200, timeout=5000)
    return golden

def app3():
    source = ColumnDataSource({"x": range(10), "y": range(10)})
    p = figure()
    p.line(x="x", y="y", source=source)
    material = pn.template.MaterialTemplate(title='app3: Material')
    material.main.append(pn.pane.Bokeh(p,sizing_mode='stretch_both'))
    cb = pn.state.add_periodic_callback(partial(update, source), 200, timeout=5000)
    return material

def app4():
    source = ColumnDataSource({"x": range(10), "y": range(10)})
    p = figure()
    p.line(x="x", y="y", source=source)
    react = pn.template.ReactTemplate(title='app4: React')
    react.main[:5,:12] = pn.pane.Bokeh(p,sizing_mode='stretch_both')
    cb = pn.state.add_periodic_callback(partial(update, source), 200, timeout=5000)
    return react

def app5():
    source = ColumnDataSource({"x": range(10), "y": range(10)})
    p = figure()
    p.line(x="x", y="y", source=source)
    bootstrap = pn.template.BootstrapTemplate(title='app5: Bootstrap')
    bootstrap.main.append(pn.pane.Bokeh(p,sizing_mode='stretch_both'))
    cb = pn.state.add_periodic_callback(partial(update, source), 200, timeout=5000)
    return bootstrap

dict_apps = {'navigator':navigator,
            'app1':app1,
            'app2':app2,
            'app3':app3,
            'app4':app4,
            'app5':app5}

port_bokeh = 5006
pn.serve(dict_apps, port=port_bokeh)

and it works like this. The main problem with this approach is you do not have the real location of the app.

Yes, it is the datatable example slightly modified as shown below. As can be seen below, the table is completely responsive in width, hiding or showing the columns according to the width of the div container.

import panel as pn
from bokeh.sampledata.autompg import autompg

raw_css = """ div.dataTables_wrapper {
                    width: 100%;
                    height: 100%;
                    margin: 0 auto;
                } 


     """ 

css = ['https://cdn.datatables.net/1.10.22/css/jquery.dataTables.min.css',
        'https://cdn.datatables.net/fixedheader/3.1.7/css/fixedHeader.dataTables.min.css',
      'https://cdn.datatables.net/responsive/2.2.6/css/responsive.dataTables.min.css']

js = {
    '$': 'https://code.jquery.com/jquery-3.5.1.js',
    'DataTable': 'https://cdn.datatables.net/1.10.22/js/jquery.dataTables.min.js',
    'respo1': 'https://cdn.datatables.net/fixedheader/3.1.7/js/dataTables.fixedHeader.min.js',
    'respo2': 'https://cdn.datatables.net/responsive/2.2.6/js/dataTables.responsive.min.js'
}

pn.extension(css_files=css, js_files=js, raw_css=[raw_css])

pn.config.sizing_mode = 'stretch_width'


script = """
<script>
if (document.readyState === "complete") {
var table = $('#example').DataTable( {
            "responsive": true,
            "scrollCollapse": true,
            "paging":         true,
            "columnDefs": [{"className": "dt-left", "targets": "_all"}]
        });


} """

html = autompg.to_html().replace("""class="dataframe\"""",
            """ id="example" class="display nowrap" style="width:100%" """)

d = pn.pane.HTML(html+script, sizing_mode='stretch_width')
d.show()

and it works like this

1 Like

Hi @nghenzi2019

I’m updating the awesome-list and I would like to add a link to this app. What would be the best link to add?

Hi @Marc

The app is in a private network, so no access is possible. We are still improving it, but when it is finished I will put some copy of the functionality in heroku or in our public server.

Thanks for the consideration !
Best regards,
N

1 Like

Neat demo! I was curious whether the username/password page was part of panel or flask?

it is part of flask, i am using flask login. Today I see this package FastAPI Users for fast api.

I do not know how to achieve the same functionality of flask login with panel, due to that I use flask.

1 Like

Hello @nghenzi2019,

I am trying out flask + iframe as you did here, however I’m facing an issue.
Let’s say you serve your flask app on port 8080 and like in your example, bokeh/panel on 5006.

If you want to avoid your users to access directly the bokeh app, you should not add 5006 to the allowed urls. But then your iframe does not render. How did you go around this permission issue?

Would it be possible to show a sample of your Flask app.py?

I am thinking to hard code the script returned by Bokeh’s server_document and put it within an iframe, but that looks not very stable.

Thanks!

You’re right. We couldn’t solve that problem. The app is on a private intranet, so there is no security risk. We only gave the port address of the flask application, and we trust that they never type the port of the panel application (unknown for them). Nobody ever complained :slight_smile: … if you find a solution please share it.

I think a way to do it is adding these to panel serve:

pn.serve(
    ...,
    sign_sessions=True,
    secret_key=SECRET_KEY,
    generate_session_ids=False,
)

And then generate the bokeh-session-id inside the flask app.

I have done something like that here. Though, it is for fastapi and not flask.

1 Like

Nope, it didn’t work, the template is not taken into account. Not sure why it would.

The objects are simply listed under document.roots, and the 3 list like elements of the template (sidebar, header, main) are not within it at all, only their children.

What makes it worse is that when giving a “name” to a List Like which is at root level, pull_session returns a gibberish name rather than the one given, so you can’t even rebuild the template.

HOWEVER, if your named column is within an object itself, then the name is preserved. So you have to make a column of column of column.

It seems pull_session is not working as I would expect it.
Perhaps we need to get around it in a very different way, it starts being ridiculous when you have to bury 2 levels down your main object.

Here is my app code:

with pull_session(url=url) as session:
            print(f'Pulling session {session.id}')
            print(session.document)
            print('#'*39)
            print(type(session.document.roots))
            isithere = session.document.select(dict(name='main_column'))
            print('Is it here?', isithere) 
            for obj in session.document.roots:
                print(f'object {obj.id} has ')
                if hasattr(obj,'label'):
                    print(f'\t - label {obj.label}' )
                if hasattr(obj,'name'):
                    print(f'\t - name {obj.name}')
                print(f'\t - type {type(obj)}')
                if hasattr(obj, 'children'):
                    for child in obj.children:
                        print(f'\t -child {child.id} has ')
                        if hasattr(child,'label'):
                            print(f'\t\t - label {child.label}' )
                        if hasattr(child,'name'):
                            print(f'\t\t - name {child.name}')
                        print(f'\t\t - type {type(child)}')
                print('') 
            script = server_session(session_id=session.id, url=url)
            return render_template("embed.html", script=script)

This is what it returns:

Pulling session WCsD99vGaPQwZd7bDvFCebfVEsDiYmva7P2Zn3Ok5WBu
<bokeh.document.document.Document object at 0x7ffb0e721100>
#######################################
<class 'list'>
Is it here? []
object 1004 has 
         - name location
         - type <class 'panel.models.location.Location'>

object 1006 has 
         - name js_area
         - type <class 'panel.models.markup.HTML'>

object 1008 has 
         - name actions
         - type <class 'panel.models.reactive_html.ReactiveHTML'>

object 1009 has 
         - name busy_indicator
         - type <class 'panel.models.markup.HTML'>

object 1010 has 
         - name 140251592952560
         - type <class 'bokeh.models.layouts.Column'>
         -child 1011 has 
                 - name Markdown01594
                 - type <class 'panel.models.markup.HTML'>
         -child 1012 has 
                 - name None
                 - type <class 'bokeh.models.layouts.Tabs'>
         -child 2240 has 
                 - name Column01960
                 - type <class 'bokeh.models.layouts.Column'>
         -child 2255 has 
                 - name Column01593
                 - type <class 'bokeh.models.layouts.Column'>

object 2392 has 
         - name 140251592951744
         - type <class 'panel.models.markup.HTML'>

object 2459 has 
         - name 140251608851504
         - type <class 'panel.models.echarts.ECharts'>

object 2526 has 
         - name 140251608854288
         - type <class 'panel.models.echarts.ECharts'>

object 2593 has 
         - name 140251592951888
         - type <class 'panel.models.layout.Card'>
         -child 2594 has 
                 - name Row01947
                 - type <class 'bokeh.models.layouts.Row'>
         -child 2596 has 
                 - name None
                 - type <class 'bokeh.models.widgets.inputs.TextInput'>
         -child 2597 has 
                 - label Hinzufügen
                 - name None
                 - type <class 'bokeh.models.widgets.buttons.Button'>
         -child 2598 has 
                 - name Alert01585
                 - type <class 'panel.models.markup.HTML'>

object 2665 has 
         - label Hilfe
         - name 140251608401904
         - type <class 'bokeh.models.widgets.buttons.Button'>

note the gibberish names and the empty list from the .select method. Note also that widgets have their labels returned.

Now if I bury one level down my “main column”, here’s the output (“is it here?” returns something):

<class 'list'>
Is it here? [Column(id='1011', ...)]
object 1004 has 
         - name location
         - type <class 'panel.models.location.Location'>

object 1006 has 
         - name js_area
         - type <class 'panel.models.markup.HTML'>

object 1008 has 
         - name actions
         - type <class 'panel.models.reactive_html.ReactiveHTML'>

object 1009 has 
         - name busy_indicator
         - type <class 'panel.models.markup.HTML'>

object 1010 has 
         - name 140342120937600
         - type <class 'bokeh.models.layouts.Column'>
         -child 1011 has 
                 - name main_column
                 - type <class 'bokeh.models.layouts.Column'>

object 2393 has 
         - name 140342120936736
         - type <class 'panel.models.markup.HTML'>

object 2460 has 
         - name 140342136676016
         - type <class 'panel.models.echarts.ECharts'>

object 2527 has 
         - name 140342136676304
         - type <class 'panel.models.echarts.ECharts'>

object 2594 has 
         - name 140342120936880
         - type <class 'panel.models.layout.Card'>
         -child 2595 has 
                 - name Row01947
                 - type <class 'bokeh.models.layouts.Row'>
         -child 2597 has 
                 - name None
                 - type <class 'bokeh.models.widgets.inputs.TextInput'>
         -child 2598 has 
                 - label Hinzufügen
                 - name None
                 - type <class 'bokeh.models.widgets.buttons.Button'>
         -child 2599 has 
                 - name Alert01585
                 - type <class 'panel.models.markup.HTML'>

object 2666 has 
         - label Hilfe
         - name 140342136760208
         - type <class 'bokeh.models.widgets.buttons.Button'>

my tempalte definition looks like that…

dashboard = pn.template.FastListTemplate(title='blabla',
    main=pn.Column(pn.Column(pn.Column(
        #my various objects...,
        name='main_column'
        )))
)

I’m giving up.

Trying to pass the template directly raises “TypeError: no loader for this environment specified”, and there is no way to pass it after the template is created.

I see only two options to preserve the security aspect (not everybody can access everywhere):

  • port all our code outside of flask and only use panel serve with panel OAuth (hopefully not hitting another wall)
  • make our own template and forget about Panel’s templates. With that we’re sure it works.