App load time is 7s with Panel servable(), how to start optimizing it?

[SteveAKopias]

@philippjfr

%%prun
hv.render(combined)

Finally, I’ve managed to try this out, first 200 line:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   948028   10.792    0.000   57.044    0.000 parameterized.py:1277(_setup_params)
   603265   10.706    0.000   10.706    0.000 {method 'reduce' of 'numpy.ufunc' objects}
  3431699   10.326    0.000   19.883    0.000 parameterized.py:859(__set__)
 16858798    8.962    0.000   11.676    0.000 parameterized.py:837(__get__)
41508293/41458385    8.725    0.000   10.159    0.000 {built-in method builtins.isinstance}
 10091811    8.496    0.000   11.513    0.000 parameterized.py:2544(param)
   948028    7.126    0.000   90.773    0.000 parameterized.py:2506(__init__)
 33310420    5.577    0.000    5.577    0.000 {method 'get' of 'dict' objects}
18704698/15794852    5.539    0.000   42.819    0.000 {built-in method builtins.getattr}
1560937/1467108    5.120    0.000    9.164    0.000 copy.py:128(deepcopy)
3441379/3431699    4.360    0.000   25.471    0.000 parameterized.py:313(_f)
  2413554    3.951    0.000    7.488    0.000 parameterized.py:2278(get_param_descriptor)
  1569222    3.906    0.000   17.021    0.000 bases.py:328(prepare_value)
2849141/1901113    3.832    0.000   77.730    0.000 parameterized.py:1060(override_initialization)
   492002    3.690    0.000   10.406    0.000 util.py:874(isfinite)
   496897    3.633    0.000   64.395    0.000 options.py:466(__init__)
   512166    3.612    0.000    9.323    0.000 dimension.py:606(matches)
  4364549    3.373    0.000    4.330    0.000 parameterized.py:160(classlist)
   948028    3.337    0.000   16.867    0.000 parameterized.py:1271(_generate_name)
  1467108    3.103    0.000   12.784    0.000 parameterized.py:1346(_instantiate_param)
1384402/1320516    3.040    0.000    7.782    0.000 tree.py:216(__setattr__)
 10091811    3.015    0.000    3.015    0.000 parameterized.py:1142(__init__)
 11584572    2.806    0.000    2.847    0.000 {built-in method builtins.issubclass}
  1581411    2.798    0.000    3.671    0.000 parameterized.py:1480(objects)
  1099948    2.772    0.000    8.139    0.000 parameterized.py:1745(get_value_generator)
13167827/13167294    2.697    0.000    3.362    0.000 {built-in method builtins.hasattr}
2417128/2082328    2.676    0.000    4.510    0.000 model.py:824(_visit_value_and_its_immediate_references)
  2806210    2.600    0.000    3.735    0.000 util.py:374(tree_attribute)
  2240461    2.374    0.000    5.459    0.000 util.py:737(__call__)
   305405    2.298    0.000   13.118    0.000 parameterized.py:1694(get_param_values)
1513795/1491575    2.236    0.000   28.916    0.000 bases.py:182(themed_default)
5295758/5295682    2.228    0.000   24.014    0.000 {built-in method builtins.setattr}
1491402/1469182    2.194    0.000   33.908    0.000 descriptors.py:704(_get_default)
322200/80550    2.106    0.000   37.498    0.000 options.py:772(options)
   832218    2.044    0.000    3.055    0.000 container.py:178(validate)
   563850    2.035    0.000    3.866    0.000 options.py:745(<genexpr>)
2275396/2253176    1.977    0.000   36.326    0.000 descriptors.py:676(_get)
1608012/1590189    1.945    0.000   12.129    0.000 {built-in method builtins.any}
  9115003    1.883    0.000    1.883    0.000 {method 'endswith' of 'str' objects}
   341962    1.785    0.000   43.215    0.000 model.py:808(_visit_immediate_value_references)
   422440    1.683    0.000    2.870    0.000 functools.py:35(update_wrapper)
   396388    1.681    0.000    2.744    0.000 __init__.py:1398(_check_type)
  2015339    1.678    0.000    5.860    0.000 {built-in method builtins.sorted}
  1512075    1.620    0.000    3.875    0.000 copy.py:66(copy)
1491402/1469182    1.597    0.000   30.865    0.000 descriptors.py:584(instance_default)
  3625810    1.399    0.000    1.399    0.000 util.py:690(<genexpr>)
   145204    1.348    0.000   11.893    0.000 util.py:965(max_range)
2403531/2359227    1.335    0.000   37.331    0.000 descriptors.py:464(__get__)
  7481724    1.302    0.000    1.302    0.000 {method 'items' of 'dict' objects}
   771978    1.270    0.000    1.514    0.000 copy.py:242(_keep_alive)
1513795/1491575    1.250    0.000    8.360    0.000 bases.py:161(_copy_default)
   161100    1.200    0.000    7.717    0.000 options.py:733(find)
  1894502    1.197    0.000    1.393    0.000 parameterized.py:1020(_validate)
   409525    1.149    0.000    3.091    0.000 options.py:548(__getitem__)
247504/236040    1.145    0.000    1.233    0.000 {built-in method numpy.array}
   953085    1.140    0.000    9.215    0.000 parameterized.py:1265(_set_name)
    75033    1.096    0.000   46.657    0.001 options.py:82(lookup_options)
344844/48153    1.092    0.000   32.915    0.001 dimension.py:637(traverse)
  1149436    1.087    0.000    1.087    0.000 wrappers.py:138(__init__)
  1643709    1.069    0.000    1.069    0.000 has_props.py:228(accumulate_dict_from_superclasses)
  5663596    1.066    0.000    1.448    0.000 parameterized.py:1229(__iter__)
5118740/4744552    1.032    0.000    1.329    0.000 {built-in method builtins.len}
    80880    1.029    0.000    1.996    0.000 dataset.py:1300(_construct_dataarray)
   140490    1.021    0.000    6.909    0.000 nanfunctions.py:228(nanmin)
  3058732    1.021    0.000    1.497    0.000 has_props.py:664(themed_values)
  4750747    0.962    0.000    0.962    0.000 parameterized.py:2148(_get_param)
  4364549    0.956    0.000    0.956    0.000 inspect.py:488(getmro)
   751468    0.935    0.000    2.514    0.000 container.py:189(wrap)
   422440    0.930    0.000    4.091    0.000 parameterized.py:273(__get__)
  1666596    0.926    0.000    3.480    0.000 dimension.py:632(<genexpr>)
   109200    0.920    0.000   31.792    0.000 options.py:639(_merge_options)
1478897/1465907    0.909    0.000    9.200    0.000 bases.py:172(_raw_default)
   417291    0.907    0.000    1.762    0.000 container.py:74(validate)
     1844    0.901    0.000   44.685    0.024 model.py:55(collect_filtered_models)
   327210    0.898    0.000   27.711    0.000 options.py:261(__init__)
1782669/1781100    0.897    0.000    2.563    0.000 {built-in method builtins.all}
   461446    0.887    0.000    4.312    0.000 streams.py:795(contents)
   267036    0.877    0.000    1.948    0.000 version.py:309(parse)
   492002    0.874    0.000    0.977    0.000 util.py:1513(is_dask_array)
318591/20537    0.870    0.000    1.443    0.000 spaces.py:759(get_nested_dmaps)
     1872    0.862    0.000   54.575    0.029 plot.py:717(_compute_group_range)
   505318    0.861    0.000    1.021    0.000 __init__.py:964(_validate)
   625597    0.860    0.000    3.561    0.000 dimension.py:359(__eq__)
    44265    0.859    0.000   17.105    0.000 util.py:1708(stream_parameters)
   190785    0.833    0.000    2.593    0.000 warnings.py:130(filterwarnings)
45689/34643    0.826    0.000   67.946    0.002 __init__.py:196(pipelined_fn)
   140490    0.818    0.000    6.911    0.000 nanfunctions.py:343(nanmax)
   362581    0.799    0.000    3.496    0.000 util.py:681(allowable)
   292148    0.792    0.000   14.013    0.000 streams.py:400(contents)
   396847    0.791    0.000    3.681    0.000 __init__.py:1372(_validate)
587136/586286    0.791    0.000    3.960    0.000 has_props.py:273(__setattr__)
   267038    0.790    0.000    0.790    0.000 {method 'split' of 're.Pattern' objects}
   612493    0.769    0.000    1.046    0.000 __init__.py:1316(_validate)
  1513795    0.748    0.000    1.731    0.000 has_props.py:593(_overridden_defaults)
  3948188    0.748    0.000    0.748    0.000 {built-in method builtins.id}
  1031884    0.746    0.000    1.977    0.000 dimension.py:302(spec)
   177468    0.744    0.000    0.744    0.000 {built-in method _warnings.warn}
    63920    0.725    0.000   17.705    0.000 dimension.py:491(__init__)
  2543132    0.711    0.000    0.711    0.000 {method 'get' of 'mappingproxy' objects}
    80220    0.709    0.000   12.993    0.000 tree.py:231(__getattr__)
  5612420    0.679    0.000    0.679    0.000 {method 'isupper' of 'str' objects}
   461446    0.676    0.000    0.934    0.000 streams.py:800(<dictcomp>)
   751492    0.666    0.000    1.376    0.000 wrappers.py:289(__init__)
    80550    0.654    0.000   41.306    0.001 options.py:755(closest)
581840/573650    0.654    0.000    1.063    0.000 copy.py:200(_deepcopy_list)
  2043741    0.648    0.000    0.972    0.000 options.py:546(<genexpr>)
   109755    0.623    0.000   11.532    0.000 options.py:516(__call__)
160847/87021    0.601    0.000   35.473    0.000 tree.py:106(set_path)
   137793    0.589    0.000    1.542    0.000 dimension.py:901(dimensions)
   402506    0.582    0.000    0.888    0.000 options.py:561(<dictcomp>)
62976/28308    0.565    0.000   29.536    0.001 dimension.py:1350(__init__)
   292148    0.563    0.000    0.743    0.000 streams.py:403(<dictcomp>)
   145204    0.554    0.000    0.772    0.000 util.py:981(<listcomp>)
  2838602    0.550    0.000    0.550    0.000 {method 'append' of 'list' objects}
   466812    0.548    0.000    0.892    0.000 model.py:82(queue_one)
  3431699    0.533    0.000    0.533    0.000 parameterized.py:945(_post_setter)
    63920    0.527    0.000   19.419    0.000 dimension.py:849(__init__)
    76528    0.525    0.000    0.535    0.000 util.py:1124(int_to_roman)
104381/56518    0.508    0.000   34.599    0.001 tree.py:44(__init__)
428397/365660    0.501    0.000    1.864    0.000 bases.py:304(is_valid)
   518221    0.496    0.000    0.957    0.000 numeric.py:1816(isscalar)
    27300    0.496    0.000   33.330    0.001 options.py:697(__setattr__)
   402506    0.496    0.000    0.808    0.000 options.py:556(<dictcomp>)
  2951310    0.495    0.000    0.495    0.000 {method 'upper' of 'str' objects}
   419945    0.494    0.000    0.961    0.000 bases.py:502(validate)
   399439    0.491    0.000    1.328    0.000 container.py:121(wrap)
   461446    0.488    0.000    2.088    0.000 streams.py:799(<dictcomp>)
   281276    0.477    0.000    1.364    0.000 {method 'any' of 'numpy.generic' objects}
   190787    0.457    0.000    0.907    0.000 warnings.py:181(_add_filter)
374652/322281    0.451    0.000   14.993    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
170930/159972    0.447    0.000    2.678    0.000 either.py:104(validate)
   109200    0.438    0.000    9.688    0.000 options.py:269(__add__)
   905538    0.435    0.000    0.476    0.000 util.py:1856(<genexpr>)
    61752    0.432    0.000    6.894    0.000 options.py:504(filtered)
   982908    0.432    0.000    0.583    0.000 generic.py:30(_check)
   264270    0.428    0.000    2.561    0.000 version.py:333(_cmp)
   588753    0.427    0.000    0.538    0.000 {built-in method _abc._abc_instancecheck}
   408515    0.424    0.000    1.903    0.000 options.py:543(cyclic)
    25704    0.423    0.000   31.461    0.001 plot.py:1416(get_extents)
   160847    0.418    0.000    4.721    0.000 tree.py:113(<listcomp>)
  1997423    0.397    0.000    0.397    0.000 bases.py:534(validation_on)
    51047    0.394    0.000   14.339    0.000 util.py:1043(dimension_range)
  1240700    0.390    0.000    0.390    0.000 model.py:245(id)
   165195    0.389    0.000    0.501    0.000 tree.py:177(__getitem__)
   205888    0.377    0.000    1.227    0.000 __init__.py:772(__get__)
    90406    0.363    0.000    1.041    0.000 overlay.py:221(group)
   193986    0.363    0.000    0.611    0.000 re.py:289(_compile)
   408515    0.360    0.000    5.345    0.000 options.py:565(options)
   190787    0.355    0.000    0.395    0.000 warnings.py:458(__enter__)
   397944    0.350    0.000    0.728    0.000 wrappers.py:197(__init__)
  1433003    0.343    0.000    0.343    0.000 parameterized.py:1182(self_or_cls)
    21182    0.340    0.000   10.916    0.001 variable.py:1685(reduce)
   341962    0.338    0.000    0.541    0.000 query.py:90(match)
   192381    0.332    0.000    0.332    0.000 {method 'remove' of 'list' objects}
   399229    0.331    0.000    0.331    0.000 has_props.py:205(accumulate_from_superclasses)
   496897    0.330    0.000    0.330    0.000 options.py:488(<listcomp>)
    86669    0.329    0.000    3.524    0.000 dimension.py:950(get_dimension)
    61752    0.320    0.000    0.320    0.000 options.py:510(<dictcomp>)
    53901    0.317    0.000    0.548    0.000 parameterized.py:2570(__setstate__)
    80550    0.315    0.000   41.699    0.001 options.py:1255(lookup_options)
    59854    0.310    0.000   13.043    0.000 parameterized.py:2995(instance)
   226468    0.310    0.000    1.201    0.000 util.py:836(asarray)
  1945636    0.307    0.000    0.307    0.000 bases.py:281(validate)
  1100137    0.302    0.000    0.302    0.000 {method 'startswith' of 'str' objects}
574591/533531    0.300    0.000    2.066    0.000 either.py:107(<genexpr>)
    59288    0.300    0.000    1.013    0.000 settings.py:290(__call__)
   178159    0.292    0.000    0.793    0.000 options.py:669(__getitem__)
   422440    0.291    0.000    0.291    0.000 functools.py:65(wraps)
174064/87032    0.290    0.000    0.396    0.000 tree.py:70(path)
   267036    0.281    0.000    0.281    0.000 version.py:314(<listcomp>)
   718734    0.275    0.000    0.471    0.000 util.py:982(<genexpr>)
14560/455    0.271    0.000  124.768    0.274 spaces.py:1087(_execute_callback)
   545465    0.270    0.000    0.274    0.000 common.py:262(__setattr__)
   190787    0.267    0.000    0.302    0.000 warnings.py:477(__exit__)
   108632    0.266    0.000    0.549    0.000 dataarray.py:370(__init__)
   588753    0.265    0.000    0.803    0.000 abc.py:96(__instancecheck__)
    59276    0.262    0.000    1.305    0.000 serialization.py:239(make_id)
   260590    0.259    0.000    2.778    0.000 version.py:69(__ge__)
    67428    0.257    0.000    1.447    0.000 util.py:1850(make_path_unique)
   283617    0.257    0.000    0.360    0.000 <frozen importlib._bootstrap>:398(parent)
   205888    0.257    0.000    0.478    0.000 __init__.py:600(__get__)
119769/118859    0.256    0.000    1.171    0.000 copy.py:258(_reconstruct)
    90436    0.256    0.000    0.494    0.000 overlay.py:250(<setcomp>)
   264612    0.250    0.000    0.675    0.000 missing.py:133(_isna)
   108540    0.249    0.000    0.249    0.000 util.py:2014(is_nan)
14560/455    0.248    0.000  126.349    0.278 spaces.py:1280(__getitem__)
   303104    0.245    0.000    0.245    0.000 util.py:1080(<genexpr>)
    59252    0.243    0.000    0.357    0.000 has_props.py:676(apply_theme)
   246548    0.239    0.000    0.333    0.000 {method 'join' of 'str' objects}
   226544    0.238    0.000    0.348    0.000 tree.py:269(__iter__)
  1479380    0.234    0.000    0.234    0.000 bases.py:269(transform)
     5532    0.229    0.000    0.229    0.000 {built-in method nt.stat}
174952/101126    0.228    0.000    0.228    0.000 tree.py:143(_propagate)
62976/28308    0.227    0.000   29.644    0.001 overlay.py:139(__init__)
    36431    0.226    0.000    0.467    0.000 parameterized.py:1237(__contains__)
  1211634    0.225    0.000    0.225    0.000 {method 'keys' of 'dict' objects}
   137338    0.225    0.000    0.761    0.000 dimension.py:933(<listcomp>)
   316922    0.223    0.000    0.433    0.000 options.py:293(__bool__)
   292148    0.223    0.000    0.223    0.000 streams.py:402(<dictcomp>)
   119769    0.221    0.000    1.022    0.000 {method '__reduce_ex__' of 'object' objects}

Will experiment with this, but at first glance these ncalls values themselves look a bit suspicious…

[Marc]

Thanks for staying on this @SteveAKopias

[philippjfr]

@SteveAKopias That indeed looks quite suspicious. It seems like it’s creating a lot of Parameterized objects, e.g. spending 8.725 seconds solely on isinstance calls seems crazy. Not quite sure where to go from here, but I’ll think about it.

[SteveAKopias]

@Marc : Well, I kinda have to if I want to make it work :D, but thank you both for being with me on this.

@philippjfr : Yeah, that’s what I figured. Finally, I can afford to spend some time on this, so I will experiment with it and also try to create a dumbed down reproducible version.

[SteveAKopias]

I’ve just cleaned and restructured the code that was in pretty “rapid development” shape, a little bit hoping it would solve the issue in itself, no such luck, so continuing with reducing the script to a simpler form.

Meanwhile, I realized that the runtime grows every time I rerun the display(combined) cell or anything similar:

# After full kernel restart and running all the cells above once, then only running this cell:
# %%timeit -n 1 -r 1
# display(pn.Row(content, widgets))
# 1st run: 8.41 s
# 2nd run: 8.94 s # tested multiple times, the 2nd run does not increase by much
# 3rd run: 12.3 s # all the consecutive runs do
# 4th run: 16.2 s
# 5th run: 20.8 s
# 6th run: 23.9 s

# After full kernel restart and running all the cells above once, then only running this cell:
# %%timeit -n 1 -r 1
# hv.render(combined)
# 1st run: 15.6 s
# 2nd run: 24.5 s
# 3rd run: 38.4 s
# 4th run: 50.7 s

So I’m pretty sure it affected the prun result above too. Now I every restart the kernel every time before running the measured cell, here are the new results. Not that they are good, but at least now I know they are precise:

# %%prun -l 200
# display(pn.Row(content, widgets))

# After full kernel restart and running all the cells above once, then only running this cell:

output = """
         19735172 function calls (19043317 primitive calls) in 14.769 seconds

   Ordered by: internal time
   List reduced from 6403 to 200 due to restriction <200>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     8452    0.842    0.000    1.009    0.000 ffi.py:149(__call__)
2441815/2438121    0.399    0.000    0.481    0.000 {built-in method builtins.isinstance}
    32709    0.337    0.000    1.702    0.000 parameterized.py:1277(_setup_params)
   121773    0.325    0.000    0.548    0.000 parameterized.py:859(__set__)
   606066    0.291    0.000    0.375    0.000 parameterized.py:837(__get__)
    23370    0.291    0.000    0.291    0.000 {method 'reduce' of 'numpy.ufunc' objects}
198716/53562    0.284    0.000    0.612    0.000 copy.py:128(deepcopy)
   359538    0.242    0.000    0.337    0.000 parameterized.py:2544(param)
  1515900    0.217    0.000    0.223    0.000 {method 'get' of 'dict' objects}
    32709    0.214    0.000    2.673    0.000 parameterized.py:2506(__init__)
182805/33235    0.208    0.000    0.298    0.000 ir.py:313(_rec_list_vars)
     2386    0.204    0.000    0.204    0.000 win32.py:92(_winapi_test)
776930/657838    0.202    0.000    1.545    0.000 {built-in method builtins.getattr}
    68423    0.152    0.000    0.717    0.000 bases.py:328(prepare_value)
122581/121773    0.140    0.000    0.728    0.000 parameterized.py:313(_f)
      631    0.127    0.000    0.128    0.000 pprint.py:99(_safe_tuple)
    86570    0.126    0.000    0.239    0.000 parameterized.py:2278(get_param_descriptor)
98297/65588    0.120    0.000    2.285    0.000 parameterized.py:1060(override_initialization)
      265    0.117    0.000    0.120    0.000 encoder.py:204(iterencode)
    16118    0.110    0.000    0.306    0.000 util.py:874(isfinite)
    16640    0.104    0.000    1.830    0.000 options.py:466(__init__)
   155135    0.104    0.000    0.135    0.000 parameterized.py:160(classlist)
      917    0.103    0.000    0.121    0.000 analysis.py:91(liveness)
    16881    0.103    0.000    0.279    0.000 dimension.py:606(matches)
549374/549285    0.102    0.000    0.135    0.000 {built-in method builtins.hasattr}
    53236    0.097    0.000    0.388    0.000 parameterized.py:1346(_instantiate_param)
   359539    0.095    0.000    0.095    0.000 parameterized.py:1142(__init__)
97163/83396    0.093    0.000    0.155    0.000 model.py:824(_visit_value_and_its_immediate_references)
   441645    0.092    0.000    0.097    0.000 {built-in method builtins.issubclass}
    40745    0.092    0.000    0.270    0.000 parameterized.py:1745(get_value_generator)
    61040    0.091    0.000    0.120    0.000 parameterized.py:1480(objects)
64730/63636    0.087    0.000    1.093    0.000 bases.py:182(themed_default)
64009/62915    0.083    0.000    1.288    0.000 descriptors.py:704(_get_default)
    32709    0.082    0.000    0.469    0.000 parameterized.py:1271(_generate_name)
    35019    0.078    0.000    0.125    0.000 container.py:178(validate)
42710/40742    0.077    0.000    0.207    0.000 tree.py:216(__setattr__)
    10220    0.076    0.000    0.432    0.000 parameterized.py:1694(get_param_values)
204462/204344    0.074    0.000    0.856    0.000 {built-in method builtins.setattr}
96810/95716    0.074    0.000    1.378    0.000 descriptors.py:676(_get)
    77114    0.073    0.000    0.174    0.000 util.py:737(__call__)
    86572    0.072    0.000    0.103    0.000 util.py:374(tree_attribute)
67159/65841    0.068    0.000    0.715    0.000 {built-in method builtins.any}
85393/45878    0.068    0.000    0.215    0.000 {method 'format' of 'str' objects}
     5450    0.065    0.000    0.129    0.000 dataset.py:1300(_construct_dataarray)
81951/79642    0.064    0.000    0.386    0.000 {built-in method builtins.sorted}
    66746    0.064    0.000    0.186    0.000 copy.py:66(copy)
    13958    0.063    0.000    1.537    0.000 model.py:808(_visit_immediate_value_references)
    19369    0.061    0.000    0.114    0.000 options.py:745(<genexpr>)
64009/62915    0.060    0.000    1.170    0.000 descriptors.py:584(instance_default)
11068/2767    0.059    0.000    1.072    0.000 options.py:772(options)
   319933    0.056    0.000    0.056    0.000 {method 'endswith' of 'str' objects}
      527    0.054    0.000    0.061    0.000 analysis.py:81(def_reach)
    14471    0.051    0.000    0.086    0.000 functools.py:35(update_wrapper)
104772/102660    0.051    0.000    1.383    0.000 descriptors.py:464(__get__)
   312304    0.050    0.000    0.050    0.000 {method 'items' of 'dict' objects}
    39825    0.048    0.000    0.058    0.000 copy.py:242(_keep_alive)
295879/276115    0.048    0.000    0.059    0.000 {built-in method builtins.len}
   352097    0.047    0.000    0.047    0.000 {built-in method builtins.id}
64730/63636    0.047    0.000    0.316    0.000 bases.py:161(_copy_default)
    33725    0.045    0.000    0.169    0.000 copy.py:209(_deepcopy_tuple)
   276765    0.044    0.000    0.044    0.000 {method 'append' of 'list' objects}
        3    0.043    0.014    0.045    0.015 crs.py:138(__init__)
    72787    0.043    0.000    0.044    0.000 has_props.py:228(accumulate_dict_from_superclasses)
11795/10774    0.041    0.000    0.049    0.000 {built-in method numpy.array}
   132074    0.040    0.000    0.060    0.000 has_props.py:664(themed_values)
   115920    0.040    0.000    0.040    0.000 util.py:690(<genexpr>)
27075/26089    0.040    0.000    0.323    0.000 copy.py:200(_deepcopy_list)
    48159    0.039    0.000    0.039    0.000 wrappers.py:138(__init__)
    65575    0.038    0.000    0.045    0.000 parameterized.py:1020(_validate)
   249337    0.038    0.000    0.052    0.000 parameterized.py:1229(__iter__)
     4832    0.038    0.000    0.342    0.000 util.py:965(max_range)
  172/167    0.037    0.000    0.226    0.001 analysis.py:23(compute_use_defs)
72709/72414    0.036    0.000    0.157    0.000 {built-in method builtins.all}
22082/18800    0.035    0.000    0.464    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
    17742    0.035    0.000    0.125    0.000 container.py:74(validate)
    32879    0.035    0.000    0.278    0.000 parameterized.py:1265(_set_name)
     5534    0.034    0.000    0.230    0.000 options.py:733(find)
    31410    0.034    0.000    0.092    0.000 container.py:189(wrap)
29369/28846    0.034    0.000    0.284    0.000 has_props.py:273(__setattr__)
63300/62646    0.033    0.000    0.344    0.000 bases.py:172(_raw_default)
     2583    0.032    0.000    1.379    0.001 options.py:82(lookup_options)
       79    0.032    0.000    1.586    0.020 model.py:55(collect_filtered_models)
    12818    0.032    0.000    0.087    0.000 options.py:548(__getitem__)
    12724    0.032    0.000    0.106    0.000 instructions.py:13(__init__)
     5096    0.032    0.000    0.185    0.000 nanfunctions.py:228(nanmin)
35211/20111    0.031    0.000    0.137    0.000 bases.py:304(is_valid)
   155139    0.031    0.000    0.031    0.000 inspect.py:488(getmro)
11385/1808    0.030    0.000    0.955    0.001 dimension.py:637(traverse)
    65938    0.030    0.000    0.038    0.000 {built-in method _abc._abc_instancecheck}
   168671    0.030    0.000    0.030    0.000 parameterized.py:2148(_get_param)
    31277    0.029    0.000    0.049    0.000 bases.py:502(validate)
    21209    0.028    0.000    0.114    0.000 dimension.py:359(__eq__)
    13120    0.028    0.000    0.164    0.000 values.py:219(_to_string)
    14458    0.028    0.000    0.122    0.000 parameterized.py:273(__get__)
43633/19213    0.028    0.000    0.189    0.000 _utils.py:44(__str__)
    64714    0.028    0.000    0.066    0.000 has_props.py:593(_overridden_defaults)
11016/747    0.027    0.000    0.043    0.000 spaces.py:759(get_nested_dmaps)
    17474    0.027    0.000    0.032    0.000 __init__.py:964(_validate)
    53392    0.026    0.000    0.099    0.000 dimension.py:632(<genexpr>)
     8956    0.026    0.000    0.058    0.000 version.py:309(parse)
    11064    0.026    0.000    0.762    0.000 options.py:261(__init__)
     5092    0.026    0.000    0.185    0.000 nanfunctions.py:343(nanmax)
    31134    0.026    0.000    0.068    0.000 event.py:227(notify)
     6813    0.026    0.000    0.082    0.000 warnings.py:130(filterwarnings)
    14482    0.026    0.000    0.120    0.000 streams.py:795(contents)
     3620    0.026    0.000    0.872    0.000 options.py:639(_merge_options)
      725    0.025    0.000    0.025    0.000 {built-in method nt.stat}
    31422    0.025    0.000    0.050    0.000 wrappers.py:289(__init__)
     1582    0.025    0.000    0.483    0.000 util.py:1708(stream_parameters)
    16118    0.024    0.000    0.027    0.000 util.py:1513(is_dask_array)
    14486    0.024    0.000    0.035    0.000 __init__.py:1372(_validate)
    22126    0.024    0.000    0.034    0.000 __init__.py:1316(_validate)
11250/10331    0.024    0.000    0.164    0.000 either.py:104(validate)
     8958    0.024    0.000    0.024    0.000 {method 'split' of 're.Pattern' objects}
    97625    0.023    0.000    0.023    0.000 {method 'get' of 'mappingproxy' objects}
    65938    0.023    0.000    0.061    0.000 abc.py:96(__instancecheck__)
     9357    0.023    0.000    0.402    0.000 streams.py:400(contents)
    11592    0.023    0.000    0.101    0.000 util.py:681(allowable)
       79    0.023    0.000    1.368    0.017 plot.py:717(_compute_group_range)
32612/32594    0.023    0.000    0.048    0.000 _utils.py:54(get_reference)
    33725    0.022    0.000    0.121    0.000 copy.py:210(<listcomp>)
    17237    0.022    0.000    0.038    0.000 _utils.py:24(deduplicate)
1798/1363    0.022    0.000    3.132    0.002 __init__.py:196(pipelined_fn)
    31727    0.021    0.000    0.056    0.000 dimension.py:302(spec)
       22    0.021    0.001    0.021    0.001 {built-in method binascii.b2a_base64}
     2134    0.021    0.000    0.550    0.000 dimension.py:491(__init__)
     6046    0.021    0.000    0.021    0.000 {built-in method _warnings.warn}
   159324    0.021    0.000    0.021    0.000 {method 'extend' of 'list' objects}
40445/40336    0.021    0.000    0.056    0.000 {method 'join' of 'str' objects}
    83751    0.020    0.000    0.020    0.000 {method 'startswith' of 'str' objects}
"""

The same thing ordered by cumulative time:

# %%prun -l 200 -s cumulative
# display(pn.Row(content, widgets))

# After full kernel restart and running all the cells above once, then only running this cell:
output = """
         19692178 function calls (19000246 primitive calls) in 15.751 seconds

   Ordered by: cumulative time
   List reduced from 6403 to 200 due to restriction <200>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     32/1    0.001    0.000   15.757   15.757 {built-in method builtins.exec}
      3/1    0.000    0.000   15.755   15.755 display.py:131(display)
        1    0.000    0.000   15.689   15.689 formatters.py:89(format)
       11    0.000    0.000   15.689    1.426 decorator.py:228(fun)
       11    0.000    0.000   15.689    1.426 formatters.py:220(catch_format_error)
        1    0.000    0.000   15.642   15.642 formatters.py:949(__call__)
        1    0.000    0.000   15.642   15.642 viewable.py:529(_repr_mimebundle_)
        1    0.000    0.000   15.298   15.298 viewable.py:422(_render_model)
        1    0.000    0.000   15.232   15.232 viewable.py:466(get_root)
     20/1    0.000    0.000   15.192   15.192 base.py:109(_get_model)
     20/1    0.000    0.000   15.191   15.191 base.py:82(_get_objects)
        2    0.000    0.000   14.999    7.500 holoviews.py:223(_get_model)
        2    0.000    0.000   14.936    7.468 holoviews.py:273(_render)
        2    0.000    0.000   14.936    7.468 renderer.py:66(get_plot)
        2    0.000    0.000   14.936    7.468 renderer.py:206(get_plot)
   501/14    0.008    0.000    9.265    0.662 spaces.py:1280(__getitem__)
   501/14    0.008    0.000    9.216    0.658 spaces.py:1087(_execute_callback)
   501/14    0.004    0.000    9.216    0.658 spaces.py:667(__call__)
   124/14    0.000    0.000    9.215    0.658 __init__.py:1041(dynamic_operation)
       15    0.000    0.000    9.166    0.611 plot.py:980(update)
   124/14    0.001    0.000    8.398    0.600 __init__.py:1026(resolve)
       13    0.000    0.000    8.266    0.636 streams.py:146(trigger)
       13    0.000    0.000    8.264    0.636 plot.py:208(refresh)
       13    0.000    0.000    8.224    0.633 plot.py:252(_trigger_refresh)
       13    0.000    0.000    8.223    0.633 plot.py:437(__getitem__)
       13    0.002    0.000    8.223    0.633 element.py:2391(update_frame)
      124    0.001    0.000    7.832    0.063 __init__.py:1033(apply)
      124    0.001    0.000    7.826    0.063 __init__.py:1012(_process)
   182/14    0.002    0.000    6.700    0.479 spaces.py:206(dynamic_mul)
       33    0.000    0.000    5.462    0.166 util.py:242(initialize_dynamic)
  181/179    0.002    0.000    5.015    0.028 operation.py:126(_apply)
       24    0.000    0.000    5.012    0.209 operation.py:176(process_element)
        6    0.000    0.000    4.346    0.724 dispatcher.py:340(_compile_for_args)
     28/6    0.001    0.000    4.345    0.724 dispatcher.py:864(compile)
     12/6    0.000    0.000    4.343    0.724 dispatcher.py:78(compile)
     12/6    0.000    0.000    4.343    0.724 dispatcher.py:85(_compile_cached)
     12/6    0.001    0.000    4.343    0.724 dispatcher.py:100(_compile_core)
     12/6    0.000    0.000    4.343    0.724 compiler.py:580(compile_extra)
     12/6    0.000    0.000    4.338    0.723 compiler.py:347(compile_extra)
     12/6    0.000    0.000    4.335    0.722 compiler.py:410(_compile_bytecode)
     13/6    0.000    0.000    4.335    0.722 compiler.py:370(_compile_core)
     13/6    0.001    0.000    4.332    0.722 compiler_machinery.py:316(run)
  289/134    0.001    0.000    4.331    0.032 compiler_lock.py:32(_acquire_compile_lock)
  288/134    0.005    0.000    4.327    0.032 compiler_machinery.py:257(_runPass)
  864/402    0.001    0.000    4.320    0.011 compiler_machinery.py:261(check)
      102    0.001    0.000    3.806    0.037 plot.py:1265(_get_frame)
       28    0.000    0.000    3.805    0.136 util.py:255(get_plot_frame)
1798/1363    0.026    0.000    3.213    0.002 __init__.py:196(pipelined_fn)
       20    0.001    0.000    2.989    0.149 datashader.py:885(_process)
    32709    0.257    0.000    2.961    0.000 parameterized.py:2506(__init__)
    13/12    0.001    0.000    2.819    0.235 typed_passes.py:451(run_pass)
    13/12    0.001    0.000    2.814    0.235 typed_passes.py:360(run_pass)
       20    0.001    0.000    2.759    0.138 core.py:896(raster)
       20    0.000    0.000    2.730    0.137 resampling.py:277(resample_2d)
       20    0.000    0.000    2.724    0.136 resampling.py:482(_resample_2d)
98297/65588    0.147    0.000    2.518    0.000 parameterized.py:1060(override_initialization)
    16640    0.112    0.000    2.067    0.000 options.py:466(__init__)
   316/87    0.002    0.000    2.020    0.023 dimension.py:677(map)
   150/83    0.000    0.000    1.894    0.023 accessors.py:575(_dispatch_opts)
   150/83    0.001    0.000    1.894    0.023 accessors.py:637(_base_opts)
       83    0.001    0.000    1.886    0.023 dimension.py:1230(options)
       83    0.000    0.000    1.876    0.023 __init__.py:192(apply_groups)
       83    0.000    0.000    1.875    0.023 __init__.py:157(_apply_groups_to_backend)
    32709    0.354    0.000    1.865    0.000 parameterized.py:1277(_setup_params)
      222    0.003    0.000    1.853    0.008 accessors.py:30(pipelined_call)
      277    0.001    0.000    1.853    0.007 __init__.py:1221(map)
    13/12    0.000    0.000    1.821    0.152 lowering.py:131(lower)
       67    0.000    0.000    1.803    0.027 accessors.py:524(__call__)
        4    0.000    0.000    1.772    0.443 datashader.py:432(_process)
       48    0.000    0.000    1.760    0.037 accessors.py:625(<lambda>)
       83    0.003    0.000    1.734    0.021 options.py:1791(set_options)
        4    0.001    0.000    1.697    0.424 core.py:172(points)
        4    0.000    0.000    1.696    0.424 core.py:1161(bypixel)
      8/4    0.000    0.000    1.682    0.421 utils.py:102(__call__)
        4    0.000    0.000    1.682    0.421 pandas.py:15(pandas_pipeline)
        4    0.000    0.000    1.682    0.421 pandas.py:23(default)
       79    0.000    0.000    1.670    0.021 model.py:98(collect_models)
       79    0.035    0.000    1.669    0.021 model.py:55(collect_filtered_models)
    13/12    0.000    0.000    1.646    0.137 lowering.py:182(lower_normal_function)
       78    0.001    0.000    1.645    0.021 model.py:554(references)
776930/657838    0.217    0.000    1.627    0.000 {built-in method builtins.getattr}
    13/12    0.000    0.000    1.624    0.135 lowering.py:196(lower_function_body)
    90/54    0.005    0.000    1.623    0.030 lowering.py:220(lower_block)
    13958    0.067    0.000    1.616    0.000 model.py:808(_visit_immediate_value_references)
 1154/774    0.007    0.000    1.611    0.002 lowering.py:321(lower_inst)
       31    0.002    0.000    1.587    0.051 plot.py:598(compute_ranges)
       79    0.026    0.000    1.548    0.020 plot.py:717(_compute_group_range)
        4    0.000    0.000    1.512    0.378 points.py:187(extend)
       15    0.000    0.000    1.500    0.100 element.py:749(_update_plot)
     2583    0.005    0.000    1.471    0.001 plot.py:290(lookup_options)
       68    0.001    0.000    1.468    0.022 plots.py:90(select)
     2583    0.034    0.000    1.466    0.001 options.py:82(lookup_options)
104772/102660    0.051    0.000    1.451    0.000 descriptors.py:464(__get__)
96810/95716    0.078    0.000    1.449    0.000 descriptors.py:676(_get)
     13/7    0.000    0.000    1.445    0.206 typed_passes.py:97(run_pass)
     13/7    0.000    0.000    1.445    0.206 typed_passes.py:61(type_inference_stage)
       83    0.004    0.000    1.436    0.017 options.py:1623(create_custom_trees)
     13/7    0.000    0.000    1.396    0.199 typeinfer.py:1052(propagate)
    26/14    0.004    0.000    1.395    0.100 typeinfer.py:141(propagate)
       15    0.001    0.000    1.363    0.091 element.py:803(_update_ranges)
64009/62915    0.089    0.000    1.356    0.000 descriptors.py:704(_get_default)
       15    0.000    0.000    1.342    0.089 element.py:759(_update_labels)
  422/300    0.004    0.000    1.287    0.004 typeinfer.py:568(resolve)
  462/338    0.001    0.000    1.279    0.004 context.py:187(resolve_function_type)
  462/338    0.002    0.000    1.276    0.004 context.py:231(_resolve_user_function_type)
     2767    0.008    0.000    1.272    0.000 options.py:1255(lookup_options)
  422/300    0.001    0.000    1.272    0.004 typeinfer.py:1500(resolve_call)
       15    0.000    0.000    1.269    0.085 element.py:763(<dictcomp>)
       30    0.002    0.000    1.269    0.042 element.py:647(_axis_properties)
     2767    0.021    0.000    1.262    0.000 options.py:755(closest)
        1    0.000    0.000    1.249    1.249 parfor_lowering.py:28(_lower_parfor_parallel)
64009/62915    0.064    0.000    1.229    0.000 descriptors.py:584(instance_default)
        2    0.000    0.000    1.212    0.606 datashader.py:1522(_process)
  160/106    0.001    0.000    1.177    0.011 typeinfer.py:558(__call__)
4975/2703    0.017    0.000    1.162    0.000 tree.py:106(set_path)
       15    0.001    0.000    1.156    0.077 plot.py:1838(get_extents)
     22/6    0.000    0.000    1.153    0.192 functions.py:521(get_call_type)
     22/6    0.000    0.000    1.153    0.192 dispatcher.py:300(get_call_template)
64730/63636    0.091    0.000    1.148    0.000 bases.py:182(themed_default)
11068/2767    0.064    0.000    1.143    0.000 options.py:772(options)
3321/1822    0.015    0.000    1.136    0.001 tree.py:44(__init__)
      906    0.016    0.000    1.103    0.001 options.py:697(__setattr__)
        2    0.001    0.001    1.099    0.549 datashader.py:1479(_process)
  312/157    0.004    0.000    1.072    0.007 operation.py:197(__call__)
        1    0.000    0.000    1.060    1.060 parfor_lowering.py:850(_create_gufunc_for_parfor_body)
     3620    0.030    0.000    1.055    0.000 options.py:639(_merge_options)
        1    0.000    0.000    1.043    1.043 compiler.py:609(compile_ir)
        1    0.000    0.000    1.042    1.042 compiler.py:355(compile_ir)
        1    0.000    0.000    1.042    1.042 compiler.py:417(_compile_ir)
     8452    0.852    0.000    1.024    0.000 ffi.py:149(__call__)
11385/1808    0.036    0.000    1.012    0.001 dimension.py:637(traverse)
       15    0.003    0.000    0.977    0.065 plot.py:1798(_get_subplot_extents)
      842    0.013    0.000    0.976    0.001 plot.py:1416(get_extents)
204462/204344    0.078    0.000    0.910    0.000 {built-in method builtins.setattr}
        1    0.000    0.000    0.895    0.895 element.py:2322(initialize_plot)
 1940/872    0.007    0.000    0.865    0.001 overlay.py:139(__init__)
 1940/872    0.016    0.000    0.861    0.001 dimension.py:1350(__init__)
        2    0.000    0.000    0.815    0.408 typed_passes.py:288(run_pass)
        2    0.000    0.000    0.815    0.408 parfor.py:2774(run)
    11064    0.028    0.000    0.813    0.000 options.py:261(__init__)
       83    0.003    0.000    0.803    0.010 options.py:1667(<dictcomp>)
      475    0.003    0.000    0.800    0.002 options.py:1533(apply_customizations)
    83/56    0.000    0.000    0.788    0.014 codegen.py:566(_ensure_finalized)
       15    0.000    0.000    0.788    0.053 codegen.py:673(finalize)
       13    0.000    0.000    0.771    0.059 spaces.py:1443(apply_map)
122581/121773    0.144    0.000    0.766    0.000 parameterized.py:313(_f)
67159/65841    0.072    0.000    0.761    0.000 {built-in method builtins.any}
    68423    0.158    0.000    0.753    0.000 bases.py:328(prepare_value)
       79    0.002    0.000    0.677    0.009 plot.py:909(_traverse_options)
      155    0.000    0.000    0.671    0.004 options.py:983(<lambda>)
      156    0.006    0.000    0.671    0.004 options.py:902(collapse_element)
      890    0.003    0.000    0.668    0.001 plot.py:918(lookup)
       13    0.000    0.000    0.654    0.050 cpu.py:197(get_executable)
       26    0.000    0.000    0.652    0.025 codegen.py:946(get_pointer_to_function)
     2134    0.017    0.000    0.646    0.000 dimension.py:849(__init__)
      802    0.005    0.000    0.646    0.001 raster.py:525(range)
198716/53562    0.296    0.000    0.642    0.000 copy.py:128(deepcopy)
 1381/950    0.007    0.000    0.634    0.001 options.py:616(__init__)
        7    0.000    0.000    0.593    0.085 ir_utils.py:1460(simplify)
     2134    0.024    0.000    0.589    0.000 dimension.py:491(__init__)
   121773    0.343    0.000    0.580    0.000 parameterized.py:859(__set__)
      389    0.004    0.000    0.546    0.001 __init__.py:498(range)
      647    0.003    0.000    0.538    0.001 plot.py:1311(get_padding)
      671    0.002    0.000    0.537    0.001 tree.py:160(__setitem__)
     1582    0.027    0.000    0.515    0.000 util.py:1708(stream_parameters)
2441205/2437511    0.425    0.000    0.514    0.000 {built-in method builtins.isinstance}
22082/18800    0.040    0.000    0.513    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
    32709    0.088    0.000    0.510    0.000 parameterized.py:1271(_generate_name)
     1002    0.002    0.000    0.498    0.000 util.py:1748(dimensionless_contents)
      464    0.009    0.000    0.497    0.001 xarray.py:252(range)
"""

On full reruns, there is only a slight change in some of the tottime/cumtime/percall values. What’s more interesting though that for the rows with the enormously high ncalls values (usually with the parameterized.py) there is a slight change in the ncalls values too. Not a lot, like in the first table the 3rd row has 32709 ncalls now, but sometimes it’s 32705 or something like that, and all the other similar values vary too. Which is weird, as I’m pretty sure it shouldn’t change how much something in a script gets called if the inputs did not change and it’s not dependent on time…

What was also surprising for me that the first function written by me to show up in the cumulative list by name is at the 268th line (not included above because of character limit) with a cumtime of 0.209. I wrote a pretty linear code, but I do have a lot of functions and it’s interesting to see that whatever the issue is it happens outside of them. I guess it’s a good idea to start to throw stuff into functions to see which one blows up the cumulative list…

[philippjfr]

Okay, a lot of it is indeed in the numba JIT compiler, which means that you should only incur that cost the first time the plots are rendered, subsequent renders should not incur the same cost. Otherwise I don’t really see any targets for you you to optimize but some possible candidates for internal optimizations in HoloViews and Panel.

I think you’re seeing the render time increase because using display means that Panel cannot correctly clean up the previously rendered models when a cell is rerun so it ends up updating multiple copies of the same objects.

[SteveAKopias]

So far I managed to isolate the parts that load the images (tiffs to be precise) and the waterpoints.

As a recap, this how combined looks like:

combined = (
        # PART 1
        hv.DynamicMap(get_tiles)

        # PART 2
        * hd.regrid(hv.DynamicMap(get_image_01)).apply.opts(cmap=pn_01_cmap, alpha=pn_01_alpha) 
        * hd.regrid(hv.DynamicMap(get_image_02)).apply.opts(cmap=pn_02_cmap, alpha=pn_02_alpha) 
        * hd.regrid(hv.DynamicMap(get_image_03)).apply.opts(cmap=pn_03_cmap, alpha=pn_03_alpha) 
        * hd.regrid(hv.DynamicMap(get_image_04)).apply.opts(cmap=pn_04_cmap, alpha=pn_04_alpha)
        * hd.regrid(hv.DynamicMap(get_image_05)).apply.opts(cmap=pn_05_cmap, alpha=pn_05_alpha)
        * hd.regrid(hv.DynamicMap(get_image_06)).apply.opts(cmap=pn_06_cmap, alpha=pn_06_alpha) 
        * hd.regrid(hv.DynamicMap(get_image_07)).apply.opts(cmap=pn_07_cmap, alpha=pn_07_alpha)
        * hd.regrid(hv.DynamicMap(get_image_08)).apply.opts(cmap=pn_08_cmap, alpha=pn_08_alpha)

        # PART 3
        * hd.regrid(hv.DynamicMap(get_white_bg_for_points)).options(cmap=['white']).apply.opts(alpha=pn_white_bg_for_points_alpha)
        * points_aggregated_categorical
        * points_aggregated_for_hover
        * hv_points

        # PART 4
        * hd.regrid(hv.DynamicMap(get_image_for_hover_tooltip))
)

• Part 1: Just returns a getattr(hv.element.tiles, pn_tile)() based on a Select.
• Part 2: Depending on the selectable layers checkboxlist it either actually loads and returns the actual image from disk based on other widgets, or returns an empty image as a placeholder.
• Part3: Has a basic placeholder as background, a categorical hd.datashade, a default hd.aggregate and a hv.Points based on the same DataFrame loaded from a CSV.
• Part4: Just a placeholder for displaying custom tooltips.

So there are only two possible type of datasources: the images are loaded from the tiffs and the points are loaded from a CSV.

So I added a few variables to control these:

# If dummy_image_scale==True, it gives back generated dummy images instead of 
#     loading the tiffs.
# dummy_image_scale defines the size of the image, 100(%) meaning 100x250
# hv.Image(np.ones((dummy_image_scale, int(dummy_image_scale *2.5))), 
#     bounds=(left,bottom,right,top))
load_dummy_images = True
dummy_image_scale = 10

# If load_dummy_wp==True, it generates a waterpoints DataFrame with the same 
#     columns but with filler data instead of loading the csv
load_dummy_wp = True

# If wp_dummy_limit>0, it limits the length of the waterpoints DataFrame (real or 
#     dummy)
# If wp_dummy_limit==0, it completely disables anything related to waterpoints, 
#     including loading/generating data, manipulating it, generating aggregates 
#     and even removes the whole PART3 from combined
wp_dummy_limit = 130000 # default: 130000, the size of the original CSV

This means that I can now completely disable 1) touching the files 2) dealing with large datasets. I started with gradual changes, and again, restarting the kernel every time.

The code I ran:

%%prun -l 100
display(pn.Row(content, widgets))

(content are widgets are the pn containers, the former containing combined, the latter containing all the widgets, of course.)

And the results with the relevant settings (bold if not default):

Original settings

{‘load_dummy_images’: False, ‘dummy_image_scale’: 100,
‘load_dummy_wp’: False, ‘wp_dummy_limit’: 130000}
19309648 function calls (18620261 primitive calls) in 13.693 seconds

Dummy Images, real WP

{‘load_dummy_images’: True, ‘dummy_image_scale’: 100,
‘load_dummy_wp’: False, ‘wp_dummy_limit’: 130000}
15689963 function calls (15303463 primitive calls) in 11.841 seconds

Dummy Images & Dummy WP

{‘load_dummy_images’: True, ‘dummy_image_scale’: 100,
‘load_dummy_wp’: True, ‘wp_dummy_limit’: 130000}
15691597 function calls (15305097 primitive calls) in 11.399 seconds

Dummy Images & Dummy WP, Dummy Image Scale: 10%

{‘load_dummy_images’: True, ‘dummy_image_scale’: 10,
‘load_dummy_wp’: True, ‘wp_dummy_limit’: 130000}
15728239 function calls (15341594 primitive calls) in 12.093 seconds

Dummy both, Image Scale: 10%, WP Limit: 130

{‘load_dummy_images’: True, ‘dummy_image_scale’: 10,
‘load_dummy_wp’: True, ‘wp_dummy_limit’: 130}
16473182 function calls (16176164 primitive calls) in 11.786 seconds

Dummy Images, Image Scale: 10%, Waterpoints disabled

{‘load_dummy_images’: True, ‘dummy_image_scale’: 10,
‘load_dummy_wp’: True, ‘wp_dummy_limit’: 0}
7451420 function calls (7295576 primitive calls) in 5.589 seconds

I’ve ran the same tests also with hv.render(combined), that produced similar changes, but this time ranging from 31s (original settings) to 12s (dummy images downscaled, waterpoints disabled).

What I have learned from this:

• Disabling all the real image loading helped only about 7%. That’s good, because I was worried that if loading big data files causes the issue, there wouldn’t be any solution. Decreasing the dummy image size didn’t improve the results, although increasing (not logged) worsened it, so that 7% is most probably due to the actual resolution difference between the real and dummy images.
• Replacing the waterpoints DataFrame with a faux version, of course, did not result in any change, I wasn’t expecting it either.
• What’s surprising, that reducing the number of records in the waterpoints DataFrame to 0.1% (or even when reducing them to 1 single record) didn’t change anything either. Which is, again, good news, as it’s not the size of the dataset that is the problem, but this was unexpected.
• Completely eliminating everything that’s related to the waterpoint DataFrame results in an about 60% time reduction, so there is definitely something going on there. However there’s still a significant amount of time remaining.

Also, @philippjfr: what makes you say this is a numba issue? This is what I’ve got for the last two tests (everthing reduced to minimal and waterpoints disabled) above:

Dummy both, Image Scale: 10%, WP Limit: 130

   Ordered by: internal time
   List reduced from 4748 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   940822    0.423    0.000    0.556    0.000 parameterized.py:837(__get__)
    33054    0.357    0.000    1.814    0.000 parameterized.py:1277(_setup_params)
   123543    0.347    0.000    0.583    0.000 parameterized.py:859(__set__)
1644391/1642463    0.342    0.000    0.427    0.000 {built-in method builtins.isinstance}
   363350    0.259    0.000    0.355    0.000 parameterized.py:2544(param)
99332/66278    0.255    0.000    2.559    0.000 parameterized.py:1060(override_initialization)
  1564530    0.236    0.000    0.236    0.000 {method 'get' of 'dict' objects}
    33054    0.229    0.000    2.973    0.000 parameterized.py:2506(__init__)
813332/696487    0.220    0.000    1.621    0.000 {built-in method builtins.getattr}
    67396    0.159    0.000    0.774    0.000 bases.py:328(prepare_value)
124347/123543    0.147    0.000    0.773    0.000 parameterized.py:313(_f)
    20523    0.139    0.000    0.392    0.000 util.py:874(isfinite)
65311/55738    0.138    0.000    0.291    0.000 copy.py:128(deepcopy)
    87988    0.132    0.000    0.259    0.000 parameterized.py:2278(get_param_descriptor)
   207750    0.131    0.000    0.346    0.000 dimension.py:302(spec)
    27590    0.116    0.000    0.116    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   157603    0.112    0.000    0.143    0.000 parameterized.py:160(classlist)
    16645    0.111    0.000    2.044    0.000 options.py:466(__init__)
    16455    0.108    0.000    0.277    0.000 dimension.py:606(matches)
540569/540479    0.107    0.000    0.120    0.000 {built-in method builtins.hasattr}
   469554    0.104    0.000    0.105    0.000 {built-in method builtins.issubclass}
    55738    0.102    0.000    0.411    0.000 parameterized.py:1346(_instantiate_param)
96114/81924    0.098    0.000    0.164    0.000 model.py:824(_visit_value_and_its_immediate_references)
   363350    0.097    0.000    0.097    0.000 parameterized.py:1142(__init__)
    60555    0.095    0.000    0.125    0.000 parameterized.py:1480(objects)
    40435    0.095    0.000    0.279    0.000 parameterized.py:1745(get_value_generator)
   106603    0.095    0.000    0.468    0.000 dimension.py:359(__eq__)
63680/62610    0.091    0.000    1.158    0.000 bases.py:182(themed_default)
    33054    0.087    0.000    0.493    0.000 parameterized.py:1271(_generate_name)
62924/61854    0.085    0.000    1.359    0.000 descriptors.py:704(_get_default)
42713/40745    0.083    0.000    0.221    0.000 tree.py:216(__setattr__)
    34497    0.081    0.000    0.136    0.000 container.py:178(validate)
    10179    0.080    0.000    0.447    0.000 parameterized.py:1694(get_param_values)
212929/212850    0.078    0.000    0.920    0.000 {built-in method builtins.setattr}
95351/94281    0.078    0.000    1.454    0.000 descriptors.py:676(_get)
    86578    0.077    0.000    0.110    0.000 util.py:374(tree_attribute)
    74609    0.076    0.000    0.174    0.000 util.py:737(__call__)
    13691    0.068    0.000    1.615    0.000 model.py:808(_visit_immediate_value_references)
    64833    0.068    0.000    0.205    0.000 copy.py:66(copy)
61361/60129    0.067    0.000    0.495    0.000 {built-in method builtins.any}
     7764    0.066    0.000    0.545    0.000 util.py:965(max_range)
19883/18013    0.065    0.000    0.091    0.000 {built-in method numpy.array}
11012/2753    0.063    0.000    1.122    0.000 options.py:772(options)
    19271    0.063    0.000    0.120    0.000 options.py:745(<genexpr>)
71682/71551    0.062    0.000    0.262    0.000 {built-in method builtins.sorted}
62924/61854    0.061    0.000    1.236    0.000 descriptors.py:584(instance_default)
   318373    0.060    0.000    0.060    0.000 {method 'endswith' of 'str' objects}
    14750    0.055    0.000    0.093    0.000 functools.py:35(update_wrapper)
103428/101364    0.053    0.000    1.455    0.000 descriptors.py:464(__get__)
63680/62610    0.049    0.000    0.337    0.000 bases.py:161(_copy_default)
   283289    0.047    0.000    0.047    0.000 {method 'items' of 'dict' objects}
    32230    0.046    0.000    0.055    0.000 copy.py:242(_keep_alive)
    71865    0.045    0.000    0.045    0.000 has_props.py:228(accumulate_dict_from_superclasses)
219785/198812    0.045    0.000    0.059    0.000 {built-in method builtins.len}
   129963    0.043    0.000    0.063    0.000 has_props.py:664(themed_values)
   114840    0.042    0.000    0.042    0.000 util.py:690(<genexpr>)
    47314    0.042    0.000    0.042    0.000 wrappers.py:138(__init__)
     3760    0.041    0.000    0.082    0.000 dataset.py:1300(_construct_dataarray)
       80    0.041    0.001    2.659    0.033 plot.py:717(_compute_group_range)

Dummy Images, Image Scale: 10%, Waterpoints disabled

 7451420 function calls (7295576 primitive calls) in 5.589 seconds

   Ordered by: internal time
   List reduced from 4039 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
17006    0.194    0.000    0.955    0.000 parameterized.py:1277(_setup_params)
62718    0.188    0.000    0.309    0.000 parameterized.py:859(__set__)
699794/698666    0.142    0.000    0.165    0.000 {built-in method builtins.isinstance}
   262817    0.141    0.000    0.179    0.000 parameterized.py:837(__get__)
   181577    0.139    0.000    0.194    0.000 parameterized.py:2544(param)
17006    0.124    0.000    1.515    0.000 parameterized.py:2506(__init__)
364326/303210    0.108    0.000    0.848    0.000 {built-in method builtins.getattr}
37325    0.091    0.000    0.449    0.000 bases.py:328(prepare_value)
   576076    0.090    0.000    0.091    0.000 {method 'get' of 'dict' objects}
63129/62718    0.078    0.000    0.410    0.000 parameterized.py:313(_f)
44381    0.071    0.000    0.137    0.000 parameterized.py:2278(get_param_descriptor)
30948/26699    0.069    0.000    0.144    0.000 copy.py:128(deepcopy)
51120/34114    0.069    0.000    1.289    0.000 parameterized.py:1060(override_initialization)
 8797    0.061    0.000    1.090    0.000 options.py:466(__init__)
80197    0.061    0.000    0.079    0.000 parameterized.py:160(classlist)
   181578    0.055    0.000    0.055    0.000 parameterized.py:1142(__init__)
48314/42498    0.054    0.000    0.091    0.000 model.py:824(_visit_value_and_its_immediate_references)
283205/283134    0.054    0.000    0.061    0.000 {built-in method builtins.hasattr}
 7371    0.053    0.000    0.152    0.000 util.py:874(isfinite)
26699    0.053    0.000    0.207    0.000 parameterized.py:1346(_instantiate_param)
   223262    0.051    0.000    0.052    0.000 {built-in method builtins.issubclass}
34534/33650    0.050    0.000    0.650    0.000 bases.py:182(themed_default)
34149/33265    0.049    0.000    0.762    0.000 descriptors.py:704(_get_default)
19103    0.048    0.000    0.140    0.000 parameterized.py:1745(get_value_generator)
29078    0.048    0.000    0.063    0.000 parameterized.py:1480(objects)
17006    0.047    0.000    0.267    0.000 parameterized.py:1271(_generate_name)
10543    0.045    0.000    0.045    0.000 {method 'reduce' of 'numpy.ufunc' objects}
 6900    0.044    0.000    0.120    0.000 dimension.py:606(matches)
22482/21434    0.044    0.000    0.118    0.000 tree.py:216(__setattr__)
17035    0.044    0.000    0.070    0.000 container.py:178(validate)
52191/51307    0.043    0.000    0.814    0.000 descriptors.py:676(_get)
105391/105279    0.043    0.000    0.578    0.000 {built-in method builtins.setattr}
45548    0.042    0.000    0.059    0.000 util.py:374(tree_attribute)
38420/38289    0.038    0.000    0.180    0.000 {built-in method builtins.sorted}
 4576    0.038    0.000    0.221    0.000 parameterized.py:1694(get_param_values)
 7600    0.037    0.000    0.865    0.000 model.py:808(_visit_immediate_value_references)
34753    0.037    0.000    0.114    0.000 copy.py:66(copy)
5956/1489    0.036    0.000    0.642    0.000 options.py:772(options)
33029    0.035    0.000    0.082    0.000 util.py:737(__call__)
10423    0.035    0.000    0.066    0.000 options.py:745(<genexpr>)
  458    0.035    0.000    0.044    0.000 ffi.py:149(__call__)
   177366    0.034    0.000    0.034    0.000 {method 'endswith' of 'str' objects}
34149/33265    0.034    0.000    0.693    0.000 descriptors.py:584(instance_default)
27927/27382    0.031    0.000    0.260    0.000 {built-in method builtins.any}
58209/56505    0.029    0.000    0.801    0.000 descriptors.py:464(__get__)
34534/33650    0.028    0.000    0.217    0.000 bases.py:161(_copy_default)
40613    0.027    0.000    0.028    0.000 has_props.py:228(accumulate_dict_from_superclasses)
19023/18560    0.027    0.000    0.259    0.000 has_props.py:273(__setattr__)
 6695    0.026    0.000    0.044    0.000 functools.py:35(update_wrapper)
 2244    0.025    0.000    0.050    0.000 dataset.py:1300(_construct_dataarray)
26061    0.025    0.000    0.025    0.000 wrappers.py:138(__init__)
   145772    0.024    0.000    0.024    0.000 {method 'items' of 'dict' objects}
71076    0.023    0.000    0.035    0.000 has_props.py:664(themed_values)
14894    0.023    0.000    0.027    0.000 copy.py:242(_keep_alive)
131821    0.022    0.000    0.029    0.000 parameterized.py:1229(__iter__)
 9850    0.022    0.000    0.109    0.000 container.py:74(validate)
54740    0.021    0.000    0.021    0.000 util.py:690(<genexpr>)
34083    0.021    0.000    0.025    0.000 parameterized.py:1020(_validate)
 2978    0.020    0.000    0.133    0.000 options.py:733(find)

So even when it runs in 5s, when it does not touch any real data, when it only handles really downsized faux data, it still has tens or hundreds of thousands of calls to parameterized.py functions. I don’t use params directly, but I use a lot of panel widgets, so my best guess that maybe they somehow get tangled together and keep triggering each other?

Now that I removed real data as a requirement, I can move towards creating a reduced example, and also will experiment with removing some widgets and functions being dependent on them.

[SteveAKopias]

https://colab.research.google.com/drive/1j0hrAPS3aCxeZ1OhuG4oiN89BPsLo3ex?usp=sharing

I’m very impressed.

• Reduced the script to about 600 lines from 4000.
• Removed every real data from it, it’s only using dummy data now.
• Removed everything not strictly necessary to display combined.
• Removed every style, JS link, custom hover tooltip, HTML template, not even displaying any of the widgets.

The runtime did not change at all.

I kept, however, the core structure of the script:

• creating widgets
• loading spatial datasets
• loading waterpoints
• displaying combined (spatial datasets + waterpoints)

The next step is to start experimenting with the widgets, checking out what happens if I one by one remove them and we’ll see. Meanwhile, if anybody has the time and mental space to check out the code (even if I know that 600 lines are still not a comfortable size) and anything jumps out as an obvious source of error, I would be very grateful for some pointers.

[philippjfr]

Thank you so much for the detailed analysis, it was extremely valuable as a starting point. I’ve started working on some optimizations in the HoloViews and Param internals which I think will make a reasonable difference. Param is used extensively inside Panel and HoloViews so you shouldn’t be alarmed to see it pop up in the profiler but there do seem to be a good amount of low-level optimizations we can apply to reduce that by what I hope will be ~50% in your case.

[philippjfr]

@SteveAKopias If you could try out Optimize and clean up options system by philippjfr · Pull Request #4954 · holoviz/holoviews · GitHub and see if it makes a difference for your use case that would be massively appreciated.

[SteveAKopias]

Wow, thanks a lot!

There is an 18% improvement when running the dummy version on my laptop with the optimize_options, so this definitely helped, although the runtime is still around 10s with hundreds of thousands of calls. (I’ve also updated the Colab Notebook with this, and that had a 15% improvement goint from 13.78s to 11.76s on average.)

Running with the original holoviews locally:

 15428135 function calls (15085925 primitive calls) in 11.893 seconds

   Ordered by: internal time
   List reduced from 5139 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 6129    0.540    0.000    0.695    0.000 ffi.py:149(__call__)
1677076/1674620    0.351    0.000    0.496    0.000 {built-in method builtins.isinstance}
55041    0.317    0.000    0.738    0.000 colors.py:197(_to_rgba_no_colorcycle)
25408    0.287    0.000    1.451    0.000 parameterized.py:1277(_setup_params)
93775    0.269    0.000    0.452    0.000 parameterized.py:859(__set__)
   458655    0.242    0.000    0.308    0.000 parameterized.py:837(__get__)
   294779    0.215    0.000    0.297    0.000 parameterized.py:2544(param)
25408    0.188    0.000    2.290    0.000 parameterized.py:2506(__init__)
628296/536118    0.182    0.000    1.355    0.000 {built-in method builtins.getattr}
   981279    0.152    0.000    0.156    0.000 {method 'get' of 'dict' objects}
21758    0.140    0.000    0.140    0.000 {method 'reduce' of 'numpy.ufunc' objects}
52645    0.127    0.000    0.629    0.000 bases.py:328(prepare_value)
59437/43811    0.123    0.000    0.266    0.000 copy.py:128(deepcopy)
55041    0.122    0.000    0.886    0.000 colors.py:161(to_rgba)
94542/93775    0.115    0.000    0.601    0.000 parameterized.py:313(_f)
114088/112909    0.115    0.000    0.664    0.000 {built-in method builtins.any}
14698    0.112    0.000    0.306    0.000 util.py:874(isfinite)
66735    0.105    0.000    0.199    0.000 parameterized.py:2278(get_param_descriptor)
  214    0.104    0.000    1.034    0.005 colors.py:776(from_list)
76393/50985    0.097    0.000    1.945    0.000 parameterized.py:1060(override_initialization)
36737    0.089    0.000    0.260    0.000 parameterized.py:1745(get_value_generator)
   119645    0.087    0.000    0.112    0.000 parameterized.py:160(classlist)
54375    0.087    0.000    0.115    0.000 parameterized.py:1480(objects)
43805    0.086    0.000    0.350    0.000 parameterized.py:1346(_instantiate_param)
12405    0.083    0.000    1.521    0.000 options.py:466(__init__)
   294779    0.082    0.000    0.082    0.000 parameterized.py:1142(__init__)
421518/421433    0.082    0.000    0.112    0.000 {built-in method builtins.hasattr}
   346629    0.080    0.000    0.084    0.000 {built-in method builtins.issubclass}
76985/68183    0.077    0.000    0.131    0.000 model.py:824(_visit_value_and_its_immediate_references)
42094/40148    0.076    0.000    0.210    0.000 tree.py:216(__setattr__)
 9465    0.076    0.000    0.419    0.000 parameterized.py:1694(get_param_values)
85322    0.075    0.000    0.108    0.000 util.py:374(tree_attribute)
105578/105361    0.074    0.000    0.399    0.000 {built-in method builtins.all}
49765/48831    0.073    0.000    0.952    0.000 bases.py:182(themed_default)
  740    0.073    0.000    0.073    0.000 win32.py:92(_winapi_test)
25408    0.071    0.000    0.393    0.000 parameterized.py:1271(_generate_name)
49079/48145    0.071    0.000    1.115    0.000 descriptors.py:704(_get_default)
78574/77640    0.066    0.000    1.194    0.000 descriptors.py:676(_get)
26228    0.063    0.000    0.100    0.000 container.py:178(validate)
10298    0.063    0.000    0.158    0.000 dimension.py:606(matches)
19145    0.062    0.000    0.118    0.000 options.py:745(<genexpr>)
   198579    0.061    0.000    0.069    0.000 {built-in method _abc._abc_instancecheck}
   198579    0.060    0.000    0.129    0.000 abc.py:96(__instancecheck__)
165246/165153    0.059    0.000    0.718    0.000 {built-in method builtins.setattr}
   220160    0.059    0.000    0.206    0.000 colors.py:266(<genexpr>)
10940/2735    0.059    0.000    1.107    0.000 options.py:772(options)
   312895    0.059    0.000    0.059    0.000 {method 'endswith' of 'str' objects}
54534    0.059    0.000    0.134    0.000 util.py:737(__call__)
   275200    0.057    0.000    0.057    0.000 colors.py:276(<genexpr>)
  265    0.057    0.000    0.060    0.000 encoder.py:204(iterencode)
62845/61088    0.055    0.000    0.189    0.000 {built-in method builtins.sorted}
10593    0.055    0.000    1.335    0.000 model.py:808(_visit_immediate_value_references)
42622/8082    0.055    0.000    0.078    0.000 ir.py:313(_rec_list_vars)
49689    0.053    0.000    0.127    0.000 copy.py:66(copy)
   357684    0.053    0.000    0.053    0.000 {method 'append' of 'list' objects}
306558/289947    0.053    0.000    0.063    0.000 {built-in method builtins.len}
49079/48145    0.052    0.000    1.016    0.000 descriptors.py:584(instance_default)
13822    0.051    0.000    0.086    0.000 functools.py:35(update_wrapper)
55481/29913    0.051    0.000    0.161    0.000 {method 'format' of 'str' objects}
 3936    0.047    0.000    0.094    0.000 dataset.py:1300(_construct_dataarray)
84694/82894    0.044    0.000    1.202    0.000 descriptors.py:464(__get__)
10940/10045    0.042    0.000    0.048    0.000 {built-in method numpy.array}
   112730    0.040    0.000    0.040    0.000 util.py:690(<genexpr>)
 4678    0.040    0.000    0.361    0.000 util.py:965(max_range)
49765/48831    0.039    0.000    0.291    0.000 bases.py:161(_copy_default)
27389    0.039    0.000    0.047    0.000 copy.py:242(_keep_alive)
   234587    0.038    0.000    0.038    0.000 {method 'items' of 'dict' objects}
55885    0.036    0.000    0.036    0.000 has_props.py:228(accumulate_dict_from_superclasses)

Running with optimize_options locally (using conda --create new --clone old and pip install git+https://github.com/holoviz/holoviews.git@optimize_options on the new):

13021956 function calls (12717894 primitive calls) in 9.759 seconds

   Ordered by: internal time
   List reduced from 5138 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     6129    0.464    0.000    0.602    0.000 ffi.py:149(__call__)
1511266/1508810    0.307    0.000    0.450    0.000 {built-in method builtins.isinstance}
   416001    0.208    0.000    0.266    0.000 parameterized.py:837(__get__)
    55041    0.186    0.000    0.599    0.000 colors.py:197(_to_rgba_no_colorcycle)
548640/466334    0.155    0.000    1.166    0.000 {built-in method builtins.getattr}
     4678    0.148    0.000    0.154    0.000 util.py:917(<listcomp>)
    21758    0.133    0.000    0.133    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   177794    0.133    0.000    0.133    0.000 {method 'items' of 'dict' objects}
   179166    0.128    0.000    0.176    0.000 parameterized.py:2544(param)
     8900    0.126    0.000    0.769    0.000 parameterized.py:1277(_setup_params)
    55041    0.119    0.000    0.743    0.000 colors.py:161(to_rgba)
   771638    0.115    0.000    0.118    0.000 {method 'get' of 'dict' objects}
    47763    0.111    0.000    0.556    0.000 bases.py:328(prepare_value)
113570/112427    0.109    0.000    0.531    0.000 {built-in method builtins.any}
    14698    0.105    0.000    0.291    0.000 util.py:810(isfinite)
      214    0.100    0.000    0.884    0.004 colors.py:776(from_list)
    31865    0.092    0.000    0.163    0.000 parameterized.py:859(__set__)
    36699    0.085    0.000    0.247    0.000 parameterized.py:1745(get_value_generator)
42948/27322    0.084    0.000    0.299    0.000 copy.py:128(deepcopy)
    54301    0.081    0.000    0.107    0.000 parameterized.py:1480(objects)
     8900    0.076    0.000    1.152    0.000 parameterized.py:2506(__init__)
    85322    0.075    0.000    0.107    0.000 util.py:335(tree_attribute)
42094/40148    0.075    0.000    0.208    0.000 tree.py:223(__setattr__)
101695/101487    0.070    0.000    0.384    0.000 {built-in method builtins.all}
     9465    0.070    0.000    0.396    0.000 parameterized.py:1694(get_param_values)
      740    0.067    0.000    0.068    0.000 win32.py:92(_winapi_test)
66756/60572    0.065    0.000    0.110    0.000 model.py:824(_visit_value_and_its_immediate_references)
44985/44135    0.063    0.000    0.829    0.000 bases.py:182(themed_default)
    10290    0.060    0.000    0.153    0.000 dimension.py:596(matches)
   198250    0.060    0.000    0.127    0.000 abc.py:96(__instancecheck__)
44299/43449    0.059    0.000    0.970    0.000 descriptors.py:704(_get_default)
289225/289140    0.059    0.000    0.087    0.000 {built-in method builtins.hasattr}
   198250    0.059    0.000    0.068    0.000 {built-in method _abc._abc_instancecheck}
   220160    0.057    0.000    0.203    0.000 colors.py:266(<genexpr>)
70781/69931    0.056    0.000    1.037    0.000 descriptors.py:676(_get)
   275200    0.056    0.000    0.056    0.000 colors.py:276(<genexpr>)
    23516    0.055    0.000    0.088    0.000 container.py:178(validate)
      265    0.055    0.000    0.058    0.000 encoder.py:204(iterencode)
42622/8082    0.053    0.000    0.076    0.000 ir.py:313(_rec_list_vars)
   230426    0.051    0.000    0.054    0.000 {built-in method builtins.issubclass}
    13824    0.051    0.000    0.086    0.000 functools.py:35(update_wrapper)
   355614    0.051    0.000    0.051    0.000 {method 'append' of 'list' objects}
287399/270788    0.048    0.000    0.058    0.000 {built-in method builtins.len}
     9343    0.048    0.000    1.141    0.000 model.py:808(_visit_immediate_value_references)
   179166    0.048    0.000    0.048    0.000 parameterized.py:1142(__init__)
    27316    0.047    0.000    0.340    0.000 parameterized.py:1346(_instantiate_param)
55440/29872    0.047    0.000    0.149    0.000 {method 'format' of 'str' objects}
    45147    0.046    0.000    0.118    0.000 copy.py:66(copy)
44299/43449    0.045    0.000    0.884    0.000 descriptors.py:584(instance_default)
    44308    0.045    0.000    0.103    0.000 util.py:682(__call__)
     3936    0.043    0.000    0.086    0.000 dataset.py:1300(_construct_dataarray)
32632/31865    0.042    0.000    0.217    0.000 parameterized.py:313(_f)
     8325    0.041    0.000    0.249    0.000 options.py:453(__init__)
   112740    0.039    0.000    0.039    0.000 util.py:636(<genexpr>)
10940/10045    0.039    0.000    0.044    0.000 {built-in method numpy.array}
     4678    0.039    0.000    0.490    0.000 util.py:901(max_range)
76784/75152    0.038    0.000    1.041    0.000 descriptors.py:464(__get__)
    21333    0.038    0.000    0.072    0.000 parameterized.py:2278(get_param_descriptor)
26869/17969    0.037    0.000    0.992    0.000 parameterized.py:1060(override_initialization)
119760/119670    0.036    0.000    0.398    0.000 {built-in method builtins.setattr}
40306/38549    0.035    0.000    0.078    0.000 {built-in method builtins.sorted}
44985/44135    0.034    0.000    0.253    0.000 bases.py:161(_copy_default)

Based on this I’m even more sure that this is not about param or hv being inefficient. You assumed, there could be a 50% imporovement and we got third of that. I’m pretty sure your assumption was precise, meaning 2/3 of the total runtime is caused by something completely unnecessary stuff. My best guess would be that I used either .apply, @pn.depends or .link incorrectly somewhere and that created a runaway effect. So there are what I’m going to investigate next, although today I will have limited time for this.

[philippjfr]

Could you give me the profiling output sorted by cumulative time as well (%%prun -s cumulative)?

[SteveAKopias]

Sure, here you are:

15427017 function calls (15084816 primitive calls) in 12.494 seconds

   Ordered by: cumulative time
   List reduced from 5139 to 200 due to restriction <200>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      5/1    0.000    0.000   12.497   12.497 {built-in method builtins.exec}
      3/1    0.000    0.000   12.496   12.496 display.py:131(display)
        1    0.000    0.000   12.468   12.468 formatters.py:89(format)
       11    0.000    0.000   12.465    1.133 decorator.py:228(fun)
       11    0.000    0.000   12.465    1.133 formatters.py:220(catch_format_error)
        1    0.000    0.000   12.460   12.460 formatters.py:949(__call__)
        1    0.000    0.000   12.460   12.460 dimension.py:1310(_repr_mimebundle_)
        1    0.000    0.000   12.460   12.460 options.py:1389(render)
        1    0.000    0.000   12.460   12.460 display_hooks.py:274(pprint_display)
        1    0.000    0.000   12.451   12.451 display_hooks.py:235(display)
        3    0.000    0.000   12.451    4.150 display_hooks.py:138(wrapped)
        1    0.000    0.000   12.451   12.451 display_hooks.py:195(map_display)
        1    0.000    0.000   12.451   12.451 display_hooks.py:52(render)
        1    0.000    0.000   12.451   12.451 renderer.py:379(components)
        1    0.000    0.000   12.263   12.263 viewable.py:422(_render_model)
        1    0.000    0.000   12.215   12.215 viewable.py:466(get_root)
        1    0.000    0.000   12.181   12.181 base.py:109(_get_model)
      2/1    0.000    0.000   12.180   12.180 base.py:82(_get_objects)
        2    0.000    0.000   12.178    6.089 holoviews.py:223(_get_model)
        1    0.000    0.000   12.127   12.127 holoviews.py:273(_render)
        1    0.000    0.000   12.127   12.127 renderer.py:66(get_plot)
        1    0.000    0.000   12.127   12.127 renderer.py:206(get_plot)
       14    0.000    0.000    8.882    0.634 plot.py:980(update)
       13    0.001    0.000    6.869    0.528 streams.py:146(trigger)
       13    0.000    0.000    6.866    0.528 plot.py:208(refresh)
       13    0.000    0.000    6.825    0.525 plot.py:252(_trigger_refresh)
       13    0.000    0.000    6.823    0.525 plot.py:437(__getitem__)
       13    0.002    0.000    6.823    0.525 element.py:2391(update_frame)
   487/14    0.008    0.000    5.852    0.418 spaces.py:1280(__getitem__)
   487/14    0.009    0.000    5.805    0.415 spaces.py:1087(_execute_callback)
   487/14    0.004    0.000    5.804    0.415 spaces.py:667(__call__)
   110/14    0.000    0.000    5.804    0.415 __init__.py:1041(dynamic_operation)
   110/14    0.000    0.000    4.968    0.355 __init__.py:1026(resolve)
      110    0.001    0.000    4.649    0.042 __init__.py:1033(apply)
      110    0.001    0.000    4.643    0.042 __init__.py:1012(_process)
   182/14    0.002    0.000    4.190    0.299 spaces.py:206(dynamic_mul)
      102    0.001    0.000    2.957    0.029 plot.py:1265(_get_frame)
       28    0.000    0.000    2.955    0.106 util.py:255(get_plot_frame)
1461/1187    0.021    0.000    2.924    0.002 __init__.py:196(pipelined_fn)
       31    0.000    0.000    2.899    0.094 util.py:242(initialize_dynamic)
  180/178    0.002    0.000    2.784    0.016 operation.py:126(_apply)
       24    0.000    0.000    2.782    0.116 operation.py:176(process_element)
    25404    0.190    0.000    2.360    0.000 parameterized.py:2506(__init__)
        4    0.000    0.000    2.183    0.546 datashader.py:432(_process)
        5    0.000    0.000    2.158    0.432 dispatcher.py:340(_compile_for_args)
     25/5    0.001    0.000    2.156    0.431 dispatcher.py:864(compile)
     11/5    0.000    0.000    2.155    0.431 dispatcher.py:78(compile)
     11/5    0.000    0.000    2.155    0.431 dispatcher.py:85(_compile_cached)
     11/5    0.001    0.000    2.155    0.431 dispatcher.py:100(_compile_core)
     11/5    0.000    0.000    2.154    0.431 compiler.py:580(compile_extra)
     11/5    0.000    0.000    2.150    0.430 compiler.py:347(compile_extra)
     11/5    0.000    0.000    2.149    0.430 compiler.py:410(_compile_bytecode)
     11/5    0.000    0.000    2.149    0.430 compiler.py:370(_compile_core)
     11/5    0.003    0.000    2.146    0.429 compiler_machinery.py:316(run)
  242/110    0.001    0.000    2.144    0.019 compiler_lock.py:32(_acquire_compile_lock)
  242/110    0.005    0.000    2.141    0.019 compiler_machinery.py:257(_runPass)
  726/330    0.001    0.000    2.135    0.006 compiler_machinery.py:261(check)
   315/86    0.002    0.000    2.129    0.025 dimension.py:677(map)
        4    0.000    0.000    2.101    0.525 core.py:172(points)
        4    0.000    0.000    2.101    0.525 core.py:1161(bypixel)
      8/4    0.000    0.000    2.088    0.522 utils.py:102(__call__)
        4    0.000    0.000    2.088    0.522 pandas.py:15(pandas_pipeline)
        4    0.000    0.000    2.088    0.522 pandas.py:23(default)
        1    0.000    0.000    2.058    2.058 element.py:2322(initialize_plot)
76381/50977    0.101    0.000    2.011    0.000 parameterized.py:1060(override_initialization)
      276    0.001    0.000    1.960    0.007 __init__.py:1221(map)
        4    0.000    0.000    1.902    0.475 points.py:187(extend)
    12403    0.086    0.000    1.582    0.000 options.py:466(__init__)
       11    0.001    0.000    1.545    0.140 typed_passes.py:451(run_pass)
       11    0.001    0.000    1.539    0.140 typed_passes.py:360(run_pass)
     2565    0.005    0.000    1.516    0.001 plot.py:290(lookup_options)
     2565    0.035    0.000    1.511    0.001 options.py:82(lookup_options)
       14    0.001    0.000    1.504    0.107 element.py:1380(initialize_plot)
    25404    0.296    0.000    1.491    0.000 parameterized.py:1277(_setup_params)
     11/5    0.000    0.000    1.467    0.293 typed_passes.py:97(run_pass)
     11/5    0.000    0.000    1.466    0.293 typed_passes.py:61(type_inference_stage)
     11/5    0.000    0.000    1.459    0.292 typeinfer.py:1052(propagate)
    22/10    0.002    0.000    1.459    0.146 typeinfer.py:141(propagate)
   150/28    0.000    0.000    1.450    0.052 context.py:187(resolve_function_type)
   138/16    0.002    0.000    1.450    0.091 typeinfer.py:568(resolve)
     62/8    0.001    0.000    1.449    0.181 typeinfer.py:558(__call__)
   138/16    0.000    0.000    1.449    0.091 typeinfer.py:1500(resolve_call)
   150/28    0.001    0.000    1.449    0.052 context.py:231(_resolve_user_function_type)
     20/4    0.000    0.000    1.448    0.362 functions.py:521(get_call_type)
     20/4    0.000    0.000    1.447    0.362 dispatcher.py:300(get_call_template)
       74    0.000    0.000    1.431    0.019 model.py:98(collect_models)
       74    0.028    0.000    1.430    0.019 model.py:55(collect_filtered_models)
       73    0.001    0.000    1.412    0.019 model.py:554(references)
       14    0.001    0.000    1.408    0.101 element.py:803(_update_ranges)
628296/536118    0.186    0.000    1.404    0.000 {built-in method builtins.getattr}
       14    0.001    0.000    1.392    0.099 element.py:1342(_init_glyphs)
    10593    0.057    0.000    1.387    0.000 model.py:808(_visit_immediate_value_references)
       14    0.000    0.000    1.359    0.097 element.py:749(_update_plot)
       23    0.001    0.000    1.313    0.057 raster.py:83(get_data)
     2734    0.009    0.000    1.313    0.000 options.py:1255(lookup_options)
       66    0.001    0.000    1.305    0.020 plots.py:90(select)
     2734    0.022    0.000    1.302    0.000 options.py:755(closest)
        2    0.001    0.000    1.295    0.647 datashader.py:1522(_process)
       25    0.001    0.000    1.275    0.051 element.py:1828(_get_colormapper)
84694/82894    0.045    0.000    1.253    0.000 descriptors.py:464(__get__)
78574/77640    0.067    0.000    1.246    0.000 descriptors.py:676(_get)
       15    0.001    0.000    1.212    0.081 plot.py:1838(get_extents)
       14    0.000    0.000    1.197    0.085 element.py:759(_update_labels)
       23    0.001    0.000    1.194    0.052 util.py:887(process_cmap)
        2    0.001    0.001    1.183    0.592 datashader.py:1479(_process)
       55    0.001    0.000    1.181    0.021 <frozen importlib._bootstrap>:1002(_find_and_load)
       52    0.000    0.000    1.178    0.023 <frozen importlib._bootstrap>:967(_find_and_load_unlocked)
10936/2734    0.064    0.000    1.177    0.000 options.py:772(options)
49079/48145    0.076    0.000    1.164    0.000 descriptors.py:704(_get_default)
       30    0.002    0.000    1.161    0.039 plot.py:598(compute_ranges)
  310/156    0.004    0.000    1.158    0.007 operation.py:197(__call__)
       63    0.001    0.000    1.155    0.018 util.py:666(_list_cmaps)
        2    0.000    0.000    1.136    0.568 <frozen importlib._bootstrap>:659(_load_unlocked)
        2    0.000    0.000    1.136    0.568 <frozen importlib._bootstrap_external>:784(exec_module)
        2    0.000    0.000    1.129    0.565 <frozen importlib._bootstrap>:220(_call_with_frames_removed)
        1    0.005    0.005    1.129    1.129 __init__.py:1(<module>)
       14    0.000    0.000    1.121    0.080 element.py:763(<dictcomp>)
       28    0.002    0.000    1.121    0.040 element.py:647(_axis_properties)
       78    0.022    0.000    1.120    0.014 plot.py:717(_compute_group_range)
      214    0.002    0.000    1.092    0.005 __init__.py:72(mpl_cm)
      214    0.108    0.001    1.086    0.005 colors.py:776(from_list)
49079/48145    0.051    0.000    1.057    0.000 descriptors.py:584(instance_default)
   122/69    0.000    0.000    1.042    0.015 accessors.py:575(_dispatch_opts)
   122/69    0.001    0.000    1.042    0.015 accessors.py:637(_base_opts)
       69    0.001    0.000    1.035    0.015 dimension.py:1230(options)
       15    0.004    0.000    1.030    0.069 plot.py:1798(_get_subplot_extents)
       69    0.000    0.000    1.025    0.015 __init__.py:192(apply_groups)
       69    0.000    0.000    1.024    0.015 __init__.py:157(_apply_groups_to_backend)
      840    0.015    0.000    1.022    0.001 plot.py:1416(get_extents)
      207    0.003    0.000    1.006    0.005 accessors.py:30(pipelined_call)
49765/48831    0.076    0.000    0.994    0.000 bases.py:182(themed_default)
       53    0.000    0.000    0.951    0.018 accessors.py:524(__call__)
    55041    0.129    0.000    0.927    0.000 colors.py:161(to_rgba)
       34    0.000    0.000    0.904    0.027 accessors.py:625(<lambda>)
 1932/868    0.007    0.000    0.893    0.001 overlay.py:139(__init__)
 1932/868    0.018    0.000    0.890    0.001 dimension.py:1350(__init__)
       69    0.002    0.000    0.874    0.013 options.py:1791(set_options)
7948/1531    0.026    0.000    0.865    0.001 dimension.py:637(traverse)
4757/2489    0.017    0.000    0.818    0.000 tree.py:106(set_path)
2683/1398    0.013    0.000    0.791    0.001 tree.py:44(__init__)
       13    0.000    0.000    0.774    0.060 spaces.py:1443(apply_map)
    55041    0.327    0.000    0.772    0.000 colors.py:197(_to_rgba_no_colorcycle)
165246/165153    0.064    0.000    0.740    0.000 {built-in method builtins.setattr}
       69    0.003    0.000    0.728    0.011 options.py:1623(create_custom_trees)
114088/112910    0.120    0.000    0.701    0.000 {built-in method builtins.any}
      154    0.000    0.000    0.688    0.004 options.py:983(<lambda>)
      155    0.006    0.000    0.688    0.004 options.py:902(collapse_element)
     6129    0.527    0.000    0.680    0.000 ffi.py:149(__call__)
       73    0.002    0.000    0.677    0.009 plot.py:909(_traverse_options)
      884    0.003    0.000    0.668    0.001 plot.py:918(lookup)
     2089    0.018    0.000    0.657    0.000 dimension.py:849(__init__)
     8464    0.023    0.000    0.654    0.000 options.py:261(__init__)
    52645    0.133    0.000    0.653    0.000 bases.py:328(prepare_value)
94528/93761    0.117    0.000    0.620    0.000 parameterized.py:313(_f)
       11    0.000    0.000    0.611    0.056 lowering.py:131(lower)
     2089    0.024    0.000    0.598    0.000 dimension.py:491(__init__)
      645    0.004    0.000    0.548    0.001 plot.py:1311(get_padding)
      486    0.009    0.000    0.536    0.001 options.py:697(__setattr__)
1677024/1674568    0.375    0.000    0.532    0.000 {built-in method builtins.isinstance}
     1940    0.016    0.000    0.510    0.000 options.py:639(_merge_options)
      272    0.004    0.000    0.507    0.002 dimension.py:539(clone)
     1542    0.026    0.000    0.499    0.000 util.py:1708(stream_parameters)
       11    0.000    0.000    0.483    0.044 cpu.py:197(get_executable)
      974    0.002    0.000    0.481    0.000 util.py:1748(dimensionless_contents)
       22    0.000    0.000    0.481    0.022 codegen.py:946(get_pointer_to_function)
    62/43    0.000    0.000    0.480    0.011 codegen.py:566(_ensure_finalized)
       11    0.000    0.000    0.480    0.044 codegen.py:673(finalize)
    93761    0.279    0.000    0.469    0.000 parameterized.py:859(__set__)
       69    0.001    0.000    0.462    0.007 options.py:1667(<dictcomp>)
      265    0.002    0.000    0.460    0.002 options.py:1533(apply_customizations)
     1623    0.013    0.000    0.452    0.000 util.py:1043(dimension_range)
      630    0.007    0.000    0.442    0.001 plot.py:1345(_get_range_extents)
       11    0.000    0.000    0.440    0.040 lowering.py:182(lower_normal_function)
     9465    0.075    0.000    0.428    0.000 parameterized.py:1694(get_param_values)
       11    0.000    0.000    0.419    0.038 lowering.py:196(lower_function_body)
       47    0.004    0.000    0.417    0.009 lowering.py:220(lower_block)
105572/105355    0.078    0.000    0.416    0.000 {built-in method builtins.all}
     2346    0.012    0.000    0.415    0.000 parameterized.py:2995(instance)
    25404    0.074    0.000    0.414    0.000 parameterized.py:1271(_generate_name)
     8713    0.024    0.000    0.412    0.000 streams.py:400(contents)
       79    0.002    0.000    0.411    0.005 raster.py:266(__init__)
      657    0.006    0.000    0.402    0.001 lowering.py:321(lower_inst)
       33    0.001    0.000    0.395    0.012 codegen.py:654(add_ir_module)
     2450    0.021    0.000    0.389    0.000 tree.py:231(__getattr__)
     4678    0.041    0.000    0.383    0.000 util.py:965(max_range)
       20    0.002    0.000    0.366    0.018 datashader.py:885(_process)
    43801    0.085    0.000    0.355    0.000 parameterized.py:1346(_instantiate_param)
17199/13918    0.029    0.000    0.344    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
48463/47904    0.030    0.000    0.333    0.000 bases.py:172(_raw_default)
       85    0.001    0.000    0.333    0.004 __init__.py:1179(clone)
      461    0.001    0.000    0.325    0.001 tree.py:160(__setitem__)
   458593    0.251    0.000    0.324    0.000 parameterized.py:837(__get__)
    14698    0.118    0.000    0.322    0.000 util.py:874(isfinite)
       11    0.000    0.000    0.313    0.028 codegen.py:720(_finalize_final_module)
      640    0.004    0.000    0.310    0.000 raster.py:525(range)
       11    0.000    0.000    0.309    0.028 lowering.py:233(create_cpython_wrapper)
       11    0.000    0.000    0.309    0.028 cpu.py:154(create_cpython_wrapper)
49765/48831    0.043    0.000    0.308    0.000 bases.py:161(_copy_default)
   294751    0.219    0.000    0.304    0.000 parameterized.py:2544(param)
       52    0.000    0.000    0.292    0.006 raster.py:415(clone)

(I will be offline for most of the day, but feel free to play around with the Colab Notebook if you need more info, it works without any additional files or dependencies and provides the same results, just with slightly sower runtimes.)

[SteveAKopias]

To clarify, these are some examples where I’m not sure I used the tools correctly:

.link() and @pn.depends()

def split_checkboxes_to_individual_values(target, event):
    """
    Function used to link a checkboxgroup to individual checkboxes.
    Namely it will be used to enable/disable pn_is_active[category_name] values
    based on the selected values of pn_layers_all
    """
    category_name = target.name.split('__')[0]
    is_active = 1 if category_name in event.new else 0
    if target.value != str(is_active):
        target.value = str(is_active)

pn_layers_options = ['terraclimate', 'geography', 'population']
pn_layers_default_values = ['geography']
pn_layers_all = pn.widgets.CheckBoxGroup(name='Layers', 
    value=pn_layers_default_values, options=pn_layers_options, inline=False)

# These widgets are not displayed, but they are used to be able to avoid triggering 
# changes in every layer when only one layer was changed, as a function dependent 
# on aCheckBoxGroup would be triggered every time, even if the function is 
# interested in the state of only one checkbox.
pn_is_active = {}
for category_label, category_name in pn_layers_options.items():
    default_is_active_value = 1 if category_name in pn_layers_default_values else 0
    pn_is_active[category_name] = pn.widgets.TextInput(
        name=category_name + '__is_active', value=str(default_is_active_value))
    pn_layers_all.link(pn_is_active[category_name], 
        callbacks={'value': split_checkboxes_to_individual_values})

# The pn_is_active[*] widgets then used like this:
@pn.depends(
    pn_layer_is_active=pn_is_active['population'],
    pn_resolution=pn_resolutions.param.value,
    pn_date=pn_dates.param.value_throttled,
    pn_population_gender=pn_population_genders.param.value,
    pn_population_age_limit=pn_population_age_limits.param.value_throttled,
)
def get_image_population(
        pn_layer_is_active,
        pn_resolution,
        pn_date,
        pn_population_gender,
        pn_population_age_limit,
):
    """
    Processes the relevant widget values and loads a population image based on them.
    """
    category_name = 'population'
    layer_is_active = int(pn_layer_is_active)

    if layer_is_active:
        # load_population_image replaced:
        population_image=hv.Image(np.ones((100, 250)), 
            bounds=(mw_left+2000,mw_bottom,mw_right,mw_top+2000))
    # if the layer is turned off, returns an empty image
    else:
        population_image = malawi_nan_image.clone()
        
    return population_image

the .apply()s building on each other:

# setting up streams to use in determining zoom
    stream_rangexy_source_original_wpc_hv_points = hv.streams.RangeXY(source=original_wpc_hv_points)
    streams = [stream_rangexy_source_original_wpc_hv_points]
    
    # only work with the currently visible points
    def get_points_in_viewport(original_points, pn_is_active_waterpoints, x_range, y_range):
        """
        Gives back currently visible waterpoints.
        If the value of the pn_is_active_waterpoints Widget is False, it returns fake_points.
        If it is True, then if range is set, it returns the points in range and every point
        otherwise.
        """
        if pn_is_active_waterpoints:
            if x_range is None or y_range is None:
                return original_points
            else:
                return original_points[x_range, y_range]
        else:
            return fake_points

    original_wpc_hv_points = hv.Points(wp, ['easting', 'northing']).apply(
        get_original_wpc_hv_points,
        pn_date=pn_dates,
        pn_wp_expiry=pn_wp_expiries
    )

    points_in_viewport = original_wpc_hv_points.apply(
       get_points_in_viewport,
       pn_is_active_waterpoints=pn_is_active_waterpoints_chk, streams=streams
    )
    
    def get_wp_hv_points(points_in_viewport, pn_is_active_waterpoints=True, threshold=5000):
        """
        Based on the currently visible (fake or real) points, it either gives back fake 
        points to force displaying the legend, but otherwise nothing or all the points, 
        depending on their number being higher or lower than the threshold.
        Note: for some reason the fake points had to be recreated from data otherwise some 
        weird bug appeared.
        """
        if pn_is_active_waterpoints:
            if len(points_in_viewport) > threshold:
                ret = hv.Points(fake_points_data, ['easting', 'northing'])
                return ret
            else:
                return points_in_viewport.options(color_points_by__cutoff_status_latest_result) \
                    .options(marker=hv.dim('cutoff_status_latest_result').categorize({'Functional': 'plus', 'Needs repair': 'circle', 'Not functional': 'square', 'Unknown': 'triangle'}))
        else:
            return hv.Points(fake_points_data, ['easting', 'northing'])

    # get either fake or real Points to display
    wp_hv_points = points_in_viewport.apply(
        get_wp_hv_points,
        pn_is_active_waterpoints=pn_is_active_waterpoints_chk
    )
    
    def get_wp_points_for_aggregate(points_in_viewport, pn_is_active_waterpoints=True, threshold=5000):
        """
        Opposite logic than the get_wp_hv_points(). Based on the currently visible 
        (fake or real) points, it either returns fake Points or the real points, 
        depending on their number being *lower* or *higher* than the threshold.
        These points then can and will be used to generate aggregate data.
        """
        if pn_is_active_waterpoints:
            if len(points_in_viewport) <= threshold:
                ret = hv.Points(fake_points_data, ['easting', 'northing'])
                return ret
            else:
                return points_in_viewport
        else:
            ret = hv.Points(fake_points_data, ['easting', 'northing'])
            return ret
        
    # get either fake or real Points to aggregate
    wp_points_for_aggregate = points_in_viewport.apply(
        get_wp_points_for_aggregate,
        pn_is_active_waterpoints=pn_is_active_waterpoints_chk)

[SteveAKopias]

After trying everything and the kitchen sink with no changes, I split the whole script into two:

• One of them only deals with the waterpoints, everything else is removed.
• The other one only deals with the spatial datasets, everything else is removed.

The only result of this that now I had two smaller script, but speedwise they simply divided the runtime between them almost 50-50.

Which is weird, as the waterpoints mainly use .apply() to be dynamic and displayed by hv.Points(), hd.datashade() and hd.aggregate(), while the spatial datasets are displayed by hd.regrid(hv.DynamicMap(get_image_func)) that use @pn.depends() for being dynamic, so without testing I would have assumed they would have nothing in common, and yet.

So next I started to hack everything away from the waterpoint section and voila, now I have a minimal example without any widgets whatsoever that runs in 5s on Colab and in 3.7s on my laptop, even though this is only a tiny chunk of the original code with 10k dummy Points.

https://colab.research.google.com/drive/1J0AJdRNqlk7Rj6BndLbUfLu0AdiFzbFF?usp=sharing

import random, numpy as np, pandas as pd, holoviews.operation.datashader as hd
import holoviews as hv, bokeh as bk, datashader as ds, panel as pn, param

pn.extension()
hv.extension('bokeh', logo=False)

number_of_waterpoints = 10000
wp_status_categories = ['F', 'R', 'N', 'U']
wp_category_color_dict = {'F': 'blue', 'R': 'red', 'N': 'yellow', 'U': 'gray'}

# creating the starting wp DataFrame
wp = pd.DataFrame(np.arange(number_of_waterpoints), columns=['id'])
wp['lat'] = np.random.random(number_of_waterpoints)*(-7.8) -9.37
wp['long'] = np.random.random(number_of_waterpoints)*(35.88 -32.7) +32.7
wp.loc[:, 'east'], wp.loc[:, 'north'] = ds.utils.lnglat_to_meters(wp.long, wp.lat)
wp['result'] = pd.Series(random.choices(wp_status_categories, k=wp.shape[0]))
wp['result'] = wp['result'].astype(pd.CategoricalDtype(wp_status_categories))

wp_hv_points = hv.Points(wp, ['east', 'north'])

# setting up streams to use in determining zoom
stream_rangexy = hv.streams.RangeXY(source=wp_hv_points)

# creating shade to display
wp_aggregated = hd.datashade(wp_hv_points,
    aggregator=ds.count_cat('result'), color_key=wp_category_color_dict,
    min_alpha=0.5, width=100, height=100, streams=[stream_rangexy])

# creating aggregate to display Count in Tooltip
wp_hover_for_aggregated = hv.util.Dynamic(
    hd.aggregate(wp_hv_points, width=100, height=100, streams=[stream_rangexy]),
    operation=hv.Image).opts(tools=['hover'], alpha=0)
   
combined = (hv.element.tiles.OSM() * wp_aggregated * wp_hover_for_aggregated) 
#  * wp_hv_points

# New cell:
%%prun -l 100
hv.extension('bokeh', logo=False)
display(combined)

I tried, and it doesn’t even matter if I include the wp_hv_points itself or not. If I comment out wp_aggregated or wp_hover_for_aggregated, the runtime is between 3-4s.

%%prun:

 4053728 function calls (3906057 primitive calls) in 5.244 seconds

   Ordered by: internal time
   List reduced from 5031 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 5399    0.724    0.000    0.727    0.000 ffi.py:111(__call__)
54784    0.349    0.000    1.018    0.000 colors.py:193(_to_rgba_no_colorcycle)
54784    0.175    0.000    1.226    0.000 colors.py:157(to_rgba)
   273920    0.171    0.000    0.171    0.000 colors.py:277(<genexpr>)
55072    0.169    0.000    0.169    0.000 {method 'astype' of 'numpy.ndarray' objects}
  611    0.149    0.000    0.170    0.000 builder.py:854(call)
607124/606418    0.142    0.000    0.170    0.000 {built-in method builtins.isinstance}
57212/57038    0.127    0.000    0.130    0.000 {built-in method numpy.array}
  214    0.104    0.000    1.387    0.006 colors.py:782(from_list)
37638/7330    0.085    0.000    0.120    0.000 ir.py:313(_rec_list_vars)
59484/58528    0.080    0.000    0.137    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
48184/24167    0.069    0.000    0.280    0.000 {method 'format' of 'str' objects}
62431/62171    0.064    0.000    0.287    0.000 {built-in method builtins.any}
56712    0.043    0.000    0.126    0.000 <__array_function__ internals>:2(can_cast)
26326/11895    0.043    0.000    0.264    0.000 _utils.py:44(__str__)
   272079    0.040    0.000    0.040    0.000 {method 'append' of 'list' objects}
74507/62353    0.037    0.000    0.190    0.000 {built-in method builtins.getattr}
19889/19883    0.037    0.000    0.072    0.000 _utils.py:54(get_reference)
   30    0.036    0.001    0.038    0.001 encoder.py:204(iterencode)
 4560    0.035    0.000    0.084    0.000 weakref.py:395(__getitem__)
10842    0.029    0.000    0.093    0.000 values.py:212(__init__)
  352    0.028    0.000    0.032    0.000 analysis.py:91(liveness)
 2009    0.028    0.000    0.044    0.000 instructions.py:638(descr)
 7601    0.027    0.000    0.095    0.000 instructions.py:13(__init__)
 2126    0.026    0.000    0.026    0.000 {method 'index' of 'tuple' objects}
52370/52310    0.026    0.000    0.035    0.000 {built-in method builtins.hasattr}
 2248/936    0.022    0.000    0.161    0.000 context.py:180(get_meminfos)
152578/149103    0.022    0.000    0.024    0.000 {built-in method builtins.len}
18297/18202    0.022    0.000    0.056    0.000 {method 'join' of 'str' objects}
3302/2595    0.021    0.000    0.047    0.000 contextlib.py:116(__exit__)
 7860    0.021    0.000    0.231    0.000 values.py:219(_to_string)
29491    0.019    0.000    0.024    0.000 parameterized.py:837(__get__)
 5207    0.019    0.000    0.034    0.000 parameterized.py:859(__set__)
54784    0.019    0.000    0.033    0.000 colors.py:123(_is_nth_color)
 1090    0.019    0.000    0.111    0.000 parameterized.py:1277(_setup_params)
 3112    0.019    0.000    0.019    0.000 {method 'reduce' of 'numpy.ufunc' objects}
  112    0.017    0.000    0.024    0.000 __init__.py:1304(_clear_cache)
 7880    0.017    0.000    0.029    0.000 values.py:239(_get_reference)
9516/9201    0.016    0.000    0.021    0.000 abstract.py:121(__eq__)
10811    0.016    0.000    0.036    0.000 _utils.py:24(deduplicate)
 7588    0.016    0.000    0.109    0.000 bases.py:328(prepare_value)
9123/2970    0.016    0.000    0.041    0.000 copy.py:132(deepcopy)
11263    0.015    0.000    0.063    0.000 values.py:232(_set_name)
14431/11722    0.015    0.000    0.023    0.000 {built-in method builtins.hash}
   50    0.015    0.000    0.022    0.000 ir.py:1249(dump)
   79    0.015    0.000    0.027    0.000 __init__.py:69(<listcomp>)
17098    0.014    0.000    0.018    0.000 {built-in method _abc._abc_instancecheck}
25431/3789    0.014    0.000    0.047    0.000 bases.py:304(is_valid)
79180    0.014    0.000    0.018    0.000 {method 'get' of 'dict' objects}
 6619    0.014    0.000    0.025    0.000 inspect.py:2467(__init__)
6772/6456    0.013    0.000    0.163    0.000 descriptors.py:704(_get_default)
 7601    0.013    0.000    0.016    0.000 builder.py:338(_insert)
14982    0.013    0.000    0.017    0.000 parameterized.py:2544(param)
   72    0.013    0.000    0.013    0.000 {built-in method posix.stat}
 3302    0.013    0.000    0.014    0.000 contextlib.py:81(__init__)
20336    0.013    0.000    0.013    0.000 _utils.py:13(is_used)
 1368    0.013    0.000    0.032    0.000 util.py:870(isfinite)
  110    0.012    0.000    0.096    0.001 analysis.py:23(compute_use_defs)
20224    0.012    0.000    0.012    0.000 __init__.py:65(rgb_to_hex)
11129/8443    0.012    0.000    0.043    0.000 abstract.py:118(__hash__)
 6578    0.012    0.000    0.013    0.000 analysis.py:69(<genexpr>)
9232/7606    0.012    0.000    0.023    0.000 {built-in method builtins.sorted}
 2009    0.011    0.000    0.032    0.000 instructions.py:623(__init__)
  672    0.011    0.000    0.024    0.000 instructions.py:166(descr)
 1806    0.011    0.000    0.011    0.000 {method 'encode' of 'str' objects}
  224    0.010    0.000    0.026    0.000 function_base.py:23(linspace)
 1090    0.010    0.000    0.155    0.000 parameterized.py:2506(__init__)
9941/9625    0.010    0.000    0.177    0.000 descriptors.py:676(_get)
 1436    0.010    0.000    0.010    0.000 {method 'decode' of 'bytes' objects}
17637    0.010    0.000    0.010    0.000 abstract.py:96(key)
10836    0.010    0.000    0.010    0.000 {method 'match' of 're.Pattern' objects}
6842/6526    0.010    0.000    0.134    0.000 bases.py:182(themed_default)
11263    0.010    0.000    0.048    0.000 _utils.py:16(register)
56712    0.010    0.000    0.010    0.000 multiarray.py:468(can_cast)
22002    0.010    0.000    0.029    0.000 container.py:77(<genexpr>)
9076/7550    0.009    0.000    0.015    0.000 model.py:824(_visit_value_and_its_immediate_references)
  177    0.009    0.000    0.011    0.000 inspect.py:2750(__init__)
  611    0.008    0.000    0.021    0.000 instructions.py:64(__init__)
  266    0.008    0.000    0.011    0.000 values.py:467(__init__)
 2036    0.008    0.000    0.034    0.000 models.py:659(get_field_position)
6772/6456    0.008    0.000    0.144    0.000 descriptors.py:584(instance_default)
34527    0.008    0.000    0.014    0.000 {built-in method builtins.issubclass}
 1283    0.008    0.000    0.013    0.000 instructions.py:398(descr)
 4100    0.008    0.000    0.016    0.000 parameterized.py:2278(get_param_descriptor)
 7472    0.008    0.000    0.019    0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)
 3160    0.008    0.000    0.010    0.000 utils.py:377(stream_list)
 8111    0.007    0.000    0.020    0.000 copy.py:66(copy)
49373    0.007    0.000    0.010    0.000 parameterized.py:1229(__iter__)
  110    0.007    0.000    0.017    0.000 analysis.py:118(compute_dead_maps)
  148    0.007    0.000    0.034    0.000 npydecl.py:96(generic)
 3460    0.007    0.000    0.037    0.000 container.py:178(validate)
 6663    0.007    0.000    0.009    0.000 enum.py:289(__call__)
  150    0.007    0.000    0.101    0.001 functions.py:280(get_call_type)
2    0.007    0.004    0.007    0.004 {method 'read' of '_io.FileIO' objects}
5477/5207    0.007    0.000    0.045    0.000 parameterized.py:313(_f)
1    0.007    0.007    1.430    1.430 __init__.py:41(<module>)
10935/9477    0.007    0.000    0.110    0.000 {built-in method builtins.next}
   40    0.007    0.000    0.039    0.001 postproc.py:175(_patch_var_dels)
  628/100    0.007    0.000    0.008    0.000 packer.py:175(rec)
  426    0.007    0.000    0.016    0.000 dataset.py:1366(_construct_dataarray)

%%prun -s cumulative:

3777019 function calls (3654897 primitive calls) in 5.450 seconds

   Ordered by: cumulative time
   List reduced from 4903 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      7/1    0.000    0.000    5.451    5.451 {built-in method builtins.exec}
        1    0.000    0.000    5.451    5.451 <string>:2(<module>)
      2/1    0.000    0.000    5.285    5.285 display.py:141(display)
        1    0.000    0.000    5.280    5.280 formatters.py:91(format)
       12    0.000    0.000    5.279    0.440 formatters.py:213(catch_format_error)
        1    0.000    0.000    5.278    5.278 <decorator-gen-5>:1(__call__)
        1    0.000    0.000    5.278    5.278 formatters.py:946(__call__)
        1    0.000    0.000    5.278    5.278 dimension.py:1310(_repr_mimebundle_)
        1    0.000    0.000    5.278    5.278 options.py:1389(render)
        1    0.000    0.000    5.278    5.278 display_hooks.py:274(pprint_display)
        1    0.000    0.000    5.278    5.278 display_hooks.py:235(display)
        3    0.000    0.000    5.278    1.759 display_hooks.py:138(wrapped)
        1    0.000    0.000    5.278    5.278 display_hooks.py:195(map_display)
        1    0.000    0.000    5.278    5.278 display_hooks.py:52(render)
        1    0.000    0.000    5.278    5.278 renderer.py:379(components)
        1    0.000    0.000    5.213    5.213 viewable.py:422(_render_model)
        1    0.000    0.000    5.194    5.194 viewable.py:466(get_root)
        1    0.000    0.000    5.185    5.185 base.py:109(_get_model)
        1    0.000    0.000    5.184    5.184 base.py:82(_get_objects)
        1    0.000    0.000    5.172    5.172 holoviews.py:223(_get_model)
        1    0.000    0.000    5.157    5.157 holoviews.py:273(_render)
        1    0.000    0.000    5.157    5.157 renderer.py:66(get_plot)
        1    0.000    0.000    5.157    5.157 renderer.py:206(get_plot)
     13/2    0.000    0.000    2.827    1.414 spaces.py:1280(__getitem__)
     13/2    0.000    0.000    2.826    1.413 spaces.py:1087(_execute_callback)
     13/2    0.000    0.000    2.826    1.413 spaces.py:667(__call__)
      9/3    0.000    0.000    2.821    0.940 __init__.py:1041(dynamic_operation)
        2    0.000    0.000    2.818    1.409 spaces.py:206(dynamic_mul)
        9    0.000    0.000    2.814    0.313 __init__.py:1033(apply)
        9    0.000    0.000    2.813    0.313 __init__.py:1012(_process)
        4    0.000    0.000    2.789    0.697 operation.py:176(process_element)
      8/6    0.000    0.000    2.789    0.465 operation.py:126(_apply)
        7    0.000    0.000    2.687    0.384 util.py:242(initialize_dynamic)
        4    0.001    0.000    2.583    0.646 datashader.py:432(_process)
        4    0.000    0.000    2.494    0.624 core.py:172(points)
        4    0.001    0.000    2.494    0.624 core.py:1161(bypixel)
      8/4    0.000    0.000    2.480    0.620 utils.py:102(__call__)
        4    0.000    0.000    2.480    0.620 pandas.py:15(pandas_pipeline)
        4    0.000    0.000    2.480    0.620 pandas.py:23(default)
        2    0.000    0.000    2.414    1.207 plot.py:980(update)
        4    0.000    0.000    2.382    0.595 dispatcher.py:337(_compile_for_args)
    244/4    0.001    0.000    2.380    0.595 compiler_lock.py:29(_acquire_compile_lock)
     24/4    0.001    0.000    2.380    0.595 dispatcher.py:795(compile)
     10/4    0.000    0.000    2.379    0.595 dispatcher.py:77(compile)
     10/4    0.000    0.000    2.379    0.595 dispatcher.py:84(_compile_cached)
     10/4    0.001    0.000    2.379    0.595 dispatcher.py:99(_compile_core)
     10/4    0.000    0.000    2.379    0.595 compiler.py:601(compile_extra)
     10/4    0.000    0.000    2.367    0.592 compiler.py:350(compile_extra)
     10/4    0.000    0.000    2.362    0.591 compiler.py:420(_compile_bytecode)
     10/4    0.000    0.000    2.362    0.590 compiler.py:380(_compile_core)
     10/4    0.001    0.000    2.359    0.590 compiler_machinery.py:318(run)
   220/88    0.006    0.000    2.357    0.027 compiler_machinery.py:259(_runPass)
  660/264    0.001    0.000    2.351    0.009 compiler_machinery.py:263(check)
        1    0.000    0.000    2.172    2.172 element.py:2316(initialize_plot)
        4    0.000    0.000    2.139    0.535 points.py:187(extend)
        2    0.000    0.000    2.118    1.059 overlay.py:30(dynamic_mul)
        2    0.000    0.000    2.106    1.053 datashader.py:1497(_process)
  136/114    0.003    0.000    2.039    0.018 __init__.py:196(pipelined_fn)
        2    0.002    0.001    2.034    1.017 datashader.py:1454(_process)
        3    0.000    0.000    2.028    0.676 element.py:1378(initialize_plot)
        3    0.000    0.000    2.003    0.668 element.py:1340(_init_glyphs)
        2    0.000    0.000    1.971    0.985 raster.py:83(get_data)
        2    0.000    0.000    1.966    0.983 element.py:1826(_get_colormapper)
        7    0.000    0.000    1.961    0.280 <frozen importlib._bootstrap>:978(_find_and_load)
        6    0.000    0.000    1.961    0.327 <frozen importlib._bootstrap>:948(_find_and_load_unlocked)
        2    0.000    0.000    1.959    0.980 util.py:885(process_cmap)
        6    0.000    0.000    1.959    0.327 util.py:666(_list_cmaps)
        2    0.000    0.000    1.959    0.979 <frozen importlib._bootstrap>:663(_load_unlocked)
        2    0.000    0.000    1.959    0.979 <frozen importlib._bootstrap_external>:722(exec_module)
        2    0.000    0.000    1.956    0.978 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
        1    0.009    0.009    1.956    1.956 __init__.py:41(<module>)
    79/40    0.001    0.000    1.954    0.049 dimension.py:677(map)
       66    0.000    0.000    1.924    0.029 __init__.py:1221(map)
      214    0.003    0.000    1.918    0.009 __init__.py:72(mpl_cm)
      6/4    0.000    0.000    1.914    0.478 operation.py:197(__call__)
      214    0.122    0.001    1.908    0.009 colors.py:782(from_list)
       10    0.001    0.000    1.769    0.177 typed_passes.py:430(run_pass)
       10    0.001    0.000    1.765    0.176 typed_passes.py:347(run_pass)
    54784    0.211    0.000    1.714    0.000 colors.py:157(to_rgba)
     10/4    0.000    0.000    1.695    0.424 typed_passes.py:85(run_pass)
     10/4    0.000    0.000    1.694    0.424 typed_passes.py:50(type_inference_stage)
     10/4    0.000    0.000    1.683    0.421 typeinfer.py:1052(propagate)
     20/8    0.002    0.000    1.683    0.210 typeinfer.py:141(propagate)
   138/16    0.002    0.000    1.670    0.104 typeinfer.py:568(resolve)
     62/8    0.001    0.000    1.670    0.209 typeinfer.py:558(__call__)
   142/20    0.000    0.000    1.670    0.083 context.py:187(resolve_function_type)
   138/16    0.000    0.000    1.669    0.104 typeinfer.py:1498(resolve_call)
   142/20    0.001    0.000    1.669    0.083 context.py:231(_resolve_user_function_type)
     20/4    0.000    0.000    1.668    0.417 functions.py:504(get_call_type)
     20/4    0.000    0.000    1.668    0.417 dispatcher.py:297(get_call_template)
    54784    0.581    0.000    1.463    0.000 colors.py:193(_to_rgba_no_colorcycle)
      9/3    0.000    0.000    0.799    0.266 __init__.py:1026(resolve)
     5399    0.714    0.000    0.718    0.000 ffi.py:111(__call__)
       10    0.001    0.000    0.692    0.069 lowering.py:131(lower)
       10    0.000    0.000    0.603    0.060 cpu.py:195(get_executable)
       20    0.000    0.000    0.601    0.030 codegen.py:492(get_pointer_to_function)
    58/40    0.000    0.000    0.600    0.015 codegen.py:123(_ensure_finalized)
       10    0.000    0.000    0.600    0.060 codegen.py:217(finalize)
       10    0.000    0.000    0.462    0.046 lowering.py:182(lower_normal_function)
       10    0.000    0.000    0.419    0.042 lowering.py:196(lower_function_body)

Using holoviews@optimize_options or 1.14.3 made no difference in these results.

Based on this there are 3 possibilities:

1. I use both hd.datashade() and hd.aggregate() terribly wrong, in that case, I hope my error will be easily identifiable now.
2. There is a bug (in Datashader?) that causes this issue with both functions.
3. This is normal, this is how long these functions should run for the first time for 10k points.

I assume it’s not #3, as that would be weird. I hope for #1, as the quickest way to resolve this, however after all this experimentation I wouldn’t be surprised if it would be #2. The thing is, that this was only the waterpoint half of the code. I didn’t have the time to similarly reduce the other half to a minimal example (the current version available here on Colab), but that doesn’t use either of these function, only hd.regrid(), and that runs similarly slowly. So if there would be an issue with some core functionality all three of these functions rely on, that would explain everything. But at the same time, it is entirely possible that it’s just an accident that these two newly separated scripts run similarly slow and they have completely different problems.

[SteveAKopias]

Based on what I learned so far from the waterpoints, I decided to quickly create a minimal example for the spatial datasets with no widgets and no @pn.depends(), just the hd.regrid(DynamicMap()) parts. And the same thing happened, the script runs for 6.6s on Colab (4.3s locally).

The minimal script:
https://colab.research.google.com/drive/1C1xQOnKnV5MgOyJgIDTMuoVgo2jAG_7n?usp=sharing

import random, numpy as np, holoviews.operation.datashader as hd
import holoviews as hv, bokeh as bk, datashader as ds, panel as pn, param
pn.extension()
hv.extension('bokeh', logo=False)
left, right, top, bottom = 3641158, 3994334, -1935463, -1047720

def get_image_01():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))

def get_image_02():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))

def get_image_03():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))

def get_image_04():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))

def get_image_05():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))

def get_image_06():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))

def get_image_07():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))

def get_image_08():
    return hv.Image(np.ones((100, 250)), bounds=(left,bottom,right,top))


combined = (
    hv.element.tiles.OSM()
    * hd.regrid(hv.DynamicMap(get_image_01))
    * hd.regrid(hv.DynamicMap(get_image_02))
    * hd.regrid(hv.DynamicMap(get_image_03))
    * hd.regrid(hv.DynamicMap(get_image_04))
    * hd.regrid(hv.DynamicMap(get_image_05))
    * hd.regrid(hv.DynamicMap(get_image_06))
    * hd.regrid(hv.DynamicMap(get_image_07))
    * hd.regrid(hv.DynamicMap(get_image_08))  
)

# New cell:
%%prun -l 100
hv.extension('bokeh', logo=False)
display(combined)

%%prun:

5428532 function calls (5331887 primitive calls) in 6.664 seconds

   Ordered by: internal time
   List reduced from 3670 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    54784    0.659    0.000    1.446    0.000 colors.py:193(_to_rgba_no_colorcycle)
    55350    0.212    0.000    0.212    0.000 {method 'astype' of 'numpy.ndarray' objects}
    54784    0.194    0.000    1.675    0.000 colors.py:157(to_rgba)
   273920    0.193    0.000    0.193    0.000 colors.py:277(<genexpr>)
59769/59513    0.169    0.000    0.171    0.000 {built-in method numpy.array}
573435/572779    0.155    0.000    0.180    0.000 {built-in method builtins.isinstance}
      214    0.117    0.001    1.861    0.009 colors.py:782(from_list)
   122051    0.114    0.000    0.139    0.000 parameterized.py:837(__get__)
    19860    0.111    0.000    0.177    0.000 parameterized.py:859(__set__)
62536/59598    0.109    0.000    0.339    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
     5357    0.108    0.000    0.593    0.000 parameterized.py:1277(_setup_params)
     8670    0.098    0.000    0.098    0.000 {method 'reduce' of 'numpy.ufunc' objects}
      432    0.095    0.000    0.096    0.000 ffi.py:111(__call__)
215776/176135    0.090    0.000    0.595    0.000 {built-in method builtins.getattr}
    82853    0.086    0.000    0.116    0.000 parameterized.py:2544(param)
76634/76271    0.082    0.000    0.423    0.000 {built-in method builtins.any}
     5357    0.075    0.000    0.935    0.000 parameterized.py:2506(__init__)
     6038    0.074    0.000    0.184    0.000 util.py:870(isfinite)
       98    0.070    0.001    0.072    0.001 encoder.py:204(iterencode)
136627/136519    0.066    0.000    0.073    0.000 {built-in method builtins.hasattr}
17469/16650    0.056    0.000    0.117    0.000 tree.py:216(__setattr__)
16154/10797    0.056    0.000    0.793    0.000 parameterized.py:1060(override_initialization)
    23778    0.054    0.000    0.269    0.000 bases.py:328(prepare_value)
   262387    0.050    0.000    0.050    0.000 {method 'get' of 'dict' objects}
    54784    0.047    0.000    0.144    0.000 <__array_function__ internals>:2(can_cast)
17689/13682    0.047    0.000    0.114    0.000 copy.py:132(deepcopy)
21750/21077    0.044    0.000    0.513    0.000 descriptors.py:704(_get_default)
      926    0.043    0.000    0.048    0.000 parameterized.py:188(get_all_slots)
     4816    0.041    0.000    0.063    0.000 options.py:745(<genexpr>)
    20007    0.040    0.000    0.055    0.000 parameterized.py:1480(objects)
31084/28346    0.039    0.000    0.064    0.000 model.py:824(_visit_value_and_its_immediate_references)
    13682    0.039    0.000    0.160    0.000 parameterized.py:1346(_instantiate_param)
   114254    0.039    0.000    0.040    0.000 {built-in method builtins.issubclass}
   258677    0.037    0.000    0.037    0.000 {method 'append' of 'list' objects}
34264/33591    0.037    0.000    0.562    0.000 descriptors.py:676(_get)
54357/54282    0.036    0.000    0.370    0.000 {built-in method builtins.setattr}
20116/19860    0.034    0.000    0.224    0.000 parameterized.py:313(_f)
    13631    0.033    0.000    0.119    0.000 parameterized.py:1745(get_value_generator)
22065/21392    0.032    0.000    0.420    0.000 bases.py:182(themed_default)
     5357    0.030    0.000    0.141    0.000 parameterized.py:1271(_generate_name)
    82853    0.029    0.000    0.029    0.000 parameterized.py:1142(__init__)
      590    0.029    0.000    0.042    0.000 instructions.py:13(__init__)
    35388    0.029    0.000    0.041    0.000 util.py:370(tree_attribute)
    14339    0.028    0.000    0.062    0.000 parameterized.py:2278(get_param_descriptor)
     3453    0.027    0.000    0.184    0.000 parameterized.py:1694(get_param_values)
     1122    0.027    0.000    0.565    0.001 options.py:82(lookup_options)
21750/21077    0.027    0.000    0.454    0.000 descriptors.py:584(instance_default)
160743/155006    0.027    0.000    0.033    0.000 {built-in method builtins.len}
     1780    0.026    0.000    0.228    0.000 util.py:961(max_range)
    25979    0.026    0.000    0.032    0.000 parameterized.py:160(classlist)
    71665    0.025    0.000    0.025    0.000 {method 'items' of 'dict' objects}
     1362    0.025    0.000    0.061    0.000 dataset.py:1366(_construct_dataarray)
    10549    0.025    0.000    0.039    0.000 container.py:178(validate)
  682/594    0.024    0.000    1.085    0.002 __init__.py:196(pipelined_fn)
      566    0.023    0.000    0.068    0.000 function_base.py:23(linspace)
      544    0.023    0.000    0.023    0.000 boundingregion.py:298(lbrt)
      120    0.023    0.000    0.023    0.000 {built-in method bottleneck.reduce.nanmin}
     4866    0.023    0.000    0.594    0.000 model.py:808(_visit_immediate_value_references)
   131598    0.023    0.000    0.023    0.000 {method 'endswith' of 'str' objects}
     5302    0.022    0.000    0.039    0.000 functools.py:37(update_wrapper)
     5302    0.022    0.000    0.065    0.000 parameterized.py:273(__get__)
    22156    0.022    0.000    0.103    0.000 copy.py:66(copy)
    78851    0.021    0.000    0.027    0.000 parameterized.py:1229(__iter__)
    54784    0.021    0.000    0.035    0.000 colors.py:123(_is_nth_color)
13164/12819    0.020    0.000    0.172    0.000 has_props.py:273(__setattr__)
12383/12252    0.020    0.000    0.088    0.000 {built-in method builtins.sorted}
2408/1204    0.020    0.000    0.298    0.000 options.py:772(options)
5563/5272    0.020    0.000    0.035    0.000 copy.py:211(_deepcopy_list)
    12536    0.020    0.000    0.046    0.000 util.py:733(__call__)
      873    0.019    0.000    0.019    0.000 util.py:1120(int_to_roman)
      112    0.019    0.000    0.026    0.000 __init__.py:1304(_clear_cache)
22065/21392    0.018    0.000    0.163    0.000 bases.py:161(_copy_default)
38049/36753    0.018    0.000    0.557    0.000 descriptors.py:464(__get__)
     2136    0.018    0.000    0.386    0.000 options.py:466(__init__)
     6876    0.017    0.000    0.024    0.000 numeric.py:1816(isscalar)
       31    0.017    0.001    0.661    0.021 plot.py:717(_compute_group_range)
    25861    0.016    0.000    0.017    0.000 has_props.py:228(accumulate_dict_from_superclasses)
     9034    0.016    0.000    0.018    0.000 copy.py:253(_keep_alive)
     1899    0.016    0.000    0.034    0.000 dimension.py:901(dimensions)
     1776    0.015    0.000    0.077    0.000 nanfunctions.py:228(nanmin)
       79    0.015    0.000    0.028    0.000 __init__.py:69(<listcomp>)
4342/4323    0.015    0.000    0.030    0.000 {method 'join' of 'str' objects}
      656    0.015    0.000    0.284    0.000 util.py:1039(dimension_range)
    41510    0.015    0.000    0.015    0.000 util.py:686(<genexpr>)
     3451    0.015    0.000    0.028    0.000 version.py:307(parse)
       20    0.014    0.001    0.014    0.001 socket.py:438(send)
    16341    0.014    0.000    0.014    0.000 wrappers.py:138(__init__)
    45183    0.014    0.000    0.027    0.000 has_props.py:664(themed_values)
       19    0.013    0.001    0.013    0.001 templates.py:179(signature)
     6151    0.013    0.000    0.025    0.000 inspect.py:2467(__init__)
     6038    0.013    0.000    0.015    0.000 util.py:1509(is_dask_array)
      912    0.013    0.000    0.381    0.000 dimension.py:491(__init__)
    20224    0.013    0.000    0.013    0.000 __init__.py:65(rgb_to_hex)
 2100/931    0.013    0.000    0.025    0.000 _utils.py:44(__str__)
     4151    0.013    0.000    0.048    0.000 util.py:677(allowable)
     1204    0.013    0.000    0.101    0.000 options.py:733(find)
     1776    0.013    0.000    0.043    0.000 nanfunctions.py:343(nanmax)
     6369    0.013    0.000    0.065    0.000 container.py:74(validate)
       53    0.012    0.000    0.615    0.012 model.py:55(collect_filtered_models)
    14559    0.012    0.000    0.021    0.000 bases.py:502(validate)

%%prun -s cumulative:

5428885 function calls (5332236 primitive calls) in 6.073 seconds

   Ordered by: cumulative time
   List reduced from 3666 to 100 due to restriction <100>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      4/1    0.000    0.000    6.073    6.073 {built-in method builtins.exec}
        1    0.000    0.000    6.073    6.073 <string>:2(<module>)
      2/1    0.000    0.000    5.889    5.889 display.py:141(display)
        1    0.000    0.000    5.874    5.874 formatters.py:91(format)
       12    0.000    0.000    5.873    0.489 formatters.py:213(catch_format_error)
        1    0.000    0.000    5.871    5.871 <decorator-gen-5>:1(__call__)
        1    0.000    0.000    5.871    5.871 formatters.py:946(__call__)
        1    0.000    0.000    5.871    5.871 dimension.py:1310(_repr_mimebundle_)
        1    0.000    0.000    5.871    5.871 options.py:1389(render)
        1    0.000    0.000    5.871    5.871 display_hooks.py:274(pprint_display)
        1    0.000    0.000    5.871    5.871 display_hooks.py:235(display)
        3    0.000    0.000    5.871    1.957 display_hooks.py:138(wrapped)
        1    0.000    0.000    5.871    5.871 display_hooks.py:195(map_display)
        1    0.000    0.000    5.871    5.871 display_hooks.py:52(render)
        1    0.000    0.000    5.871    5.871 renderer.py:379(components)
        1    0.000    0.000    5.741    5.741 viewable.py:422(_render_model)
        1    0.000    0.000    5.708    5.708 viewable.py:466(get_root)
        1    0.000    0.000    5.690    5.690 base.py:109(_get_model)
        1    0.000    0.000    5.689    5.689 base.py:82(_get_objects)
        1    0.000    0.000    5.686    5.686 holoviews.py:223(_get_model)
        1    0.000    0.000    5.659    5.659 holoviews.py:273(_render)
        1    0.000    0.000    5.658    5.658 renderer.py:66(get_plot)
        1    0.000    0.000    5.658    5.658 renderer.py:206(get_plot)
        9    0.000    0.000    4.131    0.459 plot.py:980(update)
    168/9    0.004    0.000    2.239    0.249 spaces.py:1280(__getitem__)
        8    0.000    0.000    2.233    0.279 streams.py:146(trigger)
        8    0.000    0.000    2.231    0.279 plot.py:208(refresh)
        8    0.000    0.000    2.217    0.277 plot.py:252(_trigger_refresh)
        8    0.000    0.000    2.216    0.277 plot.py:437(__getitem__)
        8    0.002    0.000    2.216    0.277 element.py:2385(update_frame)
    168/9    0.004    0.000    2.210    0.246 spaces.py:1087(_execute_callback)
    168/9    0.003    0.000    2.209    0.245 spaces.py:667(__call__)
    24/16    0.000    0.000    2.106    0.132 __init__.py:1041(dynamic_operation)
        1    0.000    0.000    1.914    1.914 element.py:2316(initialize_plot)
     63/9    0.001    0.000    1.831    0.203 spaces.py:206(dynamic_mul)
       24    0.001    0.000    1.660    0.069 __init__.py:1033(apply)
       24    0.000    0.000    1.660    0.069 __init__.py:1012(_process)
        9    0.001    0.000    1.635    0.182 element.py:1378(initialize_plot)
        9    0.001    0.000    1.557    0.173 element.py:1340(_init_glyphs)
       16    0.001    0.000    1.525    0.095 raster.py:83(get_data)
       16    0.001    0.000    1.487    0.093 element.py:1826(_get_colormapper)
       16    0.000    0.000    1.431    0.089 util.py:885(process_cmap)
       48    0.001    0.000    1.430    0.030 util.py:666(_list_cmaps)
        2    0.000    0.000    1.424    0.712 <frozen importlib._bootstrap>:978(_find_and_load)
        1    0.000    0.000    1.424    1.424 <frozen importlib._bootstrap>:948(_find_and_load_unlocked)
        1    0.000    0.000    1.424    1.424 <frozen importlib._bootstrap>:663(_load_unlocked)
        1    0.000    0.000    1.424    1.424 <frozen importlib._bootstrap_external>:722(exec_module)
        1    0.000    0.000    1.421    1.421 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
        1    0.010    0.010    1.421    1.421 __init__.py:41(<module>)
      214    0.003    0.000    1.379    0.006 __init__.py:72(mpl_cm)
      214    0.104    0.000    1.371    0.006 colors.py:782(from_list)
       19    0.000    0.000    1.359    0.072 util.py:242(initialize_dynamic)
       88    0.002    0.000    1.297    0.015 operation.py:126(_apply)
       16    0.000    0.000    1.296    0.081 operation.py:176(process_element)
    54784    0.181    0.000    1.202    0.000 colors.py:157(to_rgba)
       16    0.003    0.000    1.142    0.071 datashader.py:885(_process)
    54784    0.343    0.000    0.988    0.000 colors.py:193(_to_rgba_no_colorcycle)
  682/594    0.023    0.000    0.975    0.002 __init__.py:196(pipelined_fn)
     5357    0.081    0.000    0.899    0.000 parameterized.py:2506(__init__)
       47    0.000    0.000    0.884    0.019 plot.py:1265(_get_frame)
       18    0.000    0.000    0.883    0.049 util.py:255(get_plot_frame)
16154/10797    0.031    0.000    0.762    0.000 parameterized.py:1060(override_initialization)
    24/16    0.000    0.000    0.622    0.039 __init__.py:1026(resolve)
       53    0.000    0.000    0.603    0.011 model.py:98(collect_models)
       53    0.012    0.000    0.602    0.011 model.py:55(collect_filtered_models)
       52    0.001    0.000    0.590    0.011 model.py:554(references)
215776/176135    0.087    0.000    0.582    0.000 {built-in method builtins.getattr}
     4866    0.023    0.000    0.580    0.000 model.py:808(_visit_immediate_value_references)
34264/33591    0.034    0.000    0.548    0.000 descriptors.py:676(_get)
38049/36753    0.017    0.000    0.542    0.000 descriptors.py:464(__get__)
     5357    0.089    0.000    0.533    0.000 parameterized.py:1277(_setup_params)
        9    0.000    0.000    0.530    0.059 element.py:749(_update_plot)
       45    0.001    0.000    0.524    0.012 plots.py:90(select)
        9    0.000    0.000    0.515    0.057 overlay.py:30(dynamic_mul)
       32    0.001    0.000    0.513    0.016 __init__.py:1179(clone)
21750/21077    0.041    0.000    0.503    0.000 descriptors.py:704(_get_default)
        9    0.001    0.000    0.493    0.055 element.py:803(_update_ranges)
       16    0.002    0.000    0.489    0.031 core.py:896(raster)
        9    0.000    0.000    0.476    0.053 element.py:759(_update_labels)
       16    0.000    0.000    0.462    0.029 raster.py:415(clone)
       10    0.001    0.000    0.449    0.045 plot.py:1838(get_extents)
21750/21077    0.027    0.000    0.448    0.000 descriptors.py:584(instance_default)
       20    0.001    0.000    0.441    0.022 plot.py:598(compute_ranges)
       80    0.008    0.000    0.440    0.006 __init__.py:304(__init__)
  819/387    0.004    0.000    0.439    0.001 overlay.py:139(__init__)
  819/387    0.008    0.000    0.437    0.001 dimension.py:1350(__init__)
        9    0.000    0.000    0.429    0.048 element.py:763(<dictcomp>)
       18    0.002    0.000    0.429    0.024 element.py:647(_axis_properties)
     1122    0.002    0.000    0.424    0.000 plot.py:290(lookup_options)
      125    0.002    0.000    0.423    0.003 dimension.py:539(clone)
     1122    0.013    0.000    0.422    0.000 options.py:82(lookup_options)
       31    0.009    0.000    0.419    0.014 plot.py:717(_compute_group_range)
       32    0.002    0.000    0.417    0.013 raster.py:266(__init__)
22065/21392    0.031    0.000    0.414    0.000 bases.py:182(themed_default)
     91/9    0.001    0.000    0.412    0.046 dimension.py:677(map)
      912    0.009    0.000    0.411    0.000 dimension.py:849(__init__)
       16    0.000    0.000    0.405    0.025 resampling.py:277(resample_2d)
       16    0.000    0.000    0.403    0.025 resampling.py:482(_resample_2d)
        1    0.000    0.000    0.402    0.402 dispatcher.py:337(_compile_for_args)
     23/1    0.000    0.000    0.402    0.402 compiler_lock.py:29(_acquire_compile_lock)

Theoretically it’s still possible that I’ve made three separate mistakes in the way I’m using in each of the three different hd functions (regrid here, datashade and aggregate in the waterpoint example), but that now seems a bit unlikely and to me the issue being in either holoviews.operation.datashader or datashader itself just became a more reasonable idea.

I’m really curious what you think, @philippjfr

[philippjfr]

So we can keep optimizing internally but at a quick glance I don’t think you made any mistakes here. I reckon with some more effort we can squeeze out another second or two out of HoloViews and Param, and 1 second is spent on initializing matplotlib colormaps which only occurs on initial render and not subsequently. Beyond that I don’t think there’s much we can do, so at best we’re going to get this down 2-3 seconds.

[SteveAKopias]

Well, I still think something is funky in holoviews.operation.datashader.

Consider this example:

def get_image():
    return hv.Image(xr.DataArray(np.zeros((2, 2))))

combined_dynmap = (
    hv.element.tiles.OSM()
    * hv.DynamicMap(get_image) * hv.DynamicMap(get_image)
    * hv.DynamicMap(get_image) * hv.DynamicMap(get_image)
    * hv.DynamicMap(get_image) * hv.DynamicMap(get_image)
    * hv.DynamicMap(get_image) * hv.DynamicMap(get_image)
)

# NEW CELL:
%%prun -l 100
display(combined_dynmap)

Nothing weird here, it only displays 8 times a 2x2 image filled with zeros. On my laptop, after resetting the kernel, display(combined_dynmap) runs for 1.8s. It’s still unclear for me what the actual work is here that takes almost 2 seconds in processing, but I plan on reading through the package source code and figuring that out, for now let’s just accept that as a baseline. If I rerun the display(combined_dynmap) cell a few times without resetting the kernel, I get these runtimes: 0.87s, 0.82s, 0.86s, 0.83s etc, it stays fairly stable from this point. (This is not the point now, but in a live environment every time a user visits the website the whole application reruns, which means that there is a 1s long part that runs every time even though its output supposedly never changes, so that definitely should be cacheable in memory whatever that is, but I will investigate this later.)

Now let’s see the exact same example just with hd.regrid:

combined_regrid = (
    hv.element.tiles.OSM()
    * hd.regrid(hv.DynamicMap(get_image)) * hd.regrid(hv.DynamicMap(get_image))
    * hd.regrid(hv.DynamicMap(get_image)) * hd.regrid(hv.DynamicMap(get_image))
    * hd.regrid(hv.DynamicMap(get_image)) * hd.regrid(hv.DynamicMap(get_image))
    * hd.regrid(hv.DynamicMap(get_image)) * hd.regrid(hv.DynamicMap(get_image))
)

# NEW CELL:
%%prun -l 100
display(combined_regrid)

The same 2x2 image, 8 times, that’s a total of 32 zeros being processed. Supposedly, this shouldn’t differ much, as upsampling is disabled by default, the data is already in xarray format, so there’s not much to do for regrid. And yet, after resetting the kernel, display(combined_regrid) runs for 3.8s, so it does something for an extra 2s. And what’s worse, that when I rerun the display(combined_regrid) cell a few times without resetting the kernel, I got these runtimes: 4.92s, 5.9s, 7.4s, 9.15s, 10.84s, 12.2s, 14.27s, 16.78s, and then I got bored, but it looks like the runtime keeps growing and growing and that doesn’t look normal to me.

The other thing is these runtimes. I come from a PHP background, eg wrote most of the code of a wordpress-like website management system. My former experience is that my system has to deliver everything under one second, including the interpreter parsing the scripts, connecting to database(s), reading the database(s), opening and reading template/data files from the filesystem, processing all of these based on the received user input and at least starting to deliver it. All of this of course concurrently for many users, on potato servers, while PHP not being a super optimized language and Laravel and similar frameworks only making things worse. So you can understand my predicament with having a script now that spends 7-10s in displaying stuff after I already loaded all the data. That’s just not going to be acceptable for the visitors, and it doesn’t matter much if I do the heavy lifting upfront or only onload. Especially not, if displaying basically nothing (2x2 images) takes a lot of time too, so there is not much to gain by preloading lowres images and replacing them with higres alternatives on the fly or something like that. And maybe my expectations were also too high after seeing the Datashader website stating that with Datashader “millions of taxi trips” “appear in milliseconds without trial and error”. Now I timed it and the second cell runs in 1.224s, displaying the result of the thirds cell runs in 1.5s. And while it’s technically true these can be both written in milliseconds, it’s a bit misleading. Truth to be told, it’s pretty impressive that even though the actual example uses 50k records, it runs only 20% longer for 2M points. Based on everything I’ve seen so far, I have the sense that the internal datashader operations indeed run very fast, however there seems to be a huge overhead before that could happen, some stuff that take up about a second before each operation. Anyways, I don’t want to complain, and I’m grateful for all your help, I just needed to went after this pretty long week with the project. I will fork everything and experiment with the source code to understand what’s actually happening, then I’ll see if I find a way to optimize some of it for this specific usecase…

[philippjfr]

On my laptop, after resetting the kernel, display(combined_dynmap) runs for 1.8s

Datashader uses numba under the hood so the first time you render something numba’s JIT compiles the functions, so subsequent renders will be a lot faster.

display(combined_regrid)

As I indicated above, I do not think this is a good test case. display creates a new plot each time and since it does not destroy the old plots you are now accumulating plots which are all being updated every time you rerender. I might be wrong about this but it’s the simplest explanation for me.

So you can understand my predicament with having a script now that spends 7-10s in displaying stuff after I already loaded all the data

I appreciate that this isn’t very performant and we can work on that but I also don’t think it’s actually this slow in practice. You’re counting a number of things which aren’t actually a problem once the server is running and warmed up:

• Matplotlib takes about 1 second to initialize the cmap module
• Numba’s JIT compiler probably spend 2-3 seconds on compiling the datashader codepaths
• prun has significant overhead slowing things down by at least 1 second

So now we’re down to 2-5 seconds to initial render once the server is warmed up and I think with some more optimization of HoloViews and Param we can further reduce that by about 1 second. So yes, that’s nowhere near the 1 second you might expect from some HTTP server but this also isn’t a simple HTTP server but a system that runs quite expensive computations dynamically and serves the results of that up to the user.

Anyways, I don’t want to complain, and I’m grateful for all your help, I just needed to went after this pretty long week with the project.

I’m actually very grateful for your detailed posts, profiling and responses here, it’s a kick in the butt that the project(s) needed so I can at least optimize some of the low hanging fruit.

[Marc]

Hi @SteveAKopias

When and if you get to the FastListTemplate, the FastGridTemplate or the FastGalleryTemplate I would very much like a kick in the but and help improve the performance of these?

I just need some help to understand where the bottlenecks are and how they can be solved.