Creating a dynamically downsampling plot for a large dataset?

Hey all,

I have a dataset in a database, and I’m looking to plot it on a dashboard. The problem is that each data channel has around 8 million data points and I have about 16 channels that I’d need to plot. I’ve been looking around for possible solutions to this and came across this demo which would be ideal to achieve for my dataset: https://www.youtube.com/watch?v=kPknmEwQ3Rc

What would be the best approach to getting started with this?

Wow, that’s a blast from the past. First things first, just noting that decade old talk videos for most any OSS project are probably usually not a great source for up-to-date information about the project.

It’s certainly possible to do what is shown in the video with pure Bokeh using a Bokeh server application. However, contrary to that video, modern Bokeh will not do anything like automatically downsample the data for you. ^[1] You would need to set up event handlers for range updates and then re-compute a downsampled dataset based on the ranges, then update all the appropriate data sources…

So my actual, suggestion, if you want to get something off the ground more simply, is to look at Holoviews which is built on top of Bokeh and can integrate Datashader to handle very large datasets by via server-side rendering. For the “dashboard” part, the same Holoviz team also created Panel which is an even higher level dashboarding tool that can bring all these components together.

cc @Philipp_Rudiger @James_A_Bednar1 for any more specific pointers to resources or recent talks, etc.

1. That kind of automatic downsampling was an original goal of Bokeh circa ~12 years ago, under the feature name “Abstract Rendering” but it quickly became evident that Bokeh needed to focus on eventing and plotting and layout, and leave that sort of auto-magic data handling to higher-level tools built on Bokeh. ↩︎

Ah, thank you for clarifying. I’ll definitely look into Holoviews and Datashader for this then. Do you have any specific examples you can point me towards?

Also curious about Bokeh server and event handlers though, is there anything specific I should be looking at for this?

I’m not especially well-versed with all the “Holoviz” tools, I am just aware of generally what they can do. Hopefully one of the folks I pinged above will chime in here. Otherwise, there is a Holoviz Discourse where you can reach out to that community directly.

Also curious about Bokeh server and event handlers though, is there anything specific I should be looking at for this?

I’d start with the docs page I linked above. There’s also some live demo apps running at https://demo.bokeh.org/ that you can play around with.

You can see lots of examples of Datashader + HoloViews (or Datashader + hvPlot, which builds on HoloViews) at examples.holoviz.org, and the getting started page about interactivity at Datashader.org should get you started. Enjoy!

I highly recommend using hvplot as the entrypoint to HoloViews & Datashader because it connects with pandas/polars/duckdb/xarray/etc and it’s setting kwargs, so it can be as simple as: df.hvplot(rasterize=True, by="channel", subplots=True)

See Large Timeseries Data — hvPlot 0.11.1 documentation

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.