Deciding between Python Bokeh or d3,js for a project - can't figure out the difference

Community Support
1

Dear All Bokeh Users and Developers,

Within my group, I am responsible for maintaining and adding new capabilities to our Continuous Integration infrastructure. We have a large codebase, and with an already established a build server powered by Buildbot http://buildbot.net that conducts regular builds that we schedule to run every midnight. These builds feed benchmark files into our compiled codebase executable as part of the process. Furthermore, together with Buildbot’s ability to display results on a webpage under what is called its “waterfall” view, we have developed our own Python script that is responsible for generating input test configuration files that will be fed into the software binary that is generated in each build, and to also collect the results obtained from each run of the software binary and write out these results to the buildbot URL in the form of SQL Database scripts.

Recently, I have developed a data analysis module in python using the sqlite3, numpy, pandas and matplotlib libraries. Running that module locally on your computer will initiate a series of visits to respective build URLs to collect the SQL scripts and build up a local database in which I query from and analyze to plot graphs for different metrics, and they will be stored on the local machine that the script was run. The results give us a perspective of the health and strength of our codebase.

Now I would like to take things further to actually have a webpage on 24/7 that actually polls our Build data URLs for results, and then incorporates information from the latest build directly and displays the visualization results directly on that webpage. Furthermore, I’d like the graphs to be interactive, meaning that say I could click at a specific datapoint that corresponds to a particular build number and actually be redirected to my Build URL.

I’ve done some searching and reading up on d3.js as well as Python’s new Bokeh library (http://bokeh.pydata.org/en/latest/index.html) and am still unsure which one I should go for. They both seem to do the exact same thing, however, when I read the Bokeh FAQs, they explain that they are different from d3.js (http://bokeh.pydata.org/en/latest/docs/faq.html#does-bokeh-use-d3-js) . However, aren’t they doing the same thing? That is allowing you to visualize data on a webpage, something which you can’t do with Matplotlib for example? I’ve more or less verified that d3.js can support the feature of having data points that can be clicked so that I can be directed to my build URLs (correct me if I’m wrong), however can Bokeh also do that?

From the looks of it, if both d3.js and Bokeh can allow me to achieve my aim of having a webpage that diplays interactive graphs whereby I can click on a datapoint corresponding to a build number and be taken to that respective build URL, then does it makes sense for me to choose Bokeh over d3.js since I’ve already used Python for my data munging and analysis? Because it seems like if I use d3.js then there will have to be a break in my flow where I will need to export my curated data out from python and into maybe CSV files, and plot the d3 graphs from the CSVs.

Lastly, I want to ask how I could implement the polling. Do I have to actually add a new Buildbot build step to each of my Builders (http://docs.buildbot.net/current/tutorial/fiveminutes.html#builders-the-workhorses) where that step executes my python script to do data analysis and to store that data at an absolute path, and then plot out graphs from the data at that location?

Also I’m not really sure if Buildbot already has plugins that can already do what I want to set out to Build here. If there are any, please do recommend them to me.

Thank you for reading my verbose post here and understanding my situation. I’m new to all this and I have not been able to find any one source of information that can answer my question of how I want to implement this for my group, and I guess this is because its really specific to the context. I will greatly appreciate any suggestions or feedback as well as clarifications to my misconceptions here.

Best,

Augustine

2

Hi Augustine,

Thanks for your question. I’m sure there are others like you in a similar spot who are looking at Bokeh vs e.g. d3.

I will answer the question at a high level, and I’ve also included a small code example which you can extend to do what you need.

build_example.py (835 Bytes)

···

I’ve done some searching and reading up on d3.js as well as Python’s new Bokeh library (http://bokeh.pydata.org/en/latest/index.html) and am still unsure which one I should go for. They both seem to do the exact same thing, however, when I read the Bokeh FAQs, they explain that they are different from d3.js (http://bokeh.pydata.org/en/latest/docs/faq.html#does-bokeh-use-d3-js) . However, aren’t they doing the same thing? That is allowing you to visualize data on a webpage, something which you can’t do with Matplotlib for example?

The primary issue is not whether or not pixels are painted onto the screen via a web page. For instance, one could write some python code which outputs an HTML file that embeds a Matplotlib PNG image along with a carefully-crafted old-school imagemap via the tag.

The key difference is who the libraries are made for.

D3 requires reasonable knowledge of javascript, and also quite a bit of learning of how the D3 visualization engine itself works. Bokeh, on the other hand, targets users who are interested in writing Python or R or other non-Javascript languages, but who still want interactive graphics in the browser.

Additionally, based on what I understand, Bokeh’s Javascript plotting engine actually goes beyond D3 in certain respects, like date-time tick calculations and responsively handling scene updates.

I’ve more or less verified that d3.js can support the feature of having data points that can be clicked so that I can be directed to my build URLs (correct me if I’m wrong), however can Bokeh also do that?

Yes. I have attached a small and simple example (build_example.py) that does exactly this. When you hover over each of the points in the plot, you will see a pop-up bubble that displays some information. If you click the point, then it will open up a new browser tab to some custom URL.

does it makes sense for me to choose Bokeh over d3.js since I’ve already used Python for my data munging and analysis?

I certainly think so. :slight_smile:

Lastly, I want to ask how I could implement the polling. Do I have to actually add a new Buildbot build step to each of my Builders (http://docs.buildbot.net/current/tutorial/fiveminutes.html#builders-the-workhorses) where that step executes my python script to do data analysis and to store that data at an absolute path, and then plot out graphs from the data at that location?

You can certainly do this in a static fashion to start with, but Bokeh is actually designed to support handling of dynamic data. Once you understand how my attached build_example.py works, you can look at the simple streaming example I just added to the repository:

https://github.com/bokeh/bokeh/blob/master/examples/plotting/server/simple_stream.py

Also I’m not really sure if Buildbot already has plugins that can already do what I want to set out to Build here. If there are any, please do recommend them to me.

I am not aware of any. However, it would be very cool if your scripts could serve as the basis for such a plugin. I’m sure you’re not alone in your desire to visualize the outputs. Perhaps once you get something working, you can share it with folks in the BuildBot community and they can contribute some ideas and features.

Thanks,

Peter

3

Hello Peter,

I have a related question. I have very small server and use d3 and crossfilter since everything happens in the browser.

With Bokeh would I need a much larger server to serve my interactive dashboards?

Thanks

Fabio

4

Hello Peter,

I have a related question. I have very small server and use d3 and crossfilter since everything happens in the browser.

With Bokeh would I need a much larger server to serve my interactive dashboards?

Thanks

Fabio