Dear All Bokeh Users and Developers,
Within my group, I am responsible for maintaining and adding new capabilities to our Continuous Integration infrastructure. We have a large codebase, and with an already established a build server powered by Buildbot http://buildbot.net that conducts regular builds that we schedule to run every midnight. These builds feed benchmark files into our compiled codebase executable as part of the process. Furthermore, together with Buildbot’s ability to display results on a webpage under what is called its “waterfall” view, we have developed our own Python script that is responsible for generating input test configuration files that will be fed into the software binary that is generated in each build, and to also collect the results obtained from each run of the software binary and write out these results to the buildbot URL in the form of SQL Database scripts.
Recently, I have developed a data analysis module in python using the sqlite3, numpy, pandas and matplotlib libraries. Running that module locally on your computer will initiate a series of visits to respective build URLs to collect the SQL scripts and build up a local database in which I query from and analyze to plot graphs for different metrics, and they will be stored on the local machine that the script was run. The results give us a perspective of the health and strength of our codebase.
Now I would like to take things further to actually have a webpage on 24/7 that actually polls our Build data URLs for results, and then incorporates information from the latest build directly and displays the visualization results directly on that webpage. Furthermore, I’d like the graphs to be interactive, meaning that say I could click at a specific datapoint that corresponds to a particular build number and actually be redirected to my Build URL.
I’ve done some searching and reading up on d3.js as well as Python’s new Bokeh library (http://bokeh.pydata.org/en/latest/index.html) and am still unsure which one I should go for. They both seem to do the exact same thing, however, when I read the Bokeh FAQs, they explain that they are different from d3.js (http://bokeh.pydata.org/en/latest/docs/faq.html#does-bokeh-use-d3-js) . However, aren’t they doing the same thing? That is allowing you to visualize data on a webpage, something which you can’t do with Matplotlib for example? I’ve more or less verified that d3.js can support the feature of having data points that can be clicked so that I can be directed to my build URLs (correct me if I’m wrong), however can Bokeh also do that?
From the looks of it, if both d3.js and Bokeh can allow me to achieve my aim of having a webpage that diplays interactive graphs whereby I can click on a datapoint corresponding to a build number and be taken to that respective build URL, then does it makes sense for me to choose Bokeh over d3.js since I’ve already used Python for my data munging and analysis? Because it seems like if I use d3.js then there will have to be a break in my flow where I will need to export my curated data out from python and into maybe CSV files, and plot the d3 graphs from the CSVs.
Lastly, I want to ask how I could implement the polling. Do I have to actually add a new Buildbot build step to each of my Builders (http://docs.buildbot.net/current/tutorial/fiveminutes.html#builders-the-workhorses) where that step executes my python script to do data analysis and to store that data at an absolute path, and then plot out graphs from the data at that location?
Also I’m not really sure if Buildbot already has plugins that can already do what I want to set out to Build here. If there are any, please do recommend them to me.
Thank you for reading my verbose post here and understanding my situation. I’m new to all this and I have not been able to find any one source of information that can answer my question of how I want to implement this for my group, and I guess this is because its really specific to the context. I will greatly appreciate any suggestions or feedback as well as clarifications to my misconceptions here.
Best,
Augustine