deploying bokeh, e.g., via Vagrant?

Felix_Breuer · April 2, 2014, 5:46pm

Hello *,

bokeh + IPython together make a great tool for creating rich visualizations. But to be able to easily share such visualizations, every user needs to get both of these software packages up and running on their system. The Anaconda distribution and conda package manager non-withstanding this is unfortunately non-trivial for many users, especially because of non-Python dependencies such as redis-server and libgevent. This becomes even more challenging when bugs/cutting edge features make the use of the git master versions of bokeh and other python packages necessary.

How do you solve the problem sharing plots / visualizations?
Has anyone created, for example, a Vagrantfile and provisioning script describing a reproducible virtual machine that runs, e.g., IPython and bokeh-server? Or just a minimal provisioning script (or Vagrant “box”) setting up a basic Anaconda installation?
I will try to create a such a Vargrantfile / provisioning script and report back here. If there are other ways I could approach this problem, please let me know!

Cheers,
Felix

Felix_Breuer · April 3, 2014, 3:12pm

I have made first steps towards creating a reproducible installation of bokeh and IPython. I will write more about this later. For now, here is a basic setup creating a Vagrant virtual machine, installing ipython and bokeh plus dependencies via Ansible and automatically launching an IPython notebook. If you put the Vagrantfile in a directory with IPython notebooks, put the other two files in a subdirectory called _config and then run vagrant up, you should see a running IPython notebook at localhost:8889 once provisioning has completed.

Vagrantfile:

-- mode: ruby --

vi: set ft=ruby :

VAGRANTFILE_API_VERSION = “2”

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = “puppetlabs/ubuntu-13.10-64-nocm”

config.vm.network “forwarded_port”, guest: 8888, host: 8889
config.vm.network “forwarded_port”, guest: 5006, host: 5007

config.vm.provision :ansible do |ansible|
ansible.playbook = “_config/playbook.yml”
#ansible.inventory_path = “provisioning/hosts-vagrant”
# On Vagrant < 1.3 this used to be inventory_file…
# ansible.inventory_file = “provisioning/hosts-vagrant”
ansible.verbose = false
end
end

_config/playbook.yml:

···

hosts: all
tasks:
- name: update apt cache
  apt: update_cache=yes cache_valid_time=3600
  sudo: yes
- name: upgrade the distro
  apt: upgrade=yes
  sudo: yes
- name: install packages
  apt: pkg={{ item }} state=latest
  sudo: yes
  with_items:
  - build-essential
  - git
  - python-dev
  - python-pip
  - supervisor
  - libevent-dev # for bokeh
  - redis-server # for bokeh
  - libzmq-dev # for ipython
- name: configure supervisor
  sudo: yes
  copy: src=ipython_notebook.conf dest=/etc/supervisor/conf.d/ipython_notebook.conf owner=root mode=644
  notify: restart ipython notebook
- name: install python packages
  pip: name={{ item }} state=latest
  sudo: yes
  with_items:
  - redis # for bokeh
  - bokeh
  - pyzmq # for ipython
  - tornado # for ipython
  - ipython
    notify: restart ipython notebook
    handlers:
- name: restart ipython notebook
  sudo: yes
  supervisorctl: name=ipython_notebook state=restarted

_config/ipython_notebook.conf:

; supervisord configuration file for ipython notebook
[program:ipython_notebook]
command=ipython notebook --ip 0.0.0.0
autostart=true
autorestart=true
directory=/vagrant

Felix

On Wednesday, April 2, 2014 7:46:07 PM UTC+2, Felix Breuer wrote:

Hello *,

bokeh + IPython together make a great tool for creating rich visualizations. But to be able to easily share such visualizations, every user needs to get both of these software packages up and running on their system. The Anaconda distribution and conda package manager non-withstanding this is unfortunately non-trivial for many users, especially because of non-Python dependencies such as redis-server and libgevent. This becomes even more challenging when bugs/cutting edge features make the use of the git master versions of bokeh and other python packages necessary.

How do you solve the problem sharing plots / visualizations?

Has anyone created, for example, a Vagrantfile and provisioning script describing a reproducible virtual machine that runs, e.g., IPython and bokeh-server? Or just a minimal provisioning script (or Vagrant “box”) setting up a basic Anaconda installation?

I will try to create a such a Vargrantfile / provisioning script and report back here. If there are other ways I could approach this problem, please let me know!

Cheers,
Felix

Felix_Breuer · April 3, 2014, 6:02pm

See also https://github.com/fbreuer/ipython-vagrant.

···

On Thursday, April 3, 2014 5:12:13 PM UTC+2, Felix Breuer wrote:

Hello *,

bokeh + IPython together make a great tool for creating rich visualizations. But to be able to easily share such visualizations, every user needs to get both of these software packages up and running on their system. The Anaconda distribution and conda package manager non-withstanding this is unfortunately non-trivial for many users, especially because of non-Python dependencies such as redis-server and libgevent. This becomes even more challenging when bugs/cutting edge features make the use of the git master versions of bokeh and other python packages necessary.

How do you solve the problem sharing plots / visualizations?

Has anyone created, for example, a Vagrantfile and provisioning script describing a reproducible virtual machine that runs, e.g., IPython and bokeh-server? Or just a minimal provisioning script (or Vagrant “box”) setting up a basic Anaconda installation?

I will try to create a such a Vargrantfile / provisioning script and report back here. If there are other ways I could approach this problem, please let me know!

Cheers,
Felix

I have made first steps towards creating a reproducible installation of bokeh and IPython. I will write more about this later. For now, here is a basic setup creating a Vagrant virtual machine, installing ipython and bokeh plus dependencies via Ansible and automatically launching an IPython notebook. If you put the Vagrantfile in a directory with IPython notebooks, put the other two files in a subdirectory called _config and then run vagrant up, you should see a running IPython notebook at localhost:8889 once provisioning has completed.

Vagrantfile:

-- mode: ruby --

vi: set ft=ruby :

VAGRANTFILE_API_VERSION = “2”

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = “puppetlabs/ubuntu-13.10-64-nocm”

config.vm.network “forwarded_port”, guest: 8888, host: 8889
config.vm.network “forwarded_port”, guest: 5006, host: 5007

config.vm.provision :ansible do |ansible|
ansible.playbook = “_config/playbook.yml”
#ansible.inventory_path = “provisioning/hosts-vagrant”
# On Vagrant < 1.3 this used to be inventory_file…
# ansible.inventory_file = “provisioning/hosts-vagrant”
ansible.verbose = false
end
end

_config/playbook.yml:

hosts: all
tasks:

name: update apt cache
apt: update_cache=yes cache_valid_time=3600
sudo: yes

name: upgrade the distro
apt: upgrade=yes
sudo: yes

name: install packages
apt: pkg={{ item }} state=latest
sudo: yes
with_items:

build-essential

git

python-dev

python-pip

supervisor

libevent-dev # for bokeh

redis-server # for bokeh

libzmq-dev # for ipython

name: configure supervisor
sudo: yes
copy: src=ipython_notebook.conf dest=/etc/supervisor/conf.d/ipython_notebook.conf owner=root mode=644
notify: restart ipython notebook

name: install python packages
pip: name={{ item }} state=latest
sudo: yes
with_items:

redis # for bokeh

bokeh

pyzmq # for ipython

tornado # for ipython

ipython
notify: restart ipython notebook
handlers:

name: restart ipython notebook
sudo: yes
supervisorctl: name=ipython_notebook state=restarted

_config/ipython_notebook.conf:

; supervisord configuration file for ipython notebook
[program:ipython_notebook]
command=ipython notebook --ip 0.0.0.0
autostart=true
autorestart=true
directory=/vagrant

Felix

On Wednesday, April 2, 2014 7:46:07 PM UTC+2, Felix Breuer wrote:

Karan_Dodia · April 4, 2014, 3:20pm

Hello Felix,

First off: thank you for demonstrating a continued interest in Bokeh, and especially for engaging us on so many topics as we continue to develop it. Your involvement genuinely motivates us to create a first class product, and your input helps us shape it to fulfill the needs of real users. Our dev team has been somewhat scattered as we prepare for a new release next week, so I hope you do not mind the delayed response.

Regarding your questions at the top of this thread:

1) How do you solve the problem sharing plots / visualizations?

There are a number of approaches that we see as possibilities, and we continue to explore them all. Here are the ideas we have at the moment:

a. Use Wakari to collaborate on and share notebooks. Wakari is Continuum’s cloud-hosted service which offers customizable Python environments. Notebooks that are hosted on Wakari, and the Bokeh plots therein, can be shared and forked easily—you can see one of my examples here in response to a StackOverflow question. There are major upgrades planned for this service, but it is entirely functional and useful at the present moment!
b. Offer bokehplots.com (under construction) as an upload target to host plots and their associated data. We are eager to get this out the door, although up to this point we have focused our efforts to flesh out Bokeh as user uptake is still in the early stages. Your email may have convinced us to move up the timeframe.

c. Provide several methods to embed plots in tags. Ideally bokeh.min.js would be pulled from a CDN, and the plot source code (in Javascript) could reside in the tag, on your own bokeh-server, or on bokehplots.com.

If you have any thoughts on the above methods, please let us know!

2) Has anyone created, for example, a Vagrantfile and provisioning script describing a reproducible virtual machine that runs, e.g., IPython and bokeh-server? Or just a minimal provisioning script (or Vagrant “box”) setting up a basic Anaconda installation?

No, I do not believe this has been done yet! Anaconda distributions are easy to reproduce, and we have tried to minimize the dependencies for Bokeh as much as possible. Even redis has become an optional dependency—now the default is to rely on an in-memory backend.

3) I will try to create a such a Vargrantfile / provisioning script and report back here. If there are other ways I could approach this problem, please let me know!

The script you provided looks really good! We are perhaps a little biased, but we use conda for Python package management; however pip works well enough within the Vagrant provisioning framework. I’ll try to reproduce your work if I get the chance today.

Best regards,

Karan Dodia & the Bokeh team

···

On Thu, Apr 3, 2014 at 1:02 PM, Felix Breuer [email protected] wrote:

See also https://github.com/fbreuer/ipython-vagrant.

On Thursday, April 3, 2014 5:12:13 PM UTC+2, Felix Breuer wrote:
I have made first steps towards creating a reproducible installation of bokeh and IPython. I will write more about this later. For now, here is a basic setup creating a Vagrant virtual machine, installing ipython and bokeh plus dependencies via Ansible and automatically launching an IPython notebook. If you put the Vagrantfile in a directory with IPython notebooks, put the other two files in a subdirectory called _config and then run vagrant up, you should see a running IPython notebook at localhost:8889 once provisioning has completed.

Vagrantfile:

-- mode: ruby --

vi: set ft=ruby :

VAGRANTFILE_API_VERSION = “2”

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = “puppetlabs/ubuntu-13.10-64-nocm”

config.vm.network “forwarded_port”, guest: 8888, host: 8889
config.vm.network “forwarded_port”, guest: 5006, host: 5007

config.vm.provision :ansible do |ansible|
ansible.playbook = “_config/playbook.yml”
#ansible.inventory_path = "provisioning/hosts-vagrant"
# On Vagrant < 1.3 this used to be `inventory_file`...
# ansible.inventory_file = "provisioning/hosts-vagrant"
ansible.verbose = false
end
end

_config/playbook.yml:

hosts: all
tasks:

name: update apt cache
apt: update_cache=yes cache_valid_time=3600
sudo: yes

name: upgrade the distro

apt: upgrade=yes
sudo: yes

name: install packages
apt: pkg={{ item }} state=latest
sudo: yes
with_items:

build-essential

git

python-dev

python-pip

supervisor

libevent-dev # for bokeh

redis-server # for bokeh

libzmq-dev # for ipython

name: configure supervisor
sudo: yes
copy: src=ipython_notebook.conf dest=/etc/supervisor/conf.d/ipython_notebook.conf owner=root mode=644

notify: restart ipython notebook

name: install python packages
pip: name={{ item }} state=latest
sudo: yes
with_items:

redis # for bokeh

bokeh

pyzmq # for ipython

tornado # for ipython

ipython
notify: restart ipython notebook
handlers:

name: restart ipython notebook
sudo: yes
supervisorctl: name=ipython_notebook state=restarted

_config/ipython_notebook.conf:

; supervisord configuration file for ipython notebook
[program:ipython_notebook]
command=ipython notebook --ip 0.0.0.0
autostart=true
autorestart=true
directory=/vagrant

Felix

On Wednesday, April 2, 2014 7:46:07 PM UTC+2, Felix Breuer wrote:

Hello *,

bokeh + IPython together make a great tool for creating rich visualizations. But to be able to easily share such visualizations, every user needs to get both of these software packages up and running on their system. The Anaconda distribution and conda package manager non-withstanding this is unfortunately non-trivial for many users, especially because of non-Python dependencies such as redis-server and libgevent. This becomes even more challenging when bugs/cutting edge features make the use of the git master versions of bokeh and other python packages necessary.

How do you solve the problem sharing plots / visualizations?

Has anyone created, for example, a Vagrantfile and provisioning script describing a reproducible virtual machine that runs, e.g., IPython and bokeh-server? Or just a minimal provisioning script (or Vagrant “box”) setting up a basic Anaconda installation?

I will try to create a such a Vargrantfile / provisioning script and report back here. If there are other ways I could approach this problem, please let me know!

Cheers,
Felix

–

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/48686c42-88c8-47c3-9091-8abbf2db6c48%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Felix_Breuer · April 5, 2014, 12:12pm

Hello Karan

thanks for your reply! Here are a couple of comments in no particular order.

Sharing with collaborators.

On the whole, Wakari is probably the best way to do this in the future. After a brief look about a week ago, I did not pursue Wakari further, as the website seemed very slow/unresponsive back then. But I’ll give it another try in the future.

One other aspect is that I am using github to collabroatively work on files (mainly TeX) with my co-authors and having a sharing mechanism that fits the git workflow is desirable.

Publishing plots.

I have had very good experiences publishing visualizations on my blog using Sage cells. See here for an example: Visualizing the GCD - Felix Breuer's Blog The advantage is that readers of my blog can modify and re-run the code generating the plot right on the website. As an author, all I have to do is put an HTML snippet on the page, similar to approach c. you mention, but with the additional advantage that Sage offers servers that do the Python processing in the background - without my having to sign up for anything. Maybe something similar could be built on top of Wakari.

Vagrant, Conda etc.

I agree that Conda is very nice, but as you can tell from the other thread I posted here, installing Python packages can still be something of an odyssey. And I did not want to have to walk each of my co-authors through the entire process individually, dealing with whatever issues may arise on their system. This is especially important when Python packages have non-Python dependencies, because some of my co-authors are on Windows and do not have any C/C++ compilers installed on their system. (Correct me if I am wrong, but even the full Anaconda distribution does not include a full C/C++ build environment, right?) Of course, as long as binary packages are available for everything, no compilation is needed, but if something needs to be built from source for some reason…)

In the end it just seemed easier for me to simply specify a Vagrant VM once and have all the installation/dependency management happen inside that VM. This way I can control what happens and easily reproduce any problems I might encounter. And I do not have to deal with the quirks of my coauthors’ personal systems.

Again, thanks for the great work you are doing on bokeh and conda! I will write a blog post on the visualization we did for this research project as soon as the paper is done and post a link here.