Bokeh server with hdf5 data

Is is possible to make a bokeh server that can callback data from an HDF5 file, and only read what is needed?

I want to graph data through a bokeh server with a drop-down menu to select what hdf5 data I want to plot. But I don’t wish to load the entire dataset into memory because of the size of the hdf5.

Is this possible in Bokeh 0.11, and how could it be done?

Thanks in advance.

Kasper P.

Hi Kasper,

You can run arbitrary python code in Bokeh server callbacks, so this is really more a question about HDF5 I think. Is it possible to do efficient partial reads from disk with HDF5? I'm guessing it is, but I don't really have any experience with HDF5. Assuming it is, it's really no different from most of the existing examples in the examples/app directory: read or compute new data in your callback, then set the Bokeh data source accordingly. A concrete example is here:

  https://github.com/bokeh/bokeh/blob/master/examples/app/movies/main.py#L80-L96

I'd very much like to add an example of this type. It's been on my priority list for a while, but there's only one of me, and many things on my list. If you do create an example of downsampling HDF5, both I and the community would be extremely grateful if you could contribute some form of it as an example in a Pull Request on GitHub.

There is one caveat I will add: currently Bokeh server callbacks are blocking to new server connections. There are plans to add a thread pool executor, so that tasks that are compute heavy, or depend on blocking I/O, etc. do not block the main server from accepting new connections, but this has not been gotten to yet. This may or may not have implications for your intended usage. In practical terms this means that it would be fine for one person to run and use a "long-running callback" app locally (or if you know there will not be very many concurrent users, perhaps deploy a few instances behind a proxy as described in the User's Guide). But apps with "long-running callbacks" will probably not scale well (yet) to very many users at a time.

Thanks,

Bryan

···

On Feb 2, 2016, at 6:11 AM, Kasper P. <[email protected]> wrote:

Is is possible to make a bokeh server that can callback data from an HDF5 file, and only read what is needed?

I want to graph data through a bokeh server with a drop-down menu to select what hdf5 data I want to plot. But I don't wish to load the entire dataset into memory because of the size of the hdf5.

Is this possible in Bokeh 0.11, and how could it be done?

Thanks in advance.
Kasper P.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/222edabb-b700-46b3-8372-a52a6e197ec5%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

Hey Bryan

Thanks for the reply.

I finally got time to create a small example, and did a pull request called:

“Added simple hdf5 example app. #3876

Best regards

Kasper Pihl Rasmussen

···

On Tuesday, February 2, 2016 at 3:04:55 PM UTC+1, Bryan Van de ven wrote:

Hi Kasper,

You can run arbitrary python code in Bokeh server callbacks, so this is really more a question about HDF5 I think. Is it possible to do efficient partial reads from disk with HDF5? I’m guessing it is, but I don’t really have any experience with HDF5. Assuming it is, it’s really no different from most of the existing examples in the examples/app directory: read or compute new data in your callback, then set the Bokeh data source accordingly. A concrete example is here:

    [https://github.com/bokeh/bokeh/blob/master/examples/app/movies/main.py#L80-L96](https://github.com/bokeh/bokeh/blob/master/examples/app/movies/main.py#L80-L96)

I’d very much like to add an example of this type. It’s been on my priority list for a while, but there’s only one of me, and many things on my list. If you do create an example of downsampling HDF5, both I and the community would be extremely grateful if you could contribute some form of it as an example in a Pull Request on GitHub.

There is one caveat I will add: currently Bokeh server callbacks are blocking to new server connections. There are plans to add a thread pool executor, so that tasks that are compute heavy, or depend on blocking I/O, etc. do not block the main server from accepting new connections, but this has not been gotten to yet. This may or may not have implications for your intended usage. In practical terms this means that it would be fine for one person to run and use a “long-running callback” app locally (or if you know there will not be very many concurrent users, perhaps deploy a few instances behind a proxy as described in the User’s Guide). But apps with “long-running callbacks” will probably not scale well (yet) to very many users at a time.

Thanks,

Bryan

On Feb 2, 2016, at 6:11 AM, Kasper P. [email protected] wrote:

Is is possible to make a bokeh server that can callback data from an HDF5 file, and only read what is needed?

I want to graph data through a bokeh server with a drop-down menu to select what hdf5 data I want to plot. But I don’t wish to load the entire dataset into memory because of the size of the hdf5.

Is this possible in Bokeh 0.11, and how could it be done?

Thanks in advance.

Kasper P.


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/222edabb-b700-46b3-8372-a52a6e197ec5%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.