plotting size limits, workaround

I have 10 Tbytes of uint8 of radio telescope data that I would like to browse. I can load 10GBytes at a time.
bokeh.plotting.line fails if the number of points I indicate is more than 64K (2^16). Is this a problem for the server? Here is my code. 10GBytes is already loaded

···

import numpy as np
from bokeh.plotting import figure, output_file, show, output_notebook

bDataLength = 2**15
bDataStart = 0

x = np.arange(bDataLength)
y = telescopekData[bDataStart : bDataStart + bDataLength]

output_notebook()

p = figure(title=“summed aligned”, x_axis_label=‘x’, y_axis_label=‘y’)

p.line(x, y, line_width=2)

show§

The server uses all the same plotting code. The only thing I can think of it might add is that you might be able to write code to dynamically sub sample the data based on the current axis range or something along those lines.

The server would probably break if you try to send 10tb to the client. But if you arrange to subset the data and only send what’s needed on demand to the client, it could work.

Havoc

···

On Jan 6, 2016, at 8:45 PM, David Saroff [email protected] wrote:

I have 10 Tbytes of uint8 of radio telescope data that I would like to browse. I can load 10GBytes at a time.
bokeh.plotting.line fails if the number of points I indicate is more than 64K (2^16). Is this a problem for the server? Here is my code. 10GBytes is already loaded


import numpy as np
from bokeh.plotting import figure, output_file, show, output_notebook

bDataLength = 2**15
bDataStart = 0

x = np.arange(bDataLength)
y = telescopekData[bDataStart : bDataStart + bDataLength]

output_notebook()

p = figure(title=“summed aligned”, x_axis_label=‘x’, y_axis_label=‘y’)

p.line(x, y, line_width=2)

show§

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2f0d8885-019c-49b8-807b-920ddc47fd8f%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Right,

You'd never want to to send anything like 10gb into the browser. Pretty sure any browser would fall over. I'll mention a few things. For directly sending larger than 64k points, there is partial WebGL support in Bokeh:
  
  http://bokeh.pydata.org/en/latest/docs/user_guide/webgl.html

With this you can reasonably send up to a few 100s of thousands of points. But please note that bowser support for WebGL is inconsistent.

For data your size, you absolutely have to consider downsampling techniques. And this is not even just a tech problem, it is an "efficacy of visualization" problem. If you naively send 100 million points and only have one million pixels, it's questionable that you can get a useful visualization, even with something like alpha compositing. There's two thinks to mention here:

You can certainly use the new Bokeh server to perform downsampling, and send the result to the browser. I would like to have a version of a previous ocean data downsampling data demo on the new server very soon (there's just about an infinite number of things on the todo list, though... working as fast as I can).

We are also working on a project called DataShader (it's in the Bokeh GitHub org as a public repo) that intends to do perceptually accurate downsampling automatically. We demonstrated an earlier version of this at Strata and other conferences to interactively browse the full 350 million point across all scales with a perceptually accurate downsampling with categorical blends (on a laptop!) and it was received very well. As part of the work to make it a public repo it's undergoing a large refactor. Expect more announcements about DataShader in the next month or two.

Bryan

···

On Jan 6, 2016, at 9:04 PM, Havoc Pennington <[email protected]> wrote:

The server uses all the same plotting code. The only thing I can think of it might add is that you might be able to write code to dynamically sub sample the data based on the current axis range or something along those lines.
The server would probably break if you try to send 10tb to the client. But if you arrange to subset the data and only send what's needed on demand to the client, it could work.

Havoc

On Jan 6, 2016, at 8:45 PM, David Saroff <[email protected]> wrote:

I have 10 Tbytes of uint8 of radio telescope data that I would like to browse. I can load 10GBytes at a time.
bokeh.plotting.line fails if the number of points I indicate is more than 64K (2^16). Is this a problem for the server? Here is my code. 10GBytes is already loaded

**********************************************
import numpy as np
from bokeh.plotting import figure, output_file, show, output_notebook

bDataLength = 2**15
bDataStart = 0

x = np.arange(bDataLength)
y = telescopekData[bDataStart : bDataStart + bDataLength]

output_notebook()

p = figure(title="summed aligned", x_axis_label='x', y_axis_label='y')

p.line(x, y, line_width=2)

show(p)

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2f0d8885-019c-49b8-807b-920ddc47fd8f%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/B740CA9C-14F9-4C21-8FB0-139FA97384D6%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.