memory leak when increasing websocket_max_message_size

Community Support
1

Hi,

I am trying to update a column data source with len 40000 information having 7 columns. This causes web connection to close. The workaround I found and as earlier suggested to me was to send the data in chunks. That works okay but is there an another way as this requires to set up threads and updating the doc? I found th following issue

which says that websocket_max_message_size is allowed to be overwritten. But when I do so and increase the default 10 mib to 20 mib, I run into another problem of memory leak and besides the connection is closed anyway just because I am tyring to send large amount of data. Is there a solution to this and could you explain what is happening in background tornado web socket?

Thanks.

2

Hi,

There's nothing that can be done or said on something like this without:

* a complete minimal script that demonstrates the issue
* detailed instructions to reproduce the reported result

could you explain what is happening in background tornado web socket?

I don't really have any idea what information you are asking for, specifically.

Thanks,

Bryan

···

On Jul 29, 2018, at 05:35, [email protected] wrote:

Hi,

I am trying to update a column data source with len 40000 information having 7 columns. This causes web connection to close. The workaround I found and as earlier suggested to me was to send the data in chunks. That works okay but is there an another way as this requires to set up threads and updating the doc? I found th following issue

Connection errors when sending large amounts of data to browser via a bokeh server · Issue #7374 · bokeh/bokeh · GitHub

which says that websocket_max_message_size is allowed to be overwritten. But when I do so and increase the default 10 mib to 20 mib, I run into another problem of memory leak and besides the connection is closed anyway just because I am tyring to send large amount of data. Is there a solution to this and could you explain what is happening in background tornado web socket?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/a44d4b8d-f66b-4fe1-8b77-4860dcdc2fd8%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

3

You can find a small script below. This doesn’ t produce any errors though. The glyphs I am working with are wedges rather than circles.

import numpy as np

from bokeh.models import ColumnDataSource, DataRange1d, Plot, LinearAxis, Grid

from bokeh.plotting import figure,curdoc

from bokeh.models.markers import Circle

from bokeh.io import curdoc, show

from bokeh.models import CustomJS

N = 200000

x = np.linspace(-2, 2, N)

y = x**2

sizes = np.linspace(10, 20, N)

dummy1 = [0]*N

dummy2 = [0]*N

dummy3 = [0]*N

dummy4 = [0]*N

dummy5 = [0]*N

dummy6 = [0]*N

dummy7 = [0]*N

dummy8 = [0]*N

dummy9 = [0]*N

source = ColumnDataSource(dict(x=x, y=y,dummy1 = dummy1,dummy2 = dummy2,dummy3 = dummy3,dummy4 = dummy4,dummy5 = dummy5,dummy6 = dummy6,dummy7 = dummy7,dummy8 = dummy8,dummy9 = dummy9, sizes=sizes))

xdr = DataRange1d()

ydr = DataRange1d()

circleSelected = CustomJS(args = dict( source = source),code = ‘’’

console.log("do nothing")

''')

plot = figure(

title=None, x_range=xdr, y_range=ydr, plot_width=300, plot_height=300,

h_symmetry=False, v_symmetry=False, min_border=0, toolbar_location=None)

#glyph = Circle(x=“x”, y=“y”, size=“sizes”, line_color=“#3288bd”, fill_color=“white”, line_width=3)

plot.circle(x=“x”, y=“y”, size=“sizes”, line_color=“#3288bd”, fill_color=“white”, line_width=3, source = source)

#plot.add_glyph(source, glyph)

xaxis = LinearAxis()

plot.add_layout(xaxis, ‘below’)

yaxis = LinearAxis()

plot.add_layout(yaxis, ‘left’)

plot.add_layout(Grid(dimension=0, ticker=xaxis.ticker))

plot.add_layout(Grid(dimension=1, ticker=yaxis.ticker))

plot.js_on_event(‘tap’, circleSelected)

curdoc().add_root(plot)

···

On Sunday, July 29, 2018 at 10:44:18 PM UTC+3, Bryan Van de ven wrote:

Hi,

There’s nothing that can be done or said on something like this without:

  • a complete minimal script that demonstrates the issue

  • detailed instructions to reproduce the reported result

could you explain what is happening in background tornado web socket?

I don’t really have any idea what information you are asking for, specifically.

Thanks,

Bryan

On Jul 29, 2018, at 05:35, [email protected] wrote:

Hi,

I am trying to update a column data source with len 40000 information having 7 columns. This causes web connection to close. The workaround I found and as earlier suggested to me was to send the data in chunks. That works okay but is there an another way as this requires to set up threads and updating the doc? I found th following issue

https://github.com/bokeh/bokeh/issues/7374

which says that websocket_max_message_size is allowed to be overwritten. But when I do so and increase the default 10 mib to 20 mib, I run into another problem of memory leak and besides the connection is closed anyway just because I am tyring to send large amount of data. Is there a solution to this and could you explain what is happening in background tornado web socket?

Thanks.

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/a44d4b8d-f66b-4fe1-8b77-4860dcdc2fd8%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

4

Apologies, but I don't understand the context of "This doesn't produce any errors though" in your statement. Are you saying the script below does not exhibit the memory leak? If it does reproduce a leak, does it need to be run with any of the different command line options you mentioned earlier? Is the leak in the browser, or in the bokeh server process? How did you conclude there was a leak at all, i.e. what measurement or debugging tools did you use that we can also use to reproduce? This kind of problem is by far the most pernicious to deal with, we will need you assistance to work on it. Basically: err on the side of providing *too much* detail and information.

Thanks,

Bryan

···

On Jul 30, 2018, at 06:08, [email protected] wrote:

You can find a small script below. This doesn' t produce any errors though. The glyphs I am working with are wedges rather than circles.

import numpy as np

from bokeh.models import ColumnDataSource, DataRange1d, Plot, LinearAxis, Grid
from bokeh.plotting import figure,curdoc
from bokeh.models.markers import Circle
from bokeh.io import curdoc, show
from bokeh.models import CustomJS

N = 200000
x = np.linspace(-2, 2, N)
y = x**2
sizes = np.linspace(10, 20, N)
dummy1 = [0]*N
dummy2 = [0]*N
dummy3 = [0]*N
dummy4 = [0]*N
dummy5 = [0]*N
dummy6 = [0]*N
dummy7 = [0]*N
dummy8 = [0]*N
dummy9 = [0]*N

source = ColumnDataSource(dict(x=x, y=y,dummy1 = dummy1,dummy2 = dummy2,dummy3 = dummy3,dummy4 = dummy4,dummy5 = dummy5,dummy6 = dummy6,dummy7 = dummy7,dummy8 = dummy8,dummy9 = dummy9, sizes=sizes))

xdr = DataRange1d()
ydr = DataRange1d()

circleSelected = CustomJS(args = dict( source = source),code = '''
  console.log("do nothing")
  ''')

plot = figure(
    title=None, x_range=xdr, y_range=ydr, plot_width=300, plot_height=300,
    h_symmetry=False, v_symmetry=False, min_border=0, toolbar_location=None)

#glyph = Circle(x="x", y="y", size="sizes", line_color="#3288bd", fill_color="white", line_width=3)
plot.circle(x="x", y="y", size="sizes", line_color="#3288bd", fill_color="white", line_width=3, source = source)
#plot.add_glyph(source, glyph)

xaxis = LinearAxis()
plot.add_layout(xaxis, 'below')

yaxis = LinearAxis()
plot.add_layout(yaxis, 'left')

plot.add_layout(Grid(dimension=0, ticker=xaxis.ticker))
plot.add_layout(Grid(dimension=1, ticker=yaxis.ticker))
plot.js_on_event('tap', circleSelected)

curdoc().add_root(plot)

On Sunday, July 29, 2018 at 10:44:18 PM UTC+3, Bryan Van de ven wrote:
Hi,

There's nothing that can be done or said on something like this without:

* a complete minimal script that demonstrates the issue
* detailed instructions to reproduce the reported result

> could you explain what is happening in background tornado web socket?

I don't really have any idea what information you are asking for, specifically.

Thanks,

Bryan

> On Jul 29, 2018, at 05:35, koseog...@gmail.com wrote:
>
> Hi,
>
> I am trying to update a column data source with len 40000 information having 7 columns. This causes web connection to close. The workaround I found and as earlier suggested to me was to send the data in chunks. That works okay but is there an another way as this requires to set up threads and updating the doc? I found th following issue
>
> Connection errors when sending large amounts of data to browser via a bokeh server · Issue #7374 · bokeh/bokeh · GitHub
>
> which says that websocket_max_message_size is allowed to be overwritten. But when I do so and increase the default 10 mib to 20 mib, I run into another problem of memory leak and besides the connection is closed anyway just because I am tyring to send large amount of data. Is there a solution to this and could you explain what is happening in background tornado web socket?
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
> To post to this group, send email to bo...@continuum.io.
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/a44d4b8d-f66b-4fe1-8b77-4860dcdc2fd8%40continuum.io\.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/31a4cb0b-5553-4f9b-929e-f8d01dfdf2f0%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

5

For that matter: what version of Bokeh? There were memory leaks fixed in the recent past, with fixes now maintained under test. The answer might simply by "upgrade to a fixed version", but it's not possible to say without all the relevant information.

Thanks,

Bryan

···

On Jul 30, 2018, at 11:31, Bryan Van de ven <[email protected]> wrote:

Apologies, but I don't understand the context of "This doesn't produce any errors though" in your statement. Are you saying the script below does not exhibit the memory leak? If it does reproduce a leak, does it need to be run with any of the different command line options you mentioned earlier? Is the leak in the browser, or in the bokeh server process? How did you conclude there was a leak at all, i.e. what measurement or debugging tools did you use that we can also use to reproduce? This kind of problem is by far the most pernicious to deal with, we will need you assistance to work on it. Basically: err on the side of providing *too much* detail and information.

Thanks,

Bryan

On Jul 30, 2018, at 06:08, [email protected] wrote:

You can find a small script below. This doesn' t produce any errors though. The glyphs I am working with are wedges rather than circles.

import numpy as np

from bokeh.models import ColumnDataSource, DataRange1d, Plot, LinearAxis, Grid
from bokeh.plotting import figure,curdoc
from bokeh.models.markers import Circle
from bokeh.io import curdoc, show
from bokeh.models import CustomJS

N = 200000
x = np.linspace(-2, 2, N)
y = x**2
sizes = np.linspace(10, 20, N)
dummy1 = [0]*N
dummy2 = [0]*N
dummy3 = [0]*N
dummy4 = [0]*N
dummy5 = [0]*N
dummy6 = [0]*N
dummy7 = [0]*N
dummy8 = [0]*N
dummy9 = [0]*N

source = ColumnDataSource(dict(x=x, y=y,dummy1 = dummy1,dummy2 = dummy2,dummy3 = dummy3,dummy4 = dummy4,dummy5 = dummy5,dummy6 = dummy6,dummy7 = dummy7,dummy8 = dummy8,dummy9 = dummy9, sizes=sizes))

xdr = DataRange1d()
ydr = DataRange1d()

circleSelected = CustomJS(args = dict( source = source),code = '''
  console.log("do nothing")
  ''')

plot = figure(
   title=None, x_range=xdr, y_range=ydr, plot_width=300, plot_height=300,
   h_symmetry=False, v_symmetry=False, min_border=0, toolbar_location=None)

#glyph = Circle(x="x", y="y", size="sizes", line_color="#3288bd", fill_color="white", line_width=3)
plot.circle(x="x", y="y", size="sizes", line_color="#3288bd", fill_color="white", line_width=3, source = source)
#plot.add_glyph(source, glyph)

xaxis = LinearAxis()
plot.add_layout(xaxis, 'below')

yaxis = LinearAxis()
plot.add_layout(yaxis, 'left')

plot.add_layout(Grid(dimension=0, ticker=xaxis.ticker))
plot.add_layout(Grid(dimension=1, ticker=yaxis.ticker))
plot.js_on_event('tap', circleSelected)

curdoc().add_root(plot)

On Sunday, July 29, 2018 at 10:44:18 PM UTC+3, Bryan Van de ven wrote:
Hi,

There's nothing that can be done or said on something like this without:

* a complete minimal script that demonstrates the issue
* detailed instructions to reproduce the reported result

could you explain what is happening in background tornado web socket?

I don't really have any idea what information you are asking for, specifically.

Thanks,

Bryan

On Jul 29, 2018, at 05:35, koseog...@gmail.com wrote:

Hi,

I am trying to update a column data source with len 40000 information having 7 columns. This causes web connection to close. The workaround I found and as earlier suggested to me was to send the data in chunks. That works okay but is there an another way as this requires to set up threads and updating the doc? I found th following issue

Connection errors when sending large amounts of data to browser via a bokeh server · Issue #7374 · bokeh/bokeh · GitHub

which says that websocket_max_message_size is allowed to be overwritten. But when I do so and increase the default 10 mib to 20 mib, I run into another problem of memory leak and besides the connection is closed anyway just because I am tyring to send large amount of data. Is there a solution to this and could you explain what is happening in background tornado web socket?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
To post to this group, send email to bo...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/a44d4b8d-f66b-4fe1-8b77-4860dcdc2fd8%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/31a4cb0b-5553-4f9b-929e-f8d01dfdf2f0%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.