big data performance issue

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).

If I load web app, speed is good. If I select item from “Select” on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.

2018-08-17_8-13-18.png

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):

rig, job = new, jobs_combx.value

selected_rig = rig

selected_job = job

from_comboBx_group = False

update_main_plot(selected_rig, selected_job, from_comboBx_group)

def jobs_combx_change(attrname, old, new):

rig, job = rigs_combx.value, new

selected_rig = rig

selected_job = job

from_comboBx_group = False

update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()

update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name=‘update_main_plot_thread’, \

target = lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \

args = (update_main_plot_queue, \

doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

rig, \

job, \

from_comboBx_group))

update_main_plot_thread.start()

update_main_plot_event.set()

@gen.coroutine

def update_main_plot(selected_rig, \

selected_job, \

from_comboBx_group, \

checkbox_group_1_selections = ,\

checkbox_group_2_selections = , \

checkbox_group_3_selections = ):

doc = curdoc()

update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

selected_rig, \

selected_job, \

from_comboBx_group))

update_main_plot_event.set()

@without_document_lock

def update_main_plot_chart( doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

rig, \

job, \

from_comboBx_group):

update_main_plot_event.wait()

main_plot_dict = {}

depth_list =

if from_comboBx_group == True:

main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \

rig, job, \

checkbox_group_1_selections, \

checkbox_group_2_selections)

main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)

else:

main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \

rig, job)

main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)

depth_list = [str(x) for x in depth_list]

doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot, main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))

def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):

new_list =

in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])

i = 0

for item in in_all_data[‘HoleDepth’]:

if item != ‘-1’:

if i >= in_mainplot_data_length:

if in_mainplot_data_type == ‘VBarColors’:

new_list.append(‘white’)

else:

new_list.append(’’)

else:

var = in_mainplot_data[in_mainplot_data_type][i]

new_list.append(var)

i = i + 1

else:

if in_mainplot_data_type == ‘VBarColors’:

new_list.append(‘white’)

else:

new_list.append(’’)

return new_list

@gen.coroutine

def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):

mainplot_source.data[‘HoleDepth’] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)

mainplot_source.data[‘VBarTop’] = updateSourceData(‘VBarTop’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarBottom’] = updateSourceData(‘VBarBottom’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarColors’] = updateSourceData(‘VBarColors’, main_plot_dict, mainplot_source.data)

vBarType_list = updateSourceData(‘VBarType’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarType’] = vBarType_list

main_plot.x_range.factors =

main_plot.x_range.factors = depth_list

``

Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.

You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.

···

On Aug 17, 2018, at 4:29 PM, peng wang [email protected] wrote:

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).

If I load web app, speed is good. If I select item from “Select” on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.

<2018-08-17_8-13-18.png>

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):

rig, job = new, jobs_combx.value

selected_rig = rig

selected_job = job

from_comboBx_group = False

update_main_plot(selected_rig, selected_job, from_comboBx_group)

def jobs_combx_change(attrname, old, new):

rig, job = rigs_combx.value, new

selected_rig = rig

selected_job = job

from_comboBx_group = False

update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()

update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name=‘update_main_plot_thread’, \

target = lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \

args = (update_main_plot_queue, \

doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

rig, \

job, \

from_comboBx_group))

update_main_plot_thread.start()

update_main_plot_event.set()

@gen.coroutine

def update_main_plot(selected_rig, \

selected_job, \

from_comboBx_group, \

checkbox_group_1_selections = ,\

checkbox_group_2_selections = , \

checkbox_group_3_selections = ):

doc = curdoc()

update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

selected_rig, \

selected_job, \

from_comboBx_group))

update_main_plot_event.set()

@without_document_lock

def update_main_plot_chart( doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

rig, \

job, \

from_comboBx_group):

update_main_plot_event.wait()

main_plot_dict = {}

depth_list =

if from_comboBx_group == True:

main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \

rig, job, \

checkbox_group_1_selections, \

checkbox_group_2_selections)

main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)

else:

main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \

rig, job)

main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)

depth_list = [str(x) for x in depth_list]

doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot, main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))

def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):

new_list =

in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])

i = 0

for item in in_all_data[‘HoleDepth’]:

if item != ‘-1’:

if i >= in_mainplot_data_length:

if in_mainplot_data_type == ‘VBarColors’:

new_list.append(‘white’)

else:

new_list.append(’’)

else:

var = in_mainplot_data[in_mainplot_data_type][i]

new_list.append(var)

i = i + 1

else:

if in_mainplot_data_type == ‘VBarColors’:

new_list.append(‘white’)

else:

new_list.append(’’)

return new_list

@gen.coroutine

def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):

mainplot_source.data[‘HoleDepth’] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)

mainplot_source.data[‘VBarTop’] = updateSourceData(‘VBarTop’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarBottom’] = updateSourceData(‘VBarBottom’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarColors’] = updateSourceData(‘VBarColors’, main_plot_dict, mainplot_source.data)

vBarType_list = updateSourceData(‘VBarType’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarType’] = vBarType_list

main_plot.x_range.factors =

main_plot.x_range.factors = depth_list

``

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

<2018-08-17_8-13-18.png>

Hi,

There's not much information here to go on. However, the plots you show are not big at all, drawing for sending data to them should not be any sort of problem.

I do notice you are using updating your column data source in a sub-optimal way. When you update individual columns one at a time, i.e:

    source.data['foo'] = new_foo_data
    source.data['bar'] = new_bar_data

That pattern causes a separate update to be sent over the network on every single line, causing separate re-draws for every line. It is always a better idea to try and update a CDS "all at once", i.e.

  new _data = {'foo': new_foo_data, 'bar': new_bar_data }
  source.data = new_data

That will only result in a single update and single re-draw.

Additionally, I see you are appending to your columns? If that is the case you should try to arrange things so you can use the .stream method of the column data source. If you are only ever appending new data, then stream is *much* more efficient, it will only send the new data, instead of re-sending all the old data every time.

Finally are you sure your callback itself is not doing expensive blocking work?

Thanks,

Bryan

···

On Aug 17, 2018, at 10:16, 'Corey O'Meara' via Bokeh Discussion - Public <[email protected]> wrote:

Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.

You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.

Sent from my iPhone

On Aug 17, 2018, at 4:29 PM, peng wang <[email protected]> wrote:

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).
If I load web app, speed is good. If I select item from "Select" on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.
<2018-08-17_8-13-18.png>

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):
        rig, job = new, jobs_combx.value
        selected_rig = rig
        selected_job = job
    
        from_comboBx_group = False
        update_main_plot(selected_rig, selected_job, from_comboBx_group)

def jobs_combx_change(attrname, old, new):
        rig, job = rigs_combx.value, new
        selected_rig = rig
        selected_job = job
        from_comboBx_group = False
        update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()
update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name='update_main_plot_thread', \
                                     target = lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \
                                     args = (update_main_plot_queue, \
                                             doc, \
                                             update_main_plot_event, \
                                             mainplot_source, \
                                             main_plot, \
                                             mainplot_data_all, \
                                             checkbox_group_1_selections, \
                                             checkbox_group_2_selections,\
                                             checkbox_group_3_selections, \
                                             all_connection_dict,\
                                             rig, \
                                             job, \
                                             from_comboBx_group))
update_main_plot_thread.start()
update_main_plot_event.set()

@gen.coroutine
def update_main_plot(selected_rig, \
                    selected_job, \
                    from_comboBx_group, \
                    checkbox_group_1_selections = ,\
                    checkbox_group_2_selections = , \
                    checkbox_group_3_selections = ):

        doc = curdoc()
        update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \
                                             update_main_plot_event, \
                                             mainplot_source, \
                                             main_plot, \
                                             mainplot_data_all, \
                                             checkbox_group_1_selections, \
                                             checkbox_group_2_selections,\
                                             checkbox_group_3_selections, \
                                             all_connection_dict,\
                                             selected_rig, \
                                             selected_job, \
                                             from_comboBx_group))
        update_main_plot_event.set()

@without_document_lock
def update_main_plot_chart( doc, \
                            update_main_plot_event, \
                            mainplot_source, \
                            main_plot, \
                            mainplot_data_all, \
                            checkbox_group_1_selections, \
                            checkbox_group_2_selections,\
                            checkbox_group_3_selections, \
                            all_connection_dict,\
                            rig, \
                            job, \
                            from_comboBx_group):
    update_main_plot_event.wait()
    
    main_plot_dict = {}
    depth_list =
   
    if from_comboBx_group == True:
        main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                    rig, job, \
                                                    checkbox_group_1_selections, \
                                                    checkbox_group_2_selections)
        main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)
    else:
        main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                    rig, job)
        main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)
        
    depth_list = [str(x) for x in depth_list]
    doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot, main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))
    
def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):
    new_list =

    in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])
    i = 0
    for item in in_all_data['HoleDepth']:
        if item != '-1':
            if i >= in_mainplot_data_length:
                if in_mainplot_data_type == 'VBarColors':
                    new_list.append('white')
                else:
                    new_list.append('')
            else:
                var = in_mainplot_data[in_mainplot_data_type][i]
                new_list.append(var)
                i = i + 1
        else:
            if in_mainplot_data_type == 'VBarColors':
                new_list.append('white')
            else:
                new_list.append('')
   
    return new_list

@gen.coroutine
def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):
    mainplot_source.data['HoleDepth'] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)
    mainplot_source.data['VBarTop'] = updateSourceData('VBarTop', main_plot_dict, mainplot_source.data)
    mainplot_source.data['VBarBottom'] = updateSourceData('VBarBottom', main_plot_dict, mainplot_source.data)
    mainplot_source.data['VBarColors'] = updateSourceData('VBarColors', main_plot_dict, mainplot_source.data)
    vBarType_list = updateSourceData('VBarType', main_plot_dict, mainplot_source.data)
    mainplot_source.data['VBarType'] = vBarType_list
    main_plot.x_range.factors =
    main_plot.x_range.factors = depth_list

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.
<2018-08-17_8-13-18.png>

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4F3A4C16-951A-4F5C-AEA2-17F93272F0A9%40googlemail.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Good idea, I will give it try. thanks a lot

···

On Friday, August 17, 2018 at 11:16:52 AM UTC-6, omearac wrote:

Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.

You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.

Sent from my iPhone

On Aug 17, 2018, at 4:29 PM, peng wang [email protected] wrote:

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).

If I load web app, speed is good. If I select item from “Select” on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.

<2018-08-17_8-13-18.png>

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):

rig, job = new, jobs_combx.value

selected_rig = rig

selected_job = job

from_comboBx_group = False

update_main_plot(selected_rig, selected_job, from_comboBx_group)

def jobs_combx_change(attrname, old, new):

rig, job = rigs_combx.value, new

selected_rig = rig

selected_job = job

from_comboBx_group = False

update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()

update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name=‘update_main_plot_thread’, \

target = lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \

args = (update_main_plot_queue, \

doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

rig, \

job, \

from_comboBx_group))

update_main_plot_thread.start()

update_main_plot_event.set()

@gen.coroutine

def update_main_plot(selected_rig, \

selected_job, \

from_comboBx_group, \

checkbox_group_1_selections = ,\

checkbox_group_2_selections = , \

checkbox_group_3_selections = ):

doc = curdoc()

update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

selected_rig, \

selected_job, \

from_comboBx_group))

update_main_plot_event.set()

@without_document_lock

def update_main_plot_chart( doc, \

update_main_plot_event, \

mainplot_source, \

main_plot, \

mainplot_data_all, \

checkbox_group_1_selections, \

checkbox_group_2_selections,\

checkbox_group_3_selections, \

all_connection_dict,\

rig, \

job, \

from_comboBx_group):

update_main_plot_event.wait()

main_plot_dict = {}

depth_list =

if from_comboBx_group == True:

main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \

rig, job, \

checkbox_group_1_selections, \

checkbox_group_2_selections)

main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)

else:

main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \

rig, job)

main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)

depth_list = [str(x) for x in depth_list]

doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot, main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))

def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):

new_list =

in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])

i = 0

for item in in_all_data[‘HoleDepth’]:

if item != ‘-1’:

if i >= in_mainplot_data_length:

if in_mainplot_data_type == ‘VBarColors’:

new_list.append(‘white’)

else:

new_list.append(’’)

else:

var = in_mainplot_data[in_mainplot_data_type][i]

new_list.append(var)

i = i + 1

else:

if in_mainplot_data_type == ‘VBarColors’:

new_list.append(‘white’)

else:

new_list.append(’’)

return new_list

@gen.coroutine

def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):

mainplot_source.data[‘HoleDepth’] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)

mainplot_source.data[‘VBarTop’] = updateSourceData(‘VBarTop’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarBottom’] = updateSourceData(‘VBarBottom’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarColors’] = updateSourceData(‘VBarColors’, main_plot_dict, mainplot_source.data)

vBarType_list = updateSourceData(‘VBarType’, main_plot_dict, mainplot_source.data)

mainplot_source.data[‘VBarType’] = vBarType_list

main_plot.x_range.factors =

main_plot.x_range.factors = depth_list

``

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

<2018-08-17_8-13-18.png>

Hi there,

Thanks a lot for response,

I modified the solution as displayed below, the performance of the app is pretty fast

2018-08-20_13-17-28.png

I will continue to work on this app based on the suggestion, and try to update data datasource by “CDS” and .stream

After implementation and test, I will update this thread.

Cheers

···

On Monday, August 20, 2018 at 10:14:04 AM UTC-6, Bryan Van de ven wrote:

Hi,

There’s not much information here to go on. However, the plots you show are not big at all, drawing for sending data to them should not be any sort of problem.

I do notice you are using updating your column data source in a sub-optimal way. When you update individual columns one at a time, i.e:

source.data['foo'] = new_foo_data

source.data['bar'] = new_bar_data

That pattern causes a separate update to be sent over the network on every single line, causing separate re-draws for every line. It is always a better idea to try and update a CDS “all at once”, i.e.

    new _data = {'foo': new_foo_data, 'bar': new_bar_data }

    source.data = new_data

That will only result in a single update and single re-draw.

Additionally, I see you are appending to your columns? If that is the case you should try to arrange things so you can use the .stream method of the column data source. If you are only ever appending new data, then stream is much more efficient, it will only send the new data, instead of re-sending all the old data every time.

Finally are you sure your callback itself is not doing expensive blocking work?

Thanks,

Bryan

On Aug 17, 2018, at 10:16, ‘Corey O’Meara’ via Bokeh Discussion - Public [email protected] wrote:

Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.

You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.

Sent from my iPhone

On Aug 17, 2018, at 4:29 PM, peng wang [email protected] wrote:

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).

If I load web app, speed is good. If I select item from “Select” on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.

<2018-08-17_8-13-18.png>

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):

    rig, job = new, jobs_combx.value
    selected_rig = rig
    selected_job = job
    from_comboBx_group = False
    update_main_plot(selected_rig, selected_job, from_comboBx_group)      

def jobs_combx_change(attrname, old, new):

    rig, job = rigs_combx.value, new
    selected_rig = rig
    selected_job = job        
    from_comboBx_group = False
    update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()

update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name=‘update_main_plot_thread’, \

                                 target =  lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \
                                 args = (update_main_plot_queue, \
                                         doc, \
                                         update_main_plot_event, \
                                         mainplot_source, \
                                         main_plot, \
                                         mainplot_data_all, \
                                         checkbox_group_1_selections, \
                                         checkbox_group_2_selections,\
                                         checkbox_group_3_selections, \
                                         all_connection_dict,\
                                         rig, \
                                         job, \
                                         from_comboBx_group))

update_main_plot_thread.start()

update_main_plot_event.set()

@gen.coroutine

def update_main_plot(selected_rig, \

                selected_job, \
                from_comboBx_group, \
                checkbox_group_1_selections = [],\
                checkbox_group_2_selections = [], \
                checkbox_group_3_selections = []):
    doc = curdoc()
    update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \
                                         update_main_plot_event, \
                                         mainplot_source, \
                                         main_plot, \
                                         mainplot_data_all, \
                                         checkbox_group_1_selections, \
                                         checkbox_group_2_selections,\
                                         checkbox_group_3_selections, \
                                         all_connection_dict,\
                                         selected_rig, \
                                         selected_job, \
                                         from_comboBx_group))
    update_main_plot_event.set()

@without_document_lock

def update_main_plot_chart( doc, \

                        update_main_plot_event, \
                        mainplot_source, \
                        main_plot, \
                        mainplot_data_all, \
                        checkbox_group_1_selections, \
                        checkbox_group_2_selections,\
                        checkbox_group_3_selections, \
                        all_connection_dict,\
                        rig, \
                        job, \
                        from_comboBx_group):
update_main_plot_event.wait()        
main_plot_dict = {}
depth_list = []
if from_comboBx_group == True:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job, \
                                                checkbox_group_1_selections, \
                                                checkbox_group_2_selections)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)
else:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)
depth_list = [str(x) for x in depth_list]
doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot,  main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))

def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):

new_list = []
in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])
i = 0
for item in in_all_data['HoleDepth']:
    if item != '-1':
        if i >= in_mainplot_data_length:
            if in_mainplot_data_type == 'VBarColors':
                new_list.append('white')
            else:
                new_list.append('')
        else:  
            var = in_mainplot_data[in_mainplot_data_type][i]
            new_list.append(var)
            i = i + 1
    else:
        if in_mainplot_data_type == 'VBarColors':
            new_list.append('white')
        else:
            new_list.append('')
return new_list

@gen.coroutine

def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):

mainplot_source.data['HoleDepth'] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)
mainplot_source.data['VBarTop'] = updateSourceData('VBarTop', main_plot_dict, mainplot_source.data)      
mainplot_source.data['VBarBottom'] = updateSourceData('VBarBottom', main_plot_dict, mainplot_source.data)
mainplot_source.data['VBarColors'] = updateSourceData('VBarColors', main_plot_dict, mainplot_source.data)
vBarType_list = updateSourceData('VBarType', main_plot_dict, mainplot_source.data)
mainplot_source.data['VBarType'] = vBarType_list
main_plot.x_range.factors = []
main_plot.x_range.factors = depth_list


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

<2018-08-17_8-13-18.png>


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4F3A4C16-951A-4F5C-AEA2-17F93272F0A9%40googlemail.com.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Hi there,

I also come across a very wired issue related to my web app.

every time when I launch the web app, loading time is reasonably short, and then I click on any UI such as “Selections” or “Checkboxes”. the web app can not response untill the following error meaasge appears in “console”. Would someone let me know the possible reason? My web app performance is dramatically affected by this issue.

Thanks

2018-08-20_14-22-26.png

···

On Monday, August 20, 2018 at 10:14:04 AM UTC-6, Bryan Van de ven wrote:

Hi,

There’s not much information here to go on. However, the plots you show are not big at all, drawing for sending data to them should not be any sort of problem.

I do notice you are using updating your column data source in a sub-optimal way. When you update individual columns one at a time, i.e:

source.data['foo'] = new_foo_data

source.data['bar'] = new_bar_data

That pattern causes a separate update to be sent over the network on every single line, causing separate re-draws for every line. It is always a better idea to try and update a CDS “all at once”, i.e.

    new _data = {'foo': new_foo_data, 'bar': new_bar_data }

    source.data = new_data

That will only result in a single update and single re-draw.

Additionally, I see you are appending to your columns? If that is the case you should try to arrange things so you can use the .stream method of the column data source. If you are only ever appending new data, then stream is much more efficient, it will only send the new data, instead of re-sending all the old data every time.

Finally are you sure your callback itself is not doing expensive blocking work?

Thanks,

Bryan

On Aug 17, 2018, at 10:16, ‘Corey O’Meara’ via Bokeh Discussion - Public [email protected] wrote:

Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.

You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.

Sent from my iPhone

On Aug 17, 2018, at 4:29 PM, peng wang [email protected] wrote:

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).

If I load web app, speed is good. If I select item from “Select” on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.

<2018-08-17_8-13-18.png>

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):

    rig, job = new, jobs_combx.value
    selected_rig = rig
    selected_job = job
    from_comboBx_group = False
    update_main_plot(selected_rig, selected_job, from_comboBx_group)      

def jobs_combx_change(attrname, old, new):

    rig, job = rigs_combx.value, new
    selected_rig = rig
    selected_job = job        
    from_comboBx_group = False
    update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()

update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name=‘update_main_plot_thread’, \

                                 target =  lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \
                                 args = (update_main_plot_queue, \
                                         doc, \
                                         update_main_plot_event, \
                                         mainplot_source, \
                                         main_plot, \
                                         mainplot_data_all, \
                                         checkbox_group_1_selections, \
                                         checkbox_group_2_selections,\
                                         checkbox_group_3_selections, \
                                         all_connection_dict,\
                                         rig, \
                                         job, \
                                         from_comboBx_group))

update_main_plot_thread.start()

update_main_plot_event.set()

@gen.coroutine

def update_main_plot(selected_rig, \

                selected_job, \
                from_comboBx_group, \
                checkbox_group_1_selections = [],\
                checkbox_group_2_selections = [], \
                checkbox_group_3_selections = []):
    doc = curdoc()
    update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \
                                         update_main_plot_event, \
                                         mainplot_source, \
                                         main_plot, \
                                         mainplot_data_all, \
                                         checkbox_group_1_selections, \
                                         checkbox_group_2_selections,\
                                         checkbox_group_3_selections, \
                                         all_connection_dict,\
                                         selected_rig, \
                                         selected_job, \
                                         from_comboBx_group))
    update_main_plot_event.set()

@without_document_lock

def update_main_plot_chart( doc, \

                        update_main_plot_event, \
                        mainplot_source, \
                        main_plot, \
                        mainplot_data_all, \
                        checkbox_group_1_selections, \
                        checkbox_group_2_selections,\
                        checkbox_group_3_selections, \
                        all_connection_dict,\
                        rig, \
                        job, \
                        from_comboBx_group):
update_main_plot_event.wait()        
main_plot_dict = {}
depth_list = []
if from_comboBx_group == True:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job, \
                                                checkbox_group_1_selections, \
                                                checkbox_group_2_selections)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)
else:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)
depth_list = [str(x) for x in depth_list]
doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot,  main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))

def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):

new_list = []
in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])
i = 0
for item in in_all_data['HoleDepth']:
    if item != '-1':
        if i >= in_mainplot_data_length:
            if in_mainplot_data_type == 'VBarColors':
                new_list.append('white')
            else:
                new_list.append('')
        else:  
            var = in_mainplot_data[in_mainplot_data_type][i]
            new_list.append(var)
            i = i + 1
    else:
        if in_mainplot_data_type == 'VBarColors':
            new_list.append('white')
        else:
            new_list.append('')
return new_list

@gen.coroutine

def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):

mainplot_source.data['HoleDepth'] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)
mainplot_source.data['VBarTop'] = updateSourceData('VBarTop', main_plot_dict, mainplot_source.data)      
mainplot_source.data['VBarBottom'] = updateSourceData('VBarBottom', main_plot_dict, mainplot_source.data)
mainplot_source.data['VBarColors'] = updateSourceData('VBarColors', main_plot_dict, mainplot_source.data)
vBarType_list = updateSourceData('VBarType', main_plot_dict, mainplot_source.data)
mainplot_source.data['VBarType'] = vBarType_list
main_plot.x_range.factors = []
main_plot.x_range.factors = depth_list


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

<2018-08-17_8-13-18.png>


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4F3A4C16-951A-4F5C-AEA2-17F93272F0A9%40googlemail.com.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

new problem, when I update the columndatasource by method:
new _data = {‘foo’: new_foo_data, ‘bar’: new_bar_data }
source.data = new_data

other than:

source.data[‘foo’] = new_foo_data

source.data[‘bar’] = new_bar_data

2018-08-20_16-39-43.png

on my local pc, websocket is never lost, but it is lost when I deploy the web app on Apache. please refer to the following pic

When I just use less tables from database, for example, 5 tables, websocket is never lost, when I use more tables, for example, 10 to 15 tables, that is, more data, the websocket is immediately lost when I launch the web.

I came across the same issue when I deploy the web app on IIS, and then I use Apache, everything is fine. this is the first time that this issue appears again on Apache.

Thanks for any ideas.

···

On Monday, August 20, 2018 at 10:14:04 AM UTC-6, Bryan Van de ven wrote:

Hi,

There’s not much information here to go on. However, the plots you show are not big at all, drawing for sending data to them should not be any sort of problem.

I do notice you are using updating your column data source in a sub-optimal way. When you update individual columns one at a time, i.e:

source.data['foo'] = new_foo_data

source.data['bar'] = new_bar_data

That pattern causes a separate update to be sent over the network on every single line, causing separate re-draws for every line. It is always a better idea to try and update a CDS “all at once”, i.e.

    new _data = {'foo': new_foo_data, 'bar': new_bar_data }

    source.data = new_data

That will only result in a single update and single re-draw.

Additionally, I see you are appending to your columns? If that is the case you should try to arrange things so you can use the .stream method of the column data source. If you are only ever appending new data, then stream is much more efficient, it will only send the new data, instead of re-sending all the old data every time.

Finally are you sure your callback itself is not doing expensive blocking work?

Thanks,

Bryan

On Aug 17, 2018, at 10:16, ‘Corey O’Meara’ via Bokeh Discussion - Public [email protected] wrote:

Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.

You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.

Sent from my iPhone

On Aug 17, 2018, at 4:29 PM, peng wang [email protected] wrote:

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).

If I load web app, speed is good. If I select item from “Select” on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.

<2018-08-17_8-13-18.png>

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):

    rig, job = new, jobs_combx.value
    selected_rig = rig
    selected_job = job
    from_comboBx_group = False
    update_main_plot(selected_rig, selected_job, from_comboBx_group)      

def jobs_combx_change(attrname, old, new):

    rig, job = rigs_combx.value, new
    selected_rig = rig
    selected_job = job        
    from_comboBx_group = False
    update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()

update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name=‘update_main_plot_thread’, \

                                 target =  lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \
                                 args = (update_main_plot_queue, \
                                         doc, \
                                         update_main_plot_event, \
                                         mainplot_source, \
                                         main_plot, \
                                         mainplot_data_all, \
                                         checkbox_group_1_selections, \
                                         checkbox_group_2_selections,\
                                         checkbox_group_3_selections, \
                                         all_connection_dict,\
                                         rig, \
                                         job, \
                                         from_comboBx_group))

update_main_plot_thread.start()

update_main_plot_event.set()

@gen.coroutine

def update_main_plot(selected_rig, \

                selected_job, \
                from_comboBx_group, \
                checkbox_group_1_selections = [],\
                checkbox_group_2_selections = [], \
                checkbox_group_3_selections = []):
    doc = curdoc()
    update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \
                                         update_main_plot_event, \
                                         mainplot_source, \
                                         main_plot, \
                                         mainplot_data_all, \
                                         checkbox_group_1_selections, \
                                         checkbox_group_2_selections,\
                                         checkbox_group_3_selections, \
                                         all_connection_dict,\
                                         selected_rig, \
                                         selected_job, \
                                         from_comboBx_group))
    update_main_plot_event.set()

@without_document_lock

def update_main_plot_chart( doc, \

                        update_main_plot_event, \
                        mainplot_source, \
                        main_plot, \
                        mainplot_data_all, \
                        checkbox_group_1_selections, \
                        checkbox_group_2_selections,\
                        checkbox_group_3_selections, \
                        all_connection_dict,\
                        rig, \
                        job, \
                        from_comboBx_group):
update_main_plot_event.wait()        
main_plot_dict = {}
depth_list = []
if from_comboBx_group == True:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job, \
                                                checkbox_group_1_selections, \
                                                checkbox_group_2_selections)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)
else:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)
depth_list = [str(x) for x in depth_list]
doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot,  main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))

def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):

new_list = []
in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])
i = 0
for item in in_all_data['HoleDepth']:
    if item != '-1':
        if i >= in_mainplot_data_length:
            if in_mainplot_data_type == 'VBarColors':
                new_list.append('white')
            else:
                new_list.append('')
        else:  
            var = in_mainplot_data[in_mainplot_data_type][i]
            new_list.append(var)
            i = i + 1
    else:
        if in_mainplot_data_type == 'VBarColors':
            new_list.append('white')
        else:
            new_list.append('')
return new_list

@gen.coroutine

def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):

mainplot_source.data['HoleDepth'] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)
mainplot_source.data['VBarTop'] = updateSourceData('VBarTop', main_plot_dict, mainplot_source.data)      
mainplot_source.data['VBarBottom'] = updateSourceData('VBarBottom', main_plot_dict, mainplot_source.data)
mainplot_source.data['VBarColors'] = updateSourceData('VBarColors', main_plot_dict, mainplot_source.data)
vBarType_list = updateSourceData('VBarType', main_plot_dict, mainplot_source.data)
mainplot_source.data['VBarType'] = vBarType_list
main_plot.x_range.factors = []
main_plot.x_range.factors = depth_list


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

<2018-08-17_8-13-18.png>


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4F3A4C16-951A-4F5C-AEA2-17F93272F0A9%40googlemail.com.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Hi,

Tornado has a max web socket size, and sending messages above that size will result in closed connections. In recent versions of Bokeh, this max value is user configurable with the --websocket-max-message-size command line option. But if it's working behind Apache, and not IIS, then it sounds like some limitation of IIS in particular, than the tornado message size limit. Unfortunately I have no experience with IIS to offer any further suggestions.

Bryan

···

On Aug 20, 2018, at 15:49, peng wang <[email protected]> wrote:

new problem, when I update the columndatasource by method:
        new _data = {'foo': new_foo_data, 'bar': new_bar_data }
        source.data = new_data

other than:
     source.data['foo'] = new_foo_data
     source.data['bar'] = new_bar_data

<2018-08-20_16-39-43.png>

on my local pc, websocket is never lost, but it is lost when I deploy the web app on Apache. please refer to the following pic

When I just use less tables from database, for example, 5 tables, websocket is never lost, when I use more tables, for example, 10 to 15 tables, that is, more data, the websocket is immediately lost when I launch the web.

I came across the same issue when I deploy the web app on IIS, and then I use Apache, everything is fine. this is the first time that this issue appears again on Apache.

Thanks for any ideas.

On Monday, August 20, 2018 at 10:14:04 AM UTC-6, Bryan Van de ven wrote:
Hi,

There's not much information here to go on. However, the plots you show are not big at all, drawing for sending data to them should not be any sort of problem.

I do notice you are using updating your column data source in a sub-optimal way. When you update individual columns one at a time, i.e:

    source.data['foo'] = new_foo_data
    source.data['bar'] = new_bar_data

That pattern causes a separate update to be sent over the network on every single line, causing separate re-draws for every line. It is always a better idea to try and update a CDS "all at once", i.e.

        new _data = {'foo': new_foo_data, 'bar': new_bar_data }
        source.data = new_data

That will only result in a single update and single re-draw.

Additionally, I see you are appending to your columns? If that is the case you should try to arrange things so you can use the .stream method of the column data source. If you are only ever appending new data, then stream is *much* more efficient, it will only send the new data, instead of re-sending all the old data every time.

Finally are you sure your callback itself is not doing expensive blocking work?

Thanks,

Bryan

> On Aug 17, 2018, at 10:16, 'Corey O'Meara' via Bokeh Discussion - Public <[email protected]> wrote:
>
> Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.
>
> You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.
>
> Sent from my iPhone
>
> On Aug 17, 2018, at 4:29 PM, peng wang <[email protected]> wrote:
>
>> Hi guys,
>>
>> Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).
>> If I load web app, speed is good. If I select item from "Select" on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.
>> <2018-08-17_8-13-18.png>
>>
>>
>>
>> but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time
>>
>> I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.
>>
>> I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.
>>
>> the code is post below.
>>
>> thanks
>>
>>
>> def rigs_combx_change(attrname, old, new):
>> rig, job = new, jobs_combx.value
>> selected_rig = rig
>> selected_job = job
>>
>> from_comboBx_group = False
>> update_main_plot(selected_rig, selected_job, from_comboBx_group)
>>
>> def jobs_combx_change(attrname, old, new):
>> rig, job = rigs_combx.value, new
>> selected_rig = rig
>> selected_job = job
>> from_comboBx_group = False
>> update_main_plot(selected_rig, selected_job, from_comboBx_group)
>>
>> update_main_plot_queue = queue.Queue()
>> update_main_plot_event = threading.Event()
>>
>> update_main_plot_thread = Thread(name='update_main_plot_thread', \
>> target = lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \
>> args = (update_main_plot_queue, \
>> doc, \
>> update_main_plot_event, \
>> mainplot_source, \
>> main_plot, \
>> mainplot_data_all, \
>> checkbox_group_1_selections, \
>> checkbox_group_2_selections,\
>> checkbox_group_3_selections, \
>> all_connection_dict,\
>> rig, \
>> job, \
>> from_comboBx_group))
>> update_main_plot_thread.start()
>> update_main_plot_event.set()
>>
>>
>>
>>
>> @gen.coroutine
>> def update_main_plot(selected_rig, \
>> selected_job, \
>> from_comboBx_group, \
>> checkbox_group_1_selections = ,\
>> checkbox_group_2_selections = , \
>> checkbox_group_3_selections = ):
>>
>> doc = curdoc()
>> update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \
>> update_main_plot_event, \
>> mainplot_source, \
>> main_plot, \
>> mainplot_data_all, \
>> checkbox_group_1_selections, \
>> checkbox_group_2_selections,\
>> checkbox_group_3_selections, \
>> all_connection_dict,\
>> selected_rig, \
>> selected_job, \
>> from_comboBx_group))
>> update_main_plot_event.set()
>>
>>
>> @without_document_lock
>> def update_main_plot_chart( doc, \
>> update_main_plot_event, \
>> mainplot_source, \
>> main_plot, \
>> mainplot_data_all, \
>> checkbox_group_1_selections, \
>> checkbox_group_2_selections,\
>> checkbox_group_3_selections, \
>> all_connection_dict,\
>> rig, \
>> job, \
>> from_comboBx_group):
>> update_main_plot_event.wait()
>>
>> main_plot_dict = {}
>> depth_list =
>>
>> if from_comboBx_group == True:
>> main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
>> rig, job, \
>> checkbox_group_1_selections, \
>> checkbox_group_2_selections)
>> main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)
>> else:
>> main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
>> rig, job)
>> main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)
>>
>> depth_list = [str(x) for x in depth_list]
>> doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot, main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))
>>
>>
>> def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):
>> new_list =
>>
>> in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])
>> i = 0
>> for item in in_all_data['HoleDepth']:
>> if item != '-1':
>> if i >= in_mainplot_data_length:
>> if in_mainplot_data_type == 'VBarColors':
>> new_list.append('white')
>> else:
>> new_list.append('')
>> else:
>> var = in_mainplot_data[in_mainplot_data_type][i]
>> new_list.append(var)
>> i = i + 1
>> else:
>> if in_mainplot_data_type == 'VBarColors':
>> new_list.append('white')
>> else:
>> new_list.append('')
>>
>> return new_list
>>
>>
>>
>> @gen.coroutine
>> def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):
>> mainplot_source.data['HoleDepth'] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)
>> mainplot_source.data['VBarTop'] = updateSourceData('VBarTop', main_plot_dict, mainplot_source.data)
>> mainplot_source.data['VBarBottom'] = updateSourceData('VBarBottom', main_plot_dict, mainplot_source.data)
>> mainplot_source.data['VBarColors'] = updateSourceData('VBarColors', main_plot_dict, mainplot_source.data)
>> vBarType_list = updateSourceData('VBarType', main_plot_dict, mainplot_source.data)
>> mainplot_source.data['VBarType'] = vBarType_list
>> main_plot.x_range.factors =
>> main_plot.x_range.factors = depth_list
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
>> To post to this group, send email to [email protected].
>> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.
>> For more options, visit https://groups.google.com/a/continuum.io/d/optout.
>> <2018-08-17_8-13-18.png>
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4F3A4C16-951A-4F5C-AEA2-17F93272F0A9%40googlemail.com.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/add030fa-5c70-4f66-9b05-0b3e23bbb1b7%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.
<2018-08-20_16-39-43.png>

Thanks for your response. I suddenly notice that the problem can be caused if the sizes or lengths of columns in column data source do not equal, is this possible?

Cheers

···

Sent from Peng’s Gmail Mobile

Hi,

No that is absolutely not supported. It is a foundational assumption of CDS that all column always have the same length at all times, always. Not maintaining that invariant is 100% undefined behavior and you should get a very loud warning if it ever occurs. This is no different than Pandas DataFrames, e.g., which also always have columns of identical length.

Thanks,

Bryan

···

On Aug 20, 2018, at 17:27, Jeffery wang <[email protected]> wrote:

Thanks for your response. I suddenly notice that the problem can be caused if the sizes or lengths of columns in column data source do not equal, is this possible?

Cheers

On Mon, Aug 20, 2018 at 5:08 PM Bryan Van de ven <[email protected]> wrote:
Hi,

Tornado has a max web socket size, and sending messages above that size will result in closed connections. In recent versions of Bokeh, this max value is user configurable with the --websocket-max-message-size command line option. But if it's working behind Apache, and not IIS, then it sounds like some limitation of IIS in particular, than the tornado message size limit. Unfortunately I have no experience with IIS to offer any further suggestions.

Bryan

> On Aug 20, 2018, at 15:49, peng wang <[email protected]> wrote:
>
> new problem, when I update the columndatasource by method:
> new _data = {'foo': new_foo_data, 'bar': new_bar_data }
> source.data = new_data
>
> other than:
> source.data['foo'] = new_foo_data
> source.data['bar'] = new_bar_data
>
> <2018-08-20_16-39-43.png>
>
> on my local pc, websocket is never lost, but it is lost when I deploy the web app on Apache. please refer to the following pic
>
> When I just use less tables from database, for example, 5 tables, websocket is never lost, when I use more tables, for example, 10 to 15 tables, that is, more data, the websocket is immediately lost when I launch the web.
>
> I came across the same issue when I deploy the web app on IIS, and then I use Apache, everything is fine. this is the first time that this issue appears again on Apache.
>
> Thanks for any ideas.
>
>
>
> On Monday, August 20, 2018 at 10:14:04 AM UTC-6, Bryan Van de ven wrote:
> Hi,
>
> There's not much information here to go on. However, the plots you show are not big at all, drawing for sending data to them should not be any sort of problem.
>
> I do notice you are using updating your column data source in a sub-optimal way. When you update individual columns one at a time, i.e:
>
> source.data['foo'] = new_foo_data
> source.data['bar'] = new_bar_data
>
> That pattern causes a separate update to be sent over the network on every single line, causing separate re-draws for every line. It is always a better idea to try and update a CDS "all at once", i.e.
>
> new _data = {'foo': new_foo_data, 'bar': new_bar_data }
> source.data = new_data
>
> That will only result in a single update and single re-draw.
>
> Additionally, I see you are appending to your columns? If that is the case you should try to arrange things so you can use the .stream method of the column data source. If you are only ever appending new data, then stream is *much* more efficient, it will only send the new data, instead of re-sending all the old data every time.
>
> Finally are you sure your callback itself is not doing expensive blocking work?
>
> Thanks,
>
> Bryan
>
> > On Aug 17, 2018, at 10:16, 'Corey O'Meara' via Bokeh Discussion - Public <[email protected]> wrote:
> >
> > Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.
> >
> > You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.
> >
> > Sent from my iPhone
> >
> > On Aug 17, 2018, at 4:29 PM, peng wang <[email protected]> wrote:
> >
> >> Hi guys,
> >>
> >> Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).
> >> If I load web app, speed is good. If I select item from "Select" on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.
> >> <2018-08-17_8-13-18.png>
> >>
> >>
> >>
> >> but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time
> >>
> >> I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.
> >>
> >> I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.
> >>
> >> the code is post below.
> >>
> >> thanks
> >>
> >>
> >> def rigs_combx_change(attrname, old, new):
> >> rig, job = new, jobs_combx.value
> >> selected_rig = rig
> >> selected_job = job
> >>
> >> from_comboBx_group = False
> >> update_main_plot(selected_rig, selected_job, from_comboBx_group)
> >>
> >> def jobs_combx_change(attrname, old, new):
> >> rig, job = rigs_combx.value, new
> >> selected_rig = rig
> >> selected_job = job
> >> from_comboBx_group = False
> >> update_main_plot(selected_rig, selected_job, from_comboBx_group)
> >>
> >> update_main_plot_queue = queue.Queue()
> >> update_main_plot_event = threading.Event()
> >>
> >> update_main_plot_thread = Thread(name='update_main_plot_thread', \
> >> target = lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)), \
> >> args = (update_main_plot_queue, \
> >> doc, \
> >> update_main_plot_event, \
> >> mainplot_source, \
> >> main_plot, \
> >> mainplot_data_all, \
> >> checkbox_group_1_selections, \
> >> checkbox_group_2_selections,\
> >> checkbox_group_3_selections, \
> >> all_connection_dict,\
> >> rig, \
> >> job, \
> >> from_comboBx_group))
> >> update_main_plot_thread.start()
> >> update_main_plot_event.set()
> >>
> >>
> >>
> >>
> >> @gen.coroutine
> >> def update_main_plot(selected_rig, \
> >> selected_job, \
> >> from_comboBx_group, \
> >> checkbox_group_1_selections = ,\
> >> checkbox_group_2_selections = , \
> >> checkbox_group_3_selections = ):
> >>
> >> doc = curdoc()
> >> update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \
> >> update_main_plot_event, \
> >> mainplot_source, \
> >> main_plot, \
> >> mainplot_data_all, \
> >> checkbox_group_1_selections, \
> >> checkbox_group_2_selections,\
> >> checkbox_group_3_selections, \
> >> all_connection_dict,\
> >> selected_rig, \
> >> selected_job, \
> >> from_comboBx_group))
> >> update_main_plot_event.set()
> >>
> >>
> >> @without_document_lock
> >> def update_main_plot_chart( doc, \
> >> update_main_plot_event, \
> >> mainplot_source, \
> >> main_plot, \
> >> mainplot_data_all, \
> >> checkbox_group_1_selections, \
> >> checkbox_group_2_selections,\
> >> checkbox_group_3_selections, \
> >> all_connection_dict,\
> >> rig, \
> >> job, \
> >> from_comboBx_group):
> >> update_main_plot_event.wait()
> >>
> >> main_plot_dict = {}
> >> depth_list =
> >>
> >> if from_comboBx_group == True:
> >> main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
> >> rig, job, \
> >> checkbox_group_1_selections, \
> >> checkbox_group_2_selections)
> >> main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)
> >> else:
> >> main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
> >> rig, job)
> >> main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)
> >>
> >> depth_list = [str(x) for x in depth_list]
> >> doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot, main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))
> >>
> >>
> >> def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):
> >> new_list =
> >>
> >> in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])
> >> i = 0
> >> for item in in_all_data['HoleDepth']:
> >> if item != '-1':
> >> if i >= in_mainplot_data_length:
> >> if in_mainplot_data_type == 'VBarColors':
> >> new_list.append('white')
> >> else:
> >> new_list.append('')
> >> else:
> >> var = in_mainplot_data[in_mainplot_data_type][i]
> >> new_list.append(var)
> >> i = i + 1
> >> else:
> >> if in_mainplot_data_type == 'VBarColors':
> >> new_list.append('white')
> >> else:
> >> new_list.append('')
> >>
> >> return new_list
> >>
> >>
> >>
> >> @gen.coroutine
> >> def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):
> >> mainplot_source.data['HoleDepth'] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)
> >> mainplot_source.data['VBarTop'] = updateSourceData('VBarTop', main_plot_dict, mainplot_source.data)
> >> mainplot_source.data['VBarBottom'] = updateSourceData('VBarBottom', main_plot_dict, mainplot_source.data)
> >> mainplot_source.data['VBarColors'] = updateSourceData('VBarColors', main_plot_dict, mainplot_source.data)
> >> vBarType_list = updateSourceData('VBarType', main_plot_dict, mainplot_source.data)
> >> mainplot_source.data['VBarType'] = vBarType_list
> >> main_plot.x_range.factors =
> >> main_plot.x_range.factors = depth_list
> >>
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> >> To post to this group, send email to [email protected].
> >> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.
> >> For more options, visit https://groups.google.com/a/continuum.io/d/optout.
> >> <2018-08-17_8-13-18.png>
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> > To post to this group, send email to [email protected].
> > To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4F3A4C16-951A-4F5C-AEA2-17F93272F0A9%40googlemail.com.
> > For more options, visit https://groups.google.com/a/continuum.io/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To post to this group, send email to [email protected].
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/add030fa-5c70-4f66-9b05-0b3e23bbb1b7%40continuum.io.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout.
> <2018-08-20_16-39-43.png>

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/313347A2-B3E0-431C-82D4-E7C048FA371D%40anaconda.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.
--
Sent from Peng's Gmail Mobile

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/CANgoa1zwLic4qoj5onv4EmNiYZ%2B14jzA8xNDLs91rbX2F3qd5g%40mail.gmail.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Hi Buddy,

Thanks a lot, after completely check my code and fix different length of ColumnDataSource issue, and also server = Server({’/bk_plotter’: bk_plter.plot_doc}, io_loop=IOLoop(), allow_websocket_origin=[“172.20.101.16:8011”], websocket_max_message_size = 9999999999 * 1024 * 1024). the websocket lost issue is gone. I probably test my solution on IIS later and see if this issue is gone. as usual, I will update this thread, hopefully this thread can be successfully close and help other guys.

Cheers

···

On Monday, August 20, 2018 at 5:08:00 PM UTC-6, Bryan Van de ven wrote:

Hi,

Tornado has a max web socket size, and sending messages above that size will result in closed connections. In recent versions of Bokeh, this max value is user configurable with the --websocket-max-message-size command line option. But if it’s working behind Apache, and not IIS, then it sounds like some limitation of IIS in particular, than the tornado message size limit. Unfortunately I have no experience with IIS to offer any further suggestions.

Bryan

On Aug 20, 2018, at 15:49, peng wang [email protected] wrote:

new problem, when I update the columndatasource by method:
new _data = {‘foo’: new_foo_data, ‘bar’: new_bar_data }
source.data = new_data

other than:

 source.data['foo'] = new_foo_data
 source.data['bar'] = new_bar_data

<2018-08-20_16-39-43.png>

on my local pc, websocket is never lost, but it is lost when I deploy the web app on Apache. please refer to the following pic

When I just use less tables from database, for example, 5 tables, websocket is never lost, when I use more tables, for example, 10 to 15 tables, that is, more data, the websocket is immediately lost when I launch the web.

I came across the same issue when I deploy the web app on IIS, and then I use Apache, everything is fine. this is the first time that this issue appears again on Apache.

Thanks for any ideas.

On Monday, August 20, 2018 at 10:14:04 AM UTC-6, Bryan Van de ven wrote:

Hi,

There’s not much information here to go on. However, the plots you show are not big at all, drawing for sending data to them should not be any sort of problem.

I do notice you are using updating your column data source in a sub-optimal way. When you update individual columns one at a time, i.e:

source.data['foo'] = new_foo_data
source.data['bar'] = new_bar_data

That pattern causes a separate update to be sent over the network on every single line, causing separate re-draws for every line. It is always a better idea to try and update a CDS “all at once”, i.e.

    new _data = {'foo': new_foo_data, 'bar': new_bar_data }
    source.data = new_data

That will only result in a single update and single re-draw.

Additionally, I see you are appending to your columns? If that is the case you should try to arrange things so you can use the .stream method of the column data source. If you are only ever appending new data, then stream is much more efficient, it will only send the new data, instead of re-sending all the old data every time.

Finally are you sure your callback itself is not doing expensive blocking work?

Thanks,

Bryan

On Aug 17, 2018, at 10:16, ‘Corey O’Meara’ via Bokeh Discussion - Public [email protected] wrote:

Not sure if you load all the data at once but I believe using “patch” on a CDS is pretty fast.

You could only show some of the data and then on user interaction use a python bokeh callback to patch in and only add the new stuff.

Sent from my iPhone

On Aug 17, 2018, at 4:29 PM, peng wang [email protected] wrote:

Hi guys,

Currently, I have a project(Flask + Bokeh Server + SQL server on Apache). Dataset, which is read from database at one time and saved in memory, is pretty big(3G - 5G).
If I load web app, speed is good. If I select item from “Select” on the left bar and update plot associated with big dataset, the speed is so slow. I launch threads to improve the performance a little bit.
<2018-08-17_8-13-18.png>

but the problem is still there. I get an idea on this issue. If the ColumnDataSource and corresponding plot can be dynamically removed, I like to remove the old ones, and recreate new one when a small dataset is read from database every time

I did a lot research online, someone said that ColumnDataSource can not be removed dynamically. I also wonder if this issue is from Bokeh itself. that is, if dataset is big enough, the performance is automatically down.

I stuck on this issue in this entire week and can not do anything. welcome any ideas or clues.

the code is post below.

thanks

def rigs_combx_change(attrname, old, new):
rig, job = new, jobs_combx.value
selected_rig = rig
selected_job = job

    from_comboBx_group = False
    update_main_plot(selected_rig, selected_job, from_comboBx_group)      

def jobs_combx_change(attrname, old, new):
rig, job = rigs_combx.value, new
selected_rig = rig
selected_job = job
from_comboBx_group = False
update_main_plot(selected_rig, selected_job, from_comboBx_group)

update_main_plot_queue = queue.Queue()
update_main_plot_event = threading.Event()

update_main_plot_thread = Thread(name=‘update_main_plot_thread’,
target = lambda q, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12: q.put(all_main_plot.update_main_plot_chart(arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9, arg10, arg11, arg12)),
args = (update_main_plot_queue,
doc,
update_main_plot_event,
mainplot_source,
main_plot,
mainplot_data_all,
checkbox_group_1_selections,
checkbox_group_2_selections,
checkbox_group_3_selections,
all_connection_dict,
rig,
job,
from_comboBx_group))
update_main_plot_thread.start()
update_main_plot_event.set()

@gen.coroutine
def update_main_plot(selected_rig,
selected_job,
from_comboBx_group,
checkbox_group_1_selections = ,
checkbox_group_2_selections = ,
checkbox_group_3_selections = ):

    doc = curdoc()
    update_main_plot_queue.put(all_main_plot.update_main_plot_chart(doc, \
                                         update_main_plot_event, \
                                         mainplot_source, \
                                         main_plot, \
                                         mainplot_data_all, \
                                         checkbox_group_1_selections, \
                                         checkbox_group_2_selections,\
                                         checkbox_group_3_selections, \
                                         all_connection_dict,\
                                         selected_rig, \
                                         selected_job, \
                                         from_comboBx_group))
    update_main_plot_event.set()

@without_document_lock
def update_main_plot_chart( doc,
update_main_plot_event,
mainplot_source,
main_plot,
mainplot_data_all,
checkbox_group_1_selections,
checkbox_group_2_selections,
checkbox_group_3_selections,
all_connection_dict,
rig,
job,
from_comboBx_group):
update_main_plot_event.wait()

main_plot_dict = {}
depth_list = []

if from_comboBx_group == True:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job, \
                                                checkbox_group_1_selections, \
                                                checkbox_group_2_selections)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict, checkbox_group_3_selections)
else:
    main_plot_dict = update_checkBx_groups_dict(all_connection_dict, \
                                                rig, job)
    main_plot_dict, depth_list = update_main_plot_dict(main_plot_dict)
   
depth_list = [str(x) for x in depth_list]
doc.add_next_tick_callback(partial(update_main_plot_source, main_plot = main_plot,  main_plot_dict = main_plot_dict, depth_list = depth_list, mainplot_data_all = mainplot_data_all, mainplot_source = mainplot_source))

def updateSourceData(in_mainplot_data_type, in_mainplot_data, in_all_data):
new_list =

in_mainplot_data_length = len(in_mainplot_data[in_mainplot_data_type])
i = 0
for item in in_all_data['HoleDepth']:
    if item != '-1':
        if i >= in_mainplot_data_length:
            if in_mainplot_data_type == 'VBarColors':
                new_list.append('white')
            else:
                new_list.append('')
        else:  
            var = in_mainplot_data[in_mainplot_data_type][i]
            new_list.append(var)
            i = i + 1
    else:
        if in_mainplot_data_type == 'VBarColors':
            new_list.append('white')
        else:
            new_list.append('')

return new_list

@gen.coroutine
def update_main_plot_source(main_plot, main_plot_dict, depth_list, mainplot_data_all, mainplot_source):
mainplot_source.data[‘HoleDepth’] = update_holeDepth_list(main_plot_dict, mainplot_data_all, depth_list)
mainplot_source.data[‘VBarTop’] = updateSourceData(‘VBarTop’, main_plot_dict, mainplot_source.data)
mainplot_source.data[‘VBarBottom’] = updateSourceData(‘VBarBottom’, main_plot_dict, mainplot_source.data)
mainplot_source.data[‘VBarColors’] = updateSourceData(‘VBarColors’, main_plot_dict, mainplot_source.data)
vBarType_list = updateSourceData(‘VBarType’, main_plot_dict, mainplot_source.data)
mainplot_source.data[‘VBarType’] = vBarType_list
main_plot.x_range.factors =
main_plot.x_range.factors = depth_list


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/2e97c94e-c5cd-4d1e-a370-ad9c0519c2f3%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.
<2018-08-17_8-13-18.png>


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4F3A4C16-951A-4F5C-AEA2-17F93272F0A9%40googlemail.com.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/add030fa-5c70-4f66-9b05-0b3e23bbb1b7%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

<2018-08-20_16-39-43.png>