Streaming with NaN values

I upgraded Bokeh with 0.12.4 and I think the workaround with streaming NaN values no longer works.

I am patching my DataFrame with ‘NaN’ string before turning it to dict and feeding to ColumnDataSource.stream :
stream(df.fillna(‘NaN’).to_dict(orient=“list”))

``

I have several line charts in gridplot and the chart with NaNs is not displaying any line. (No errors produced as well). Stream update contains a single number with NaN, I am willing to connect a line between available numbers.

Any ideas?

Thanks

BTW: Is there anything I should do to enable binary protocol instead of JSON?

Hi,

NumPy arrays (and thus also Pandas Series that are Float32, Float64, or Int32 dtype are automatically serialized using the binary protocol. This also no includes nans and infs. There is no need for any workaround when starting form those array types.

I can't quite tell from your message exactly what you are doing, so I can't elaborate any further. Maybe you can provide more code for more context.

Bryan

···

On Feb 13, 2017, at 14:04, Meir Tseitlin <[email protected]> wrote:

I upgraded Bokeh with 0.12.4 and I think the workaround with streaming NaN values no longer works.

I am patching my DataFrame with 'NaN' string before turning it to dict and feeding to ColumnDataSource.stream :
stream(df.fillna('NaN').to_dict(orient="list"))

I have several line charts in gridplot and the chart with NaNs is not displaying any line. (No errors produced as well). Stream update contains a single number with NaN, I am willing to connect a line between available numbers.

Any ideas?

Thanks

BTW: Is there anything I should do to enable binary protocol instead of JSON?

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/772d38de-e3cb-461b-93bc-2a28861b03aa%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

Thank you Bryan,

  • I am creating several figures with time-series data (each figure has a single line), I provide a single ColumnDataSource with the data to all figures (each ‘y’ has it’s own name and ‘x’=‘time’).
  • I am running Bokeh server with add_periodic_callback, streaming additional data to ColumnDataSource
  • The figure containing NaNs does not draw any line.
  • My types are float64, but if I am not doing fillna(‘NaN’), I receive: “Out of range float values are not JSON compliant: nan”

Example of dict I am feeding to stream:
{

u’var9’: [

122.876,

122.876,

122.88,

122.889,

122.885,

122.884,

122.885,

122.887,

122.875

],

u’var40’: [

‘NaN’,

‘NaN’,

53.7135,

‘NaN’,

‘NaN’,

‘NaN’,

‘NaN’,

‘NaN’,

‘NaN’

],

u’var17’: [

22.9009,

23.0699,

22.6906,

22.8626,

22.8737,

22.6161,

23.0456,

22.9609,

22.9154

],

u’var7’: [

2799.43,

2799.77,

2799.13,

2799.05,

2799.37,

2799.4,

2798.81,

2799.76,

2799.08

],

‘time’: [

Timestamp(‘2017-02-13 20:32:35.263716’),

Timestamp(‘2017-02-13 20:32:35.671465’),

Timestamp(‘2017-02-13 20:32:36.079146’),

Timestamp(‘2017-02-13 20:32:36.489926’),

Timestamp(‘2017-02-13 20:32:36.896583’),

Timestamp(‘2017-02-13 20:32:37.302591’),

Timestamp(‘2017-02-13 20:32:37.708200’),

Timestamp(‘2017-02-13 20:32:38.112995’),

Timestamp(‘2017-02-13 20:32:38.520433’)

]

}

···

On Monday, February 13, 2017 at 10:18:54 PM UTC+2, Bryan Van de ven wrote:

Hi,

NumPy arrays (and thus also Pandas Series that are Float32, Float64, or Int32 dtype are automatically serialized using the binary protocol. This also no includes nans and infs. There is no need for any workaround when starting form those array types.

I can’t quite tell from your message exactly what you are doing, so I can’t elaborate any further. Maybe you can provide more code for more context.

Bryan

On Feb 13, 2017, at 14:04, Meir Tseitlin [email protected] wrote:

I upgraded Bokeh with 0.12.4 and I think the workaround with streaming NaN values no longer works.

I am patching my DataFrame with ‘NaN’ string before turning it to dict and feeding to ColumnDataSource.stream :

stream(df.fillna(‘NaN’).to_dict(orient=“list”))

I have several line charts in gridplot and the chart with NaNs is not displaying any line. (No errors produced as well). Stream update contains a single number with NaN, I am willing to connect a line between available numbers.

Any ideas?

Thanks

BTW: Is there anything I should do to enable binary protocol instead of JSON?


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/772d38de-e3cb-461b-93bc-2a28861b03aa%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

The columns in your dict are python lists, not arrays. You need to construct actual NumPy arrays for the columns in order for Bokeh to be able to use the binary array protocol.

Thanks,

Bryan

···

On Feb 13, 2017, at 14:36, Meir Tseitlin <[email protected]> wrote:

Thank you Bryan,

  • I am creating several figures with time-series data (each figure has a single line), I provide a single ColumnDataSource with the data to all figures (each 'y' has it's own name and 'x'='time').
  • I am running Bokeh server with add_periodic_callback, streaming additional data to ColumnDataSource
  • The figure containing NaNs does not draw any line.
  • My types are float64, but if I am not doing fillna('NaN'), I receive: "Out of range float values are not JSON compliant: nan"

Example of dict I am feeding to stream:
{
  u'var9': [
    122.876,
    122.876,
    122.88,
    122.889,
    122.885,
    122.884,
    122.885,
    122.887,
    122.875
  ],
  u'var40': [
    'NaN',
    'NaN',
    53.7135,
    'NaN',
    'NaN',
    'NaN',
    'NaN',
    'NaN',
    'NaN'
  ],
  u'var17': [
    22.9009,
    23.0699,
    22.6906,
    22.8626,
    22.8737,
    22.6161,
    23.0456,
    22.9609,
    22.9154
  ],
  u'var7': [
    2799.43,
    2799.77,
    2799.13,
    2799.05,
    2799.37,
    2799.4,
    2798.81,
    2799.76,
    2799.08
  ],
  'time': [
    Timestamp('2017-02-13 20:32:35.263716'),
    Timestamp('2017-02-13 20:32:35.671465'),
    Timestamp('2017-02-13 20:32:36.079146'),
    Timestamp('2017-02-13 20:32:36.489926'),
    Timestamp('2017-02-13 20:32:36.896583'),
    Timestamp('2017-02-13 20:32:37.302591'),
    Timestamp('2017-02-13 20:32:37.708200'),
    Timestamp('2017-02-13 20:32:38.112995'),
    Timestamp('2017-02-13 20:32:38.520433')
  ]
}

On Monday, February 13, 2017 at 10:18:54 PM UTC+2, Bryan Van de ven wrote:
Hi,

NumPy arrays (and thus also Pandas Series that are Float32, Float64, or Int32 dtype are automatically serialized using the binary protocol. This also no includes nans and infs. There is no need for any workaround when starting form those array types.

I can't quite tell from your message exactly what you are doing, so I can't elaborate any further. Maybe you can provide more code for more context.

Bryan

> On Feb 13, 2017, at 14:04, Meir Tseitlin <[email protected]> wrote:
>
> I upgraded Bokeh with 0.12.4 and I think the workaround with streaming NaN values no longer works.
>
> I am patching my DataFrame with 'NaN' string before turning it to dict and feeding to ColumnDataSource.stream :
> stream(df.fillna('NaN').to_dict(orient="list"))
>
> I have several line charts in gridplot and the chart with NaNs is not displaying any line. (No errors produced as well). Stream update contains a single number with NaN, I am willing to connect a line between available numbers.
>
> Any ideas?
>
> Thanks
>
> BTW: Is there anything I should do to enable binary protocol instead of JSON?
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
> To post to this group, send email to bo...@continuum.io.
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/772d38de-e3cb-461b-93bc-2a28861b03aa%40continuum.io\.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/6ed629f8-38de-4258-a91c-ccd7d101ffc5%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

Hi Bryan,

I am still struggling with this issue.

Can you provide an example of how to stream data (ColumnDataSource.stream) with numpy arrays?

Thanks

···

On Tuesday, February 14, 2017 at 12:49:33 AM UTC+2, Bryan Van de ven wrote:

The columns in your dict are python lists, not arrays. You need to construct actual NumPy arrays for the columns in order for Bokeh to be able to use the binary array protocol.

Thanks,

Bryan

On Feb 13, 2017, at 14:36, Meir Tseitlin [email protected] wrote:

Thank you Bryan,

    • I am creating several figures with time-series data (each figure has a single line), I provide a single ColumnDataSource with the data to all figures (each 'y' has it's own name and 'x'='time').
    • I am running Bokeh server with add_periodic_callback, streaming additional data to  ColumnDataSource
    • The figure containing NaNs does not draw any line.
    • My types are float64, but if I am not doing fillna('NaN'), I receive: "Out of range float values are not JSON compliant: nan"

Example of dict I am feeding to stream:

{

u’var9’: [

122.876,
122.876,
122.88,
122.889,
122.885,
122.884,
122.885,
122.887,
122.875

],

u’var40’: [

'NaN',
'NaN',
53.7135,
'NaN',
'NaN',
'NaN',
'NaN',
'NaN',
'NaN'

],

u’var17’: [

22.9009,
23.0699,
22.6906,
22.8626,
22.8737,
22.6161,
23.0456,
22.9609,
22.9154

],

u’var7’: [

2799.43,
2799.77,
2799.13,
2799.05,
2799.37,
2799.4,
2798.81,
2799.76,
2799.08

],

‘time’: [

Timestamp('2017-02-13 20:32:35.263716'),
Timestamp('2017-02-13 20:32:35.671465'),
Timestamp('2017-02-13 20:32:36.079146'),
Timestamp('2017-02-13 20:32:36.489926'),
Timestamp('2017-02-13 20:32:36.896583'),
Timestamp('2017-02-13 20:32:37.302591'),
Timestamp('2017-02-13 20:32:37.708200'),
Timestamp('2017-02-13 20:32:38.112995'),
Timestamp('2017-02-13 20:32:38.520433')

]

}

On Monday, February 13, 2017 at 10:18:54 PM UTC+2, Bryan Van de ven wrote:

Hi,

NumPy arrays (and thus also Pandas Series that are Float32, Float64, or Int32 dtype are automatically serialized using the binary protocol. This also no includes nans and infs. There is no need for any workaround when starting form those array types.

I can’t quite tell from your message exactly what you are doing, so I can’t elaborate any further. Maybe you can provide more code for more context.

Bryan

On Feb 13, 2017, at 14:04, Meir Tseitlin [email protected] wrote:

I upgraded Bokeh with 0.12.4 and I think the workaround with streaming NaN values no longer works.

I am patching my DataFrame with ‘NaN’ string before turning it to dict and feeding to ColumnDataSource.stream :
stream(df.fillna(‘NaN’).to_dict(orient=“list”))

I have several line charts in gridplot and the chart with NaNs is not displaying any line. (No errors produced as well). Stream update contains a single number with NaN, I am willing to connect a line between available numbers.

Any ideas?

Thanks

BTW: Is there anything I should do to enable binary protocol instead of JSON?


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/772d38de-e3cb-461b-93bc-2a28861b03aa%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/6ed629f8-38de-4258-a91c-ccd7d101ffc5%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

I think I figured out how to send it as numpy arrays (just converted value arrays to numpy) within the dictionary (leaving time values as standard array).

Yet the effect is very similar - when one of numpy arrays contain nan - the whole update is not displayed.

···

On Wednesday, April 12, 2017 at 1:13:30 PM UTC+3, Meir Tseitlin wrote:

Hi Bryan,

I am still struggling with this issue.

Can you provide an example of how to stream data (ColumnDataSource.stream) with numpy arrays?

Thanks

On Tuesday, February 14, 2017 at 12:49:33 AM UTC+2, Bryan Van de ven wrote:

The columns in your dict are python lists, not arrays. You need to construct actual NumPy arrays for the columns in order for Bokeh to be able to use the binary array protocol.

Thanks,

Bryan

On Feb 13, 2017, at 14:36, Meir Tseitlin [email protected] wrote:

Thank you Bryan,

    • I am creating several figures with time-series data (each figure has a single line), I provide a single ColumnDataSource with the data to all figures (each 'y' has it's own name and 'x'='time').
    • I am running Bokeh server with add_periodic_callback, streaming additional data to  ColumnDataSource
    • The figure containing NaNs does not draw any line.
    • My types are float64, but if I am not doing fillna('NaN'), I receive: "Out of range float values are not JSON compliant: nan"

Example of dict I am feeding to stream:

{

u’var9’: [

122.876,
122.876,
122.88,
122.889,
122.885,
122.884,
122.885,
122.887,
122.875

],

u’var40’: [

'NaN',
'NaN',
53.7135,
'NaN',
'NaN',
'NaN',
'NaN',
'NaN',
'NaN'

],

u’var17’: [

22.9009,
23.0699,
22.6906,
22.8626,
22.8737,
22.6161,
23.0456,
22.9609,
22.9154

],

u’var7’: [

2799.43,
2799.77,
2799.13,
2799.05,
2799.37,
2799.4,
2798.81,
2799.76,
2799.08

],

‘time’: [

Timestamp('2017-02-13 20:32:35.263716'),
Timestamp('2017-02-13 20:32:35.671465'),
Timestamp('2017-02-13 20:32:36.079146'),
Timestamp('2017-02-13 20:32:36.489926'),
Timestamp('2017-02-13 20:32:36.896583'),
Timestamp('2017-02-13 20:32:37.302591'),
Timestamp('2017-02-13 20:32:37.708200'),
Timestamp('2017-02-13 20:32:38.112995'),
Timestamp('2017-02-13 20:32:38.520433')

]

}

On Monday, February 13, 2017 at 10:18:54 PM UTC+2, Bryan Van de ven wrote:

Hi,

NumPy arrays (and thus also Pandas Series that are Float32, Float64, or Int32 dtype are automatically serialized using the binary protocol. This also no includes nans and infs. There is no need for any workaround when starting form those array types.

I can’t quite tell from your message exactly what you are doing, so I can’t elaborate any further. Maybe you can provide more code for more context.

Bryan

On Feb 13, 2017, at 14:04, Meir Tseitlin [email protected] wrote:

I upgraded Bokeh with 0.12.4 and I think the workaround with streaming NaN values no longer works.

I am patching my DataFrame with ‘NaN’ string before turning it to dict and feeding to ColumnDataSource.stream :
stream(df.fillna(‘NaN’).to_dict(orient=“list”))

I have several line charts in gridplot and the chart with NaNs is not displaying any line. (No errors produced as well). Stream update contains a single number with NaN, I am willing to connect a line between available numbers.

Any ideas?

Thanks

BTW: Is there anything I should do to enable binary protocol instead of JSON?


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/772d38de-e3cb-461b-93bc-2a28861b03aa%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/6ed629f8-38de-4258-a91c-ccd7d101ffc5%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Yes, this is expected and also beyond our control. There is no support in the JSON specification for "nan" values. So if you use lists (which get converted to JSON) then things will not work. However if you use numpy arrays then they get encoded in a different way that allows for nan's to be communicated properly. If you need to stream nan's then you need to put the data in a numpy array.

Thanks,

Bryan

···

On Apr 12, 2017, at 06:50, Meir Tseitlin <[email protected]> wrote:

I think I figured out how to send it as numpy arrays (just converted value arrays to numpy) within the dictionary (leaving time values as standard array).

Yet the effect is very similar - when one of numpy arrays contain nan - the whole update is not displayed.

On Wednesday, April 12, 2017 at 1:13:30 PM UTC+3, Meir Tseitlin wrote:
Hi Bryan,

I am still struggling with this issue.

Can you provide an example of how to stream data (ColumnDataSource.stream) with numpy arrays?

Thanks

On Tuesday, February 14, 2017 at 12:49:33 AM UTC+2, Bryan Van de ven wrote:
The columns in your dict are python lists, not arrays. You need to construct actual NumPy arrays for the columns in order for Bokeh to be able to use the binary array protocol.

Thanks,

Bryan

> On Feb 13, 2017, at 14:36, Meir Tseitlin <[email protected]> wrote:
>
> Thank you Bryan,
>
> • I am creating several figures with time-series data (each figure has a single line), I provide a single ColumnDataSource with the data to all figures (each 'y' has it's own name and 'x'='time').
> • I am running Bokeh server with add_periodic_callback, streaming additional data to ColumnDataSource
> • The figure containing NaNs does not draw any line.
> • My types are float64, but if I am not doing fillna('NaN'), I receive: "Out of range float values are not JSON compliant: nan"
>
> Example of dict I am feeding to stream:
> {
> u'var9': [
> 122.876,
> 122.876,
> 122.88,
> 122.889,
> 122.885,
> 122.884,
> 122.885,
> 122.887,
> 122.875
> ],
> u'var40': [
> 'NaN',
> 'NaN',
> 53.7135,
> 'NaN',
> 'NaN',
> 'NaN',
> 'NaN',
> 'NaN',
> 'NaN'
> ],
> u'var17': [
> 22.9009,
> 23.0699,
> 22.6906,
> 22.8626,
> 22.8737,
> 22.6161,
> 23.0456,
> 22.9609,
> 22.9154
> ],
> u'var7': [
> 2799.43,
> 2799.77,
> 2799.13,
> 2799.05,
> 2799.37,
> 2799.4,
> 2798.81,
> 2799.76,
> 2799.08
> ],
> 'time': [
> Timestamp('2017-02-13 20:32:35.263716'),
> Timestamp('2017-02-13 20:32:35.671465'),
> Timestamp('2017-02-13 20:32:36.079146'),
> Timestamp('2017-02-13 20:32:36.489926'),
> Timestamp('2017-02-13 20:32:36.896583'),
> Timestamp('2017-02-13 20:32:37.302591'),
> Timestamp('2017-02-13 20:32:37.708200'),
> Timestamp('2017-02-13 20:32:38.112995'),
> Timestamp('2017-02-13 20:32:38.520433')
> ]
> }
>
>
> On Monday, February 13, 2017 at 10:18:54 PM UTC+2, Bryan Van de ven wrote:
> Hi,
>
> NumPy arrays (and thus also Pandas Series that are Float32, Float64, or Int32 dtype are automatically serialized using the binary protocol. This also no includes nans and infs. There is no need for any workaround when starting form those array types.
>
> I can't quite tell from your message exactly what you are doing, so I can't elaborate any further. Maybe you can provide more code for more context.
>
> Bryan
>
> > On Feb 13, 2017, at 14:04, Meir Tseitlin <[email protected]> wrote:
> >
> > I upgraded Bokeh with 0.12.4 and I think the workaround with streaming NaN values no longer works.
> >
> > I am patching my DataFrame with 'NaN' string before turning it to dict and feeding to ColumnDataSource.stream :
> > stream(df.fillna('NaN').to_dict(orient="list"))
> >
> > I have several line charts in gridplot and the chart with NaNs is not displaying any line. (No errors produced as well). Stream update contains a single number with NaN, I am willing to connect a line between available numbers.
> >
> > Any ideas?
> >
> > Thanks
> >
> > BTW: Is there anything I should do to enable binary protocol instead of JSON?
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
> > To post to this group, send email to bo...@continuum.io.
> > To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/772d38de-e3cb-461b-93bc-2a28861b03aa%40continuum.io\.
> > For more options, visit https://groups.google.com/a/continuum.io/d/optout\.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
> To post to this group, send email to bo...@continuum.io.
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/6ed629f8-38de-4258-a91c-ccd7d101ffc5%40continuum.io\.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/0a2008db-0046-46d1-943b-3644576fd36f%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.