Working with a streaming ColumnDataSource with complex numbers

I’m basically building an UI for a simulator. So the high level overview is that you set up a simulation model, and then when you run the simulation you get a stream of data points with some axis (time or frequency usually) and columns for each node in the model. For some of these simulations the data points are complex numbers.

When I try to do anything with complex numbers I get errors that they can’t be encoded as JSON.

That makes sense. And you can’t plot raw complex numbers anyway.

But that leaves more of a design question, where do you do data processing in Bokeh? For processing you definitely want to work on complex numbers, not on separate real/imaginary or magnitude/phase components.

This ties into another problem I have with ColumnDataSources is that they don’t really have an index or data processing functionality like a Pandas dataframe. So it seems like a ColumnDataSource is more of a display structure than a data processing one, correct?

So now I want to stream data from my simulator. How would I set that up so that it’s streamed efficiently but allows data processing? At least I need to be able to process complex numbers to save them as separate components, but I want to be able to do other processing on the data.

Yes that is completely correct, a Bokeh CDS is, more than anything else, a structure for supporting the serialization of columnar data between the Python and JS runtimes. It’s not really concerned at all with any compute aspects at all. ^[1]

As for complex numbers, as you note they are not supported in JSON (and there is not built-in complex type in JavaScript at all). The best we could do is to automatically serialized columns of complex numbers into two columns of Re and Im parts. That would work fine Python → JS but we would also have to consider the JS → Python direction and that seems much more complicated and novel. There are no other instances of “mapping” multiple columns in one runtime to a single column in another automagically, and adding support for that would require new and non-trivial development on some of the lowest-level layers of Bokeh. There’s been almost no other requests I can recall in the last decade for complex number support, so it’s just not clear to me that kind of effort would be justified.

So presently, I would say you would need to decompose to Re and Im columns manually on the Python side, and then deal with them on the JS side by looping over the columns to combine the Re and Im parts (maybe utilizing a third-party JS lib like math.js that has a complex number object).

1. There are a few built-in transforms for simple things like stacking or cumulative sums/prods of columns, as well as a generic CustomJSTransform that users can define. ↩︎

Thanks. Yea I guess as a data visualization framework there isn’t much point in dealing with complex numbers, because you’ll always end up plotting either magnitude and phase or real and imaginary parts.

So I’m thinking I need to develop my own processing layer that acts on chunks of DataFrames that get appended to a ColumnDataStore for plotting. I’ll have to think about what works nice with streaming and Pandas. It will probably be somewhat map-reduce shaped. But that’s quite a big design project so for now I’ll probably just do something simple.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.