I’m basically building an UI for a simulator. So the high level overview is that you set up a simulation model, and then when you run the simulation you get a stream of data points with some axis (time or frequency usually) and columns for each node in the model. For some of these simulations the data points are complex numbers.
When I try to do anything with complex numbers I get errors that they can’t be encoded as JSON.
That makes sense. And you can’t plot raw complex numbers anyway.
But that leaves more of a design question, where do you do data processing in Bokeh? For processing you definitely want to work on complex numbers, not on separate real/imaginary or magnitude/phase components.
This ties into another problem I have with ColumnDataSources is that they don’t really have an index or data processing functionality like a Pandas dataframe. So it seems like a ColumnDataSource is more of a display structure than a data processing one, correct?
So now I want to stream data from my simulator. How would I set that up so that it’s streamed efficiently but allows data processing? At least I need to be able to process complex numbers to save them as separate components, but I want to be able to do other processing on the data.
Yes that is completely correct, a Bokeh CDS is, more than anything else, a structure for supporting the serialization of columnar data between the Python and JS runtimes. It’s not really concerned at all with any compute aspects at all. 
So presently, I would say you would need to decompose to Re and Im columns manually on the Python side, and then deal with them on the JS side by looping over the columns to combine the Re and Im parts (maybe utilizing a third-party JS lib like math.js that has a complex number object).
There are a few built-in transforms for simple things like stacking or cumulative sums/prods of columns, as well as a generic CustomJSTransform that users can define. ↩︎
Thanks. Yea I guess as a data visualization framework there isn’t much point in dealing with complex numbers, because you’ll always end up plotting either magnitude and phase or real and imaginary parts.
So I’m thinking I need to develop my own processing layer that acts on chunks of DataFrames that get appended to a ColumnDataStore for plotting. I’ll have to think about what works nice with streaming and Pandas. It will probably be somewhat map-reduce shaped. But that’s quite a big design project so for now I’ll probably just do something simple.