Testing/Verification with bokeh

I have a question related to testing with bokeh. What is the best way to test and verify that the plots produced are correct. Would this be done at a json level? I saw on the documentation that there is the bokeh json myapp.py command, but that doesn’t seem practical to use with unit testing. Smaller examples may be possible to check visually, by hand, but how would it work for larger examples.

In something like the gapminder example, with many data points changing with the slider, how can you verify that the plot is correct? Would you just verify that the json produced is correct or assert that for a certain input, bokeh produces the correct json output? In general, I’m trying to understand how to apply unit testing to a bokeh project. Are there any good examples of testing things like this?

Thanks!

Hi Stephanie,

In general I would say that unit-testing Bokeh itself (i.e. that input XYZ results in a correct output plot of XYZ) is the core Bokeh dev team's job, not yours. That is, in general, consumers of libraries should test their code and their usage of library code, not the library code itself.

So I think what you really want (and maybe what you mean -- I can't tell) is to test that you are providing the correct inputs to Bokeh functions, for your situation. One option might be to use the Mock library, to ensure that you are calling bokeh functions with the values you mean to. Alternatively, if you have "helper" functions that wrap lots of common Bokeh code up, you could test that you are giving the correct inputs at that level. At a finer grained level you could actually create a Bokeh plot object in memory, then inspect and compare things like selected model property values or column data source data dictionaries.

I think trying to do visual/image comparisons would be: very difficult, very fragile, not useful. If you really wanted to go that route, perhaps selenium could be useful as a way to "screen cap" a broswer and compare to a reference image. I'd still discourage this notion. We do that kind of testing to test the library itself, but you should be testing your *usage* of the library. That is, that you are calling Bokeh functions appropriately. Or put another way, your tests should be as close as possible to the data you want to feed to Bokeh. The final rendered images are "far" from the expected inputs, in an "ease of testing" sense.

Bryan

···

On Oct 12, 2016, at 5:07 PM, [email protected] wrote:

I have a question related to testing with bokeh. What is the best way to test and verify that the plots produced are correct. Would this be done at a json level? I saw on the documentation that there is the bokeh json myapp.py command, but that doesn't seem practical to use with unit testing. Smaller examples may be possible to check visually, by hand, but how would it work for larger examples.

In something like the gapminder example, with many data points changing with the slider, how can you verify that the plot is correct? Would you just verify that the json produced is correct or assert that for a certain input, bokeh produces the correct json output? In general, I'm trying to understand how to apply unit testing to a bokeh project. Are there any good examples of testing things like this?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/e7fea9c7-2169-434b-b674-1c186eca3bf8%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

Bryan,

Is it reasonable or possible to compare the JSON that is associated or produced for each figure or plot? I agree that one should test for the correct inputs but instead of ensuring that the rendered plot is the same, maybe one could just ensure that the JSON is the same. Would love to hear your thoughts around this.

Best,

Sean

···

On Wednesday, October 12, 2016 at 5:30:05 PM UTC-4, Bryan Van de ven wrote:

Hi Stephanie,

In general I would say that unit-testing Bokeh itself (i.e. that input XYZ results in a correct output plot of XYZ) is the core Bokeh dev team’s job, not yours. That is, in general, consumers of libraries should test their code and their usage of library code, not the library code itself.

So I think what you really want (and maybe what you mean – I can’t tell) is to test that you are providing the correct inputs to Bokeh functions, for your situation. One option might be to use the Mock library, to ensure that you are calling bokeh functions with the values you mean to. Alternatively, if you have “helper” functions that wrap lots of common Bokeh code up, you could test that you are giving the correct inputs at that level. At a finer grained level you could actually create a Bokeh plot object in memory, then inspect and compare things like selected model property values or column data source data dictionaries.

I think trying to do visual/image comparisons would be: very difficult, very fragile, not useful. If you really wanted to go that route, perhaps selenium could be useful as a way to “screen cap” a broswer and compare to a reference image. I’d still discourage this notion. We do that kind of testing to test the library itself, but you should be testing your usage of the library. That is, that you are calling Bokeh functions appropriately. Or put another way, your tests should be as close as possible to the data you want to feed to Bokeh. The final rendered images are “far” from the expected inputs, in an “ease of testing” sense.

Bryan

On Oct 12, 2016, at 5:07 PM, [email protected] wrote:

I have a question related to testing with bokeh. What is the best way to test and verify that the plots produced are correct. Would this be done at a json level? I saw on the documentation that there is the bokeh json myapp.py command, but that doesn’t seem practical to use with unit testing. Smaller examples may be possible to check visually, by hand, but how would it work for larger examples.

In something like the gapminder example, with many data points changing with the slider, how can you verify that the plot is correct? Would you just verify that the json produced is correct or assert that for a certain input, bokeh produces the correct json output? In general, I’m trying to understand how to apply unit testing to a bokeh project. Are there any good examples of testing things like this?

Thanks!


You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/e7fea9c7-2169-434b-b674-1c186eca3bf8%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

I would not jump to this approach first, personally. It seems like a pain but it's also fragile. At a minimum, you'd have to take care to order the JSON in a consistent way. But even then, what if new version of Bokeh introduce new properties in a backwards compatible way? Then your JSON comparison breaks even though the visual output is the same. I still maintain the best bests users can do are to make sure they are calling Bokeh as they intend to call Bokeh. This could either be done via mocking, or by creating your own thin wrapper layer of functions that accept inputs and spit out Bokeh objects in a way customized to the use case. Those could be black-box tested directly.

Thanks,

Bryan

···

On Oct 13, 2016, at 12:19 PM, Sean Law <[email protected]> wrote:

Bryan,

Is it reasonable or possible to compare the JSON that is associated or produced for each figure or plot? I agree that one should test for the correct inputs but instead of ensuring that the rendered plot is the same, maybe one could just ensure that the JSON is the same. Would love to hear your thoughts around this.

Best,
Sean

On Wednesday, October 12, 2016 at 5:30:05 PM UTC-4, Bryan Van de ven wrote:
Hi Stephanie,

In general I would say that unit-testing Bokeh itself (i.e. that input XYZ results in a correct output plot of XYZ) is the core Bokeh dev team's job, not yours. That is, in general, consumers of libraries should test their code and their usage of library code, not the library code itself.

So I think what you really want (and maybe what you mean -- I can't tell) is to test that you are providing the correct inputs to Bokeh functions, for your situation. One option might be to use the Mock library, to ensure that you are calling bokeh functions with the values you mean to. Alternatively, if you have "helper" functions that wrap lots of common Bokeh code up, you could test that you are giving the correct inputs at that level. At a finer grained level you could actually create a Bokeh plot object in memory, then inspect and compare things like selected model property values or column data source data dictionaries.

I think trying to do visual/image comparisons would be: very difficult, very fragile, not useful. If you really wanted to go that route, perhaps selenium could be useful as a way to "screen cap" a broswer and compare to a reference image. I'd still discourage this notion. We do that kind of testing to test the library itself, but you should be testing your *usage* of the library. That is, that you are calling Bokeh functions appropriately. Or put another way, your tests should be as close as possible to the data you want to feed to Bokeh. The final rendered images are "far" from the expected inputs, in an "ease of testing" sense.

Bryan

> On Oct 12, 2016, at 5:07 PM, matsui.s...@gmail.com wrote:
>
> I have a question related to testing with bokeh. What is the best way to test and verify that the plots produced are correct. Would this be done at a json level? I saw on the documentation that there is the bokeh json myapp.py command, but that doesn't seem practical to use with unit testing. Smaller examples may be possible to check visually, by hand, but how would it work for larger examples.
>
> In something like the gapminder example, with many data points changing with the slider, how can you verify that the plot is correct? Would you just verify that the json produced is correct or assert that for a certain input, bokeh produces the correct json output? In general, I'm trying to understand how to apply unit testing to a bokeh project. Are there any good examples of testing things like this?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
> To post to this group, send email to bo...@continuum.io.
> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/e7fea9c7-2169-434b-b674-1c186eca3bf8%40continuum.io\.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/063ef9c7-3b4a-4336-83e1-91af939b15b6%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.