Using pytest to verify outputs in bokeh front-end

Community Support
1

Objective:

I want to use pytest to verify numerical outputs in my bokeh front-end given specific numerical inputs. This testing is to complement my functional / unit tests on the function(s) relating inputs to outputs, so to ensure the GUI is showing the correct numbers.

Work Done:

I setup a test module where my bokeh app gets launched as a subprocess, and then a test is run to verify the output for a given input. The input is set through the value attribute of a bokeh.model.NumericInput, taken from the Document object obtained through a call to bokeh.client.session.pull_session. Per the bokeh documentation, my understanding is that in the context of pull_session I can access the bokeh.model.NumericInput through the session.document and changes made to the value of bokeh.model.NumericInput on the client-side document will be synced to the server-side document.

Outcome:

My test shows that the NumericInput model value for the input is set as expected, however the NumericInput value for the output is unchanged. The bokeh application invokes on_change() methods for the values of input models such when the values change, the output model value updates through a callback. It seems that changes to model values in the session.document are not being detected as such on the server-side.

Output:

(base) c:\example>conda activate example_env

(example_env) c:\example>pytest test_gui.py
================================================= test session starts =================================================
platform win32 -- Python 3.10.13, pytest-7.4.0, pluggy-1.0.0
rootdir: c:\example
collected 1 item

test_gui.py F                                                                                                    [100%]

====================================================== FAILURES =======================================================
____________________________________________________ TestUI.test_z ____________________________________________________

self = <test_gui.TestUI object at 0x00000251DC3F0610>

    def test_z(self):
>       change_x(set_value=3, test_value=4)

test_gui.py:36:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

set_value = 3, test_value = 4

    def change_x(set_value, test_value):
        with pull_session(url=app_url) as client_session:
            models = {m.title: m for m in client_session.document.models if hasattr(m, "title")}
            models["x"].update(value=set_value)
            client_session.push()
            assert models["x"].value == set_value
>           assert models["z"].value == test_value
E           AssertionError: assert 3 == 4
E            +  where 3 = NumericInput(id='1004', ...).value

test_gui.py:23: AssertionError
------------------------------------------------ Captured stderr call -------------------------------------------------
2023-10-20 10:05:51,053 Starting Bokeh server version 2.4.3 (running on Tornado 6.3.3)
2023-10-20 10:05:51,061 User authentication hooks NOT provided (default user enabled)
2023-10-20 10:05:51,061 These host origins can connect to the websocket: ['localhost:5006']
2023-10-20 10:05:51,062 Patterns are:
2023-10-20 10:05:51,063   [('/example/?',
2023-10-20 10:05:51,063     <class 'bokeh.server.views.doc_handler.DocHandler'>,
2023-10-20 10:05:51,064     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000019B5C4D3AC0>,
2023-10-20 10:05:51,064      'bokeh_websocket_path': '/example/ws'}),
2023-10-20 10:05:51,066    ('/example/ws',
2023-10-20 10:05:51,066     <class 'bokeh.server.views.ws.WSHandler'>,
2023-10-20 10:05:51,066     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000019B5C4D3AC0>,
2023-10-20 10:05:51,066      'bokeh_websocket_path': '/example/ws',
2023-10-20 10:05:51,066      'compression_level': None,
2023-10-20 10:05:51,066      'mem_level': None}),
2023-10-20 10:05:51,066    ('/example/metadata',
2023-10-20 10:05:51,067     <class 'bokeh.server.views.metadata_handler.MetadataHandler'>,
2023-10-20 10:05:51,067     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000019B5C4D3AC0>,
2023-10-20 10:05:51,067      'bokeh_websocket_path': '/example/ws'}),
2023-10-20 10:05:51,067    ('/example/autoload.js',
2023-10-20 10:05:51,067     <class 'bokeh.server.views.autoload_js_handler.AutoloadJsHandler'>,
2023-10-20 10:05:51,067     {'application_context': <bokeh.server.contexts.ApplicationContext object at 0x0000019B5C4D3AC0>,
2023-10-20 10:05:51,067      'bokeh_websocket_path': '/example/ws'}),
2023-10-20 10:05:51,067    ('/example/static/(.*)',
2023-10-20 10:05:51,067     <class 'bokeh.server.views.static_handler.StaticHandler'>,
2023-10-20 10:05:51,067     {}),
2023-10-20 10:05:51,067    ('/?',
2023-10-20 10:05:51,067     <class 'bokeh.server.views.root_handler.RootHandler'>,
2023-10-20 10:05:51,067     {'applications': {'/example': <bokeh.server.contexts.ApplicationContext object at 0x0000019B5C4D3AC0>},
2023-10-20 10:05:51,067      'index': None,
2023-10-20 10:05:51,068      'prefix': '',
2023-10-20 10:05:51,068      'use_redirect': True}),
2023-10-20 10:05:51,068    ('/static/extensions/(.*)',
2023-10-20 10:05:51,068     <class 'bokeh.server.views.multi_root_static_handler.MultiRootStaticHandler'>,
2023-10-20 10:05:51,068     {'root': {}}),
2023-10-20 10:05:51,068    ('/static/(.*)',
2023-10-20 10:05:51,068     <class 'bokeh.server.views.static_handler.StaticHandler'>)]
2023-10-20 10:05:51,079 Bokeh app running at: http://localhost:5006/example
2023-10-20 10:05:51,079 Starting Bokeh server with process id: 28024
2023-10-20 10:05:51,794 Subprotocol header received
2023-10-20 10:05:51,795 WebSocket connection opened
2023-10-20 10:05:51,901 Receiver created for Protocol()
2023-10-20 10:05:51,901 ProtocolHandler created for Protocol()
2023-10-20 10:05:51,901 ServerConnection created
2023-10-20 10:05:51,903 Sending pull-doc-reply from session 'ZKIAk2M1qlVyEDSHGCVtRXDkLLJiuiCYacKSznS1PWpq'
2023-10-20 10:05:52,554 pushing doc to session 'ZKIAk2M1qlVyEDSHGCVtRXDkLLJiuiCYacKSznS1PWpq'
---------------------------------------------- Captured stderr teardown -----------------------------------------------
2023-10-20 10:05:52,561 WebSocket connection closed: code=1000, reason='closed'
2023-10-20 10:05:52,697 Subprotocol header received
2023-10-20 10:05:52,698 WebSocket connection opened
2023-10-20 10:05:52,698 Receiver created for Protocol()
2023-10-20 10:05:52,698 ProtocolHandler created for Protocol()
2023-10-20 10:05:52,699 ServerConnection created
2023-10-20 10:05:52,702 Sending pull-doc-reply from session 'zoexReHATJ8iLT3OGkjtJQj6A4YGHIhwyPmcAYxN1Rj7'
================================================== warnings summary ===================================================
..\Users\xxxxxx\AppData\Local\Continuum\anaconda3\envs\example_env\lib\site-packages\bokeh\core\property\primitive.py:37
  C:\Users\xxxxxx\AppData\Local\Continuum\anaconda3\envs\example_env\lib\site-packages\bokeh\core\property\primitive.py:37: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    bokeh_bool_types += (np.bool8,)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================== short test summary info ===============================================
FAILED test_gui.py::TestUI::test_z - AssertionError: assert 3 == 4
============================================ 1 failed, 1 warning in 13.35s ============================================

Minimum, Reproducible Example:

""" example/main.py """

from bokeh.application import Application
from bokeh.models.widgets import NumericInput, Div
from bokeh.layouts import column, row
from bokeh.plotting import curdoc

header = lambda s: Div(text=f'<h1 style="text-align: center">{s}</h1>')

class AddTwoNumbers(Application):
    def __init__(self):
        self.x = NumericInput(title="x", width=80, value=2)
        self.y = NumericInput(title="y", width=80, value=1)
        self.z = NumericInput(title="z", width=80, value=(self.x.value + self.y.value))

        self.x.on_change("value", self.update_z)
        self.y.on_change("value", self.update_z)

        self.layout = column(header("x + y = z"), row(self.x, self.y, self.z))

    def update_z(self, attr, new, old):
        self.z.value = self.x.value + self.y.value

def create_doc(doc):
    app = AddTwoNumbers()
    doc.add_root(app.layout)
    doc.title = "Add Two Numbers"

create_doc(curdoc())

""" example/test_gui.py """

import os
import pytest
import sys
import subprocess
from pathlib import Path
from time import sleep

from bokeh.client.session import pull_session

global GUI
GUI = None
app_url = "http://localhost:5006/example"

ui_dir = Path(os.path.dirname(__file__))
UI_STARTUP_SEC = 5

def change_x(set_value, test_value):
    with pull_session(url=app_url) as client_session:
        models = {m.title: m for m in client_session.document.models if hasattr(m, "title")}
        models["x"].update(value=set_value)
        client_session.push()
        assert models["x"].value == set_value
        assert models["z"].value == test_value

def setup_module():
    global GUI
    GUI = subprocess.Popen(f"bokeh serve {ui_dir}/ --show --log-level debug")
    sleep(UI_STARTUP_SEC)

def teardown_module():
    global GUI
    GUI.kill()

class TestUI:
    def test_z(self):
        change_x(set_value=3, test_value=4)

if __name__ == "__main__":
    sys.exit(pytest.main())

Environment:

(example_env) c:\example>conda list
# packages in environment example_env:
#
# Name                    Version                   Build
blas                      1.0                         mkl
bokeh                     2.4.3           py310haa95532_0
bzip2                     1.0.8                he774522_0
ca-certificates           2023.08.22           haa95532_0
colorama                  0.4.6           py310haa95532_0
exceptiongroup            1.0.4           py310haa95532_0
freetype                  2.12.1               ha860e81_0
giflib                    5.2.1                h8cc25b3_3
iniconfig                 1.1.1              pyhd3eb1b0_0
intel-openmp              2023.1.0         h59b6b97_46319
jinja2                    3.1.2           py310haa95532_0
jpeg                      9e                   h2bbff1b_1
lerc                      3.0                  hd77b12b_0
libdeflate                1.17                 h2bbff1b_1
libffi                    3.4.4                hd77b12b_0
libpng                    1.6.39               h8cc25b3_0
libtiff                   4.5.1                hd77b12b_0
libwebp                   1.3.2                hbc33d0d_0
libwebp-base              1.3.2                h2bbff1b_0
lz4-c                     1.9.4                h2bbff1b_0
markupsafe                2.1.1           py310h2bbff1b_0
mkl                       2023.1.0         h6b88ed4_46357
mkl-service               2.4.0           py310h2bbff1b_1
mkl_fft                   1.3.8           py310h2bbff1b_0
mkl_random                1.2.4           py310h59b6b97_0
numpy                     1.26.0          py310h055cbcc_0
numpy-base                1.26.0          py310h65a83cf_0
openjpeg                  2.4.0                h4fc8c34_0
openssl                   3.0.11               h2bbff1b_2
packaging                 23.1            py310haa95532_0
pillow                    10.0.1          py310h045eedc_0
pluggy                    1.0.0           py310haa95532_1
pytest                    7.4.0           py310haa95532_0
python                    3.10.13              he1021f5_0
pyyaml                    6.0             py310h2bbff1b_1
sqlite                    3.41.2               h2bbff1b_0
tbb                       2021.8.0             h59b6b97_0
tk                        8.6.12               h2bbff1b_0
tomli                     2.0.1           py310haa95532_0
tornado                   6.3.3           py310h2bbff1b_0
typing_extensions         4.7.1           py310haa95532_0
tzdata                    2023c                h04d1e81_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
xz                        5.4.2                h8cc25b3_0
yaml                      0.2.5                he774522_0
zlib                      1.2.13               h8cc25b3_0
zstd                      1.5.5                hd43e919_0
2

This is not going to work. The auto-sync between the Bokeh server and the BokehJS runtime happens because there is an event loop running on both ends, which is not the case here. After the client_session.push(), a change happens in the server state, but nothing at all happens in the test process that would allow that change to “return” to the client.

You could potentially re-pull the session manually, though that would potentially be flaky due to timing issues. Otherwise you’ll need to actually start an event loop and let the session run it. You can refer to test_client_server.py for some idea or inspiration. Maybe you could even adapt ManagedServerLoop to test things in-process on two loops. (Note: take it and modify to your needs, we are highly unlikely to be interested in changing MSL from any user asks—it’s an internal testing tool that suits our current needs exactly as it is)

But at the end of the day, why do you want to do this? This kind of testing seems duplicative of testing the library itself.

3

Hi @Bryan, thank you for the quick response and providing input on my problem. I had suspected that the unit tests for bokeh itself would be a good reference for what I’m trying to achieve. Your direction to a particular test module is very helpful.

My reason for needing to do this sort of testing is to verify that my bokeh app is not changing the inputs in a way that leads to different outputs than those observed in a functional test involving the same inputs.

The following code works for me, however the usage of run_sync seems clunky. I’m using it to start the IOloop, set a property on the server, then stop the IOloop. Any recommendations for doing this step in a better way?

import pytest
import sys

from bokeh.document import document
from bokeh.application import Application
from bokeh.application.handlers.function import FunctionHandler
from bokeh.client.session import pull_session, push_session
from bokeh.core.types import ID
from bokeh._testing.plugins.managed_server_loop import ManagedServerLoop, MSL
from tornado.ioloop import IOLoop

from main import create_doc


app_url = "http://localhost:5006/"


@pytest.fixture
def app():
    return {"/": Application(FunctionHandler(create_doc))}


@pytest.fixture
def unused_tcp_port():
    return 5006


class TestUI:
    def test_z(self, ManagedServerLoop: MSL, app: Application, unused_tcp_port: int) -> None:
        with ManagedServerLoop(app, unused_tcp_port, io_loop=IOLoop(make_current=True)) as server:
            doc = document.Document()
            create_doc(doc)

            client_session = push_session(
                doc, session_id=ID("test_z"), url=app_url, io_loop=server.io_loop
            )
            server_session = server.get_session("/", client_session.id)

            client_models = {
                m.title: m for m in client_session.document.models if hasattr(m, "title")
            }
            server_models = {
                m.title: m for m in server_session.document.models if hasattr(m, "title")
            }

            set_value, test_value = 3, 4

            def do_nothing():
                return None

            def set_property_on_server():
                server_models["x"].value = set_value

            server.io_loop.add_callback(server_session.with_document_locked, set_property_on_server)
            server.io_loop.run_sync(do_nothing)
            assert server_models["x"].value == set_value

            def verify_change_made_on_client():
                return client_models["x"].value == set_value

            client_session._connection._loop_until(verify_change_made_on_client)
            assert client_models["x"].value == set_value

            def verify_change_made_on_client():
                return client_models["z"].value == test_value

            client_session._connection._loop_until(verify_change_made_on_client)
            assert client_models["z"].value == test_value

            def verify_change_made_on_server():
                return server_models["z"].value == test_value

            client_session._connection._loop_until(verify_change_made_on_server)
            assert server_models["z"].value == test_value

            client_session.close()
            client_session._loop_until_closed()
            assert not client_session.connected


if __name__ == "__main__":
    sys.exit(pytest.main())
4

Not offhand, I am a few years away from any real low-level work in this area, I’d have to spend more time than I have available right now to really comment. I do seem to recall that the original (new) Bokeh server developer had added comments or a function about needing to manually force a “cycle” of the event loop in places, which sounds about like what you’ve done above. I’m not sure there is anything better he did though, you’d have to poke around the existing tests to look.

5

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.