Py.test tests/unit returns 19 failed tests

Problem


Executing py.test tests/unit or py.test returns lots of failed cases.

Error logs :


===End Flaky Test Report===
============================================================================================= short test summary info =============================================================================================
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_dev_cdn[1.4.0rc1] - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_dev_cdn[2.0.0dev3] - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_release_cdn - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_release_dev_cdn - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_release_server - assert 0 == 4
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png[chromium-dimensions0] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png[chromium-dimensions1] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png[chromium-dimensions2] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png[chromium-dimensions3] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png[chromium-dimensions4] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png_with_glyph[chromium-dimensions0] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'va...
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png_with_glyph[chromium-dimensions1] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'va...
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png_with_glyph[chromium-dimensions2] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'va...
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png_with_glyph[chromium-dimensions3] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'va...
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png_with_glyph[chromium-dimensions4] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'va...
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png_with_unicode_minified[chromium] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
FAILED tests/unit/bokeh/io/test_export.py::test_get_screenshot_as_png_with_unicode_unminified[chromium] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'v...
FAILED tests/unit/bokeh/io/test_export.py::test_get_svgs_with_svg_present[chromium] - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
FAILED tests/unit/bokeh/io/test_export.py::test_get_svgs_no_svg_present - selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
====================================================================== 19 failed, 4296 passed, 6 skipped, 158 warnings in 213.13s (0:03:33) =======================================================================

Environment


  • Platform : Ubuntu 18.04
  • Selenium version : 3.141.0
  • chromeDriver version : 2.41
  • Chromium : 80.0.3987.87

Selenium install


$ sudo apt-get install xvfb
$
$ wget https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip
$ unzip chromedriver_linux64.zip
$ sudo mv chromedriver /usr/bin/chromedriver
$ sudo chown root:root /usr/bin/chromedriver
$ sudo chmod +x /usr/bin/chromedriver
$ 
$ pip install pyvirtualdisplay selenium

There’s not really enough information there to go on. You have not said what platform, how you installed things, exactly what command you used to invoke the tests, or run the tests with verbose output so that the whole error can be viewed. All I can do is note that:

  • The tests are currently passing on automated CI
  • The tests are also passing for me locally

The selenium tests are finicky to get running locally, but what is actually important is that they pass on CI (which they are, as I mentioned). So, the simplest thing for you might be to just skip these tests by adding -m not selenium to your pytest invocation.

After updating the local repo there seems to be no failed Selenium tests.
But still there are some failed Test_autoload_static tests.


Will it be all right if I skip Selenium tests when running the tests locally ?!

What are the Test_autoload_static tests ? is it all right if they fail ?

Test output


==================================================================================================== FAILURES =====================================================================================================
_________________________________________________________________________________ Test_autoload_static.test_js_dev_cdn[1.4.0rc1] __________________________________________________________________________________

self = <test_standalone.Test_autoload_static object at 0x7fc5553d90a0>, version = '1.4.0rc1', monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc55547c8e0>
driver = <selenium.webdriver.chrome.webdriver.WebDriver (session="bdc97e65f12e65032c7ef2ee9ec9da75")>
test_file_path_and_url = ('/home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_dev_cdn.html', 'http://127.0.0.1:8000//home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_dev_cdn.html')
test_plot = Figure(id='8928', ...)

    @pytest.mark.parametrize("version", ["1.4.0rc1", "2.0.0dev3"])
    @pytest.mark.selenium
    def test_js_dev_cdn(self, version, monkeypatch, driver, test_file_path_and_url, test_plot) -> None:
        monkeypatch.setattr(buv, "__version__", "1.4.0rc1")
        monkeypatch.setattr(resources, "__version__", "1.4.0rc1")
        js, tag = bes.autoload_static(test_plot, CDN, "some/path")
    
        page = PAGE.render(js=js, tag=tag)
    
        path, url = test_file_path_and_url
        with open(path, "w") as f:
            f.write(page)
    
        driver.get(url)
    
        scripts = driver.find_elements_by_css_selector('head script')
>       assert len(scripts) == 4
E       assert 0 == 4
E        +  where 0 = len([])

tests/unit/bokeh/embed/test_standalone.py:105: AssertionError
_________________________________________________________________________________ Test_autoload_static.test_js_dev_cdn[2.0.0dev3] _________________________________________________________________________________

self = <test_standalone.Test_autoload_static object at 0x7fc555649310>, version = '2.0.0dev3', monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc55555df70>
driver = <selenium.webdriver.chrome.webdriver.WebDriver (session="bdc97e65f12e65032c7ef2ee9ec9da75")>
test_file_path_and_url = ('/home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_dev_cdn.html', 'http://127.0.0.1:8000//home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_dev_cdn.html')
test_plot = Figure(id='9008', ...)

    @pytest.mark.parametrize("version", ["1.4.0rc1", "2.0.0dev3"])
    @pytest.mark.selenium
    def test_js_dev_cdn(self, version, monkeypatch, driver, test_file_path_and_url, test_plot) -> None:
        monkeypatch.setattr(buv, "__version__", "1.4.0rc1")
        monkeypatch.setattr(resources, "__version__", "1.4.0rc1")
        js, tag = bes.autoload_static(test_plot, CDN, "some/path")
    
        page = PAGE.render(js=js, tag=tag)
    
        path, url = test_file_path_and_url
        with open(path, "w") as f:
            f.write(page)
    
        driver.get(url)
    
        scripts = driver.find_elements_by_css_selector('head script')
>       assert len(scripts) == 4
E       assert 0 == 4
E        +  where 0 = len([])

tests/unit/bokeh/embed/test_standalone.py:105: AssertionError
____________________________________________________________________________________ Test_autoload_static.test_js_release_cdn _____________________________________________________________________________________

self = <test_standalone.Test_autoload_static object at 0x7fc55556b130>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc55552e0d0>
driver = <selenium.webdriver.chrome.webdriver.WebDriver (session="bdc97e65f12e65032c7ef2ee9ec9da75")>
test_file_path_and_url = ('/home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_release_cdn.html', 'http://127.0.0.1:8000//home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_release_cdn.html')
test_plot = Figure(id='9088', ...)

    @pytest.mark.selenium
    def test_js_release_cdn(self, monkeypatch, driver, test_file_path_and_url, test_plot) -> None:
        monkeypatch.setattr(buv, "__version__", "2.0.0")
        monkeypatch.setattr(resources, "__version__", "2.0.0")
        js, tag = bes.autoload_static(test_plot, CDN, "some/path")
    
        page = PAGE.render(js=js, tag=tag)
    
        path, url = test_file_path_and_url
        with open(path, "w") as f:
            f.write(page)
    
        driver.get(url)
    
        scripts = driver.find_elements_by_css_selector('head script')
        for x in scripts:
            print(x.get_attribute("src"))
>       assert len(scripts) == 4
E       assert 0 == 4
E        +  where 0 = len([])

tests/unit/bokeh/embed/test_standalone.py:127: AssertionError
__________________________________________________________________________________ Test_autoload_static.test_js_release_dev_cdn ___________________________________________________________________________________

self = <test_standalone.Test_autoload_static object at 0x7fc555538970>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc5555ca6d0>
driver = <selenium.webdriver.chrome.webdriver.WebDriver (session="bdc97e65f12e65032c7ef2ee9ec9da75")>
test_file_path_and_url = ('/home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_release_dev_cdn.html', 'http://127.0.0.1:8000//home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_release_dev_cdn.html')
test_plot = Figure(id='9168', ...)

    @pytest.mark.selenium
    def test_js_release_dev_cdn(self, monkeypatch, driver, test_file_path_and_url, test_plot) -> None:
        monkeypatch.setattr(buv, "__version__", "2.0.0-foo")
        monkeypatch.setattr(resources, "__version__", "2.0.0-foo")
        js, tag = bes.autoload_static(test_plot, CDN, "some/path")
    
        page = PAGE.render(js=js, tag=tag)
    
        path, url = test_file_path_and_url
        with open(path, "w") as f:
            f.write(page)
    
        driver.get(url)
    
        scripts = driver.find_elements_by_css_selector('head script')
        for x in scripts:
            print(x.get_attribute("src"))
>       assert len(scripts) == 4
E       assert 0 == 4
E        +  where 0 = len([])

tests/unit/bokeh/embed/test_standalone.py:149: AssertionError
___________________________________________________________________________________ Test_autoload_static.test_js_release_server ___________________________________________________________________________________

self = <test_standalone.Test_autoload_static object at 0x7fc555562a60>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc555562e20>
driver = <selenium.webdriver.chrome.webdriver.WebDriver (session="bdc97e65f12e65032c7ef2ee9ec9da75")>
test_file_path_and_url = ('/home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_release_server.html', 'http://127.0.0.1:8000//home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/embed/test_js_release_server.html')
test_plot = Figure(id='9248', ...)

    @pytest.mark.selenium
    def test_js_release_server(self, monkeypatch, driver, test_file_path_and_url, test_plot) -> None:
        monkeypatch.setattr(buv, "__version__", "2.0.0")
        monkeypatch.setattr(resources, "__version__", "2.0.0")
        js, tag = bes.autoload_static(test_plot, resources.Resources(mode="server"), "some/path")
    
        page = PAGE.render(js=js, tag=tag)
    
        path, url = test_file_path_and_url
        with open(path, "w") as f:
            f.write(page)
    
        driver.get(url)
    
        scripts = driver.find_elements_by_css_selector('head script')
>       assert len(scripts) == 4
E       assert 0 == 4
E        +  where 0 = len([])

tests/unit/bokeh/embed/test_standalone.py:169: AssertionError
================================================================================================ warnings summary =================================================================================================
/home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/site-packages/pytest_asyncio/plugin.py:39: 67 tests with warnings
  /home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/site-packages/pytest_asyncio/plugin.py:39: PytestDeprecationWarning: direct construction of Function has been deprecated, please use Function.from_parent
    item = pytest.Function(name, parent=collector)

/home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/site-packages/pytest_asyncio/plugin.py:45: 67 tests with warnings
  /home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/site-packages/pytest_asyncio/plugin.py:45: PytestDeprecationWarning: direct construction of Function has been deprecated, please use Function.from_parent
    item = pytest.Function(name, parent=collector)  # To reload keywords.

tests/unit/bokeh/document/_util_document.py:38
  /home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/document/_util_document.py:38: RuntimeWarning: Property "name" in class Model was overridden by a class attribute "name" in class ModelThatOverridesName; it never makes sense to do this. Either Model.name or ModelThatOverridesName.name should be removed, or Model.name should not be a Property, or use Override(), depending on the intended effect.
    class ModelThatOverridesName(Model):

tests/unit/bokeh/test_model.py::test_all_builtin_models_default_constructible
  BokehDeprecationWarning: 'WidgetBox' is deprecated and will be removed in Bokeh 3.0, use 'bokeh.models.Column' instead

tests/unit/bokeh/application/handlers/test_directory.py::Test_DirectoryHandler::test_directory_mainipynb_adds_roots
  /home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/site-packages/IPython/core/inputsplitter.py:21: DeprecationWarning: IPython.core.inputsplitter is deprecated since IPython 7 in favor of `IPython.core.inputtransformer2`
    warn('IPython.core.inputsplitter is deprecated since IPython 7 in favor of `IPython.core.inputtransformer2`',

tests/unit/bokeh/core/test_json_encoder.py::TestBokehJSONEncoder::test_numpyint
  /home/arion/PROJECTS/GSoC 2020/bokeh/tests/unit/bokeh/core/test_json_encoder.py:69: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
    npint = np.asscalar(np.int64(1))

tests/unit/bokeh/core/property/test_bases.py::TestProperty::test_matches_incompatible_arrays
  /home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/site-packages/numpy/core/numeric.py:2327: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
    return bool(asarray(a1 == a2).all())

tests/unit/bokeh/plotting/test__lengends.py: 16 tests with warnings
  BokehDeprecationWarning: 'legend' keyword is deprecated, use explicit 'legend_label', 'legend_field', or 'legend_group' keywords instead

tests/unit/bokeh/sampledata/test_airports.py::Test___all__::test___all__
  /home/arion/PROJECTS/GSoC 2020/bokeh/bokeh/sampledata/airports.py:64: FutureWarning: pandas.io.json.json_normalize is deprecated, use pandas.json_normalize instead
    data = pd.io.json.json_normalize(airports['features'], meta=schema)

tests/unit/bokeh/server/test_server__server.py::test__autocreate_signed_session_autoload
  /home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/asyncio/base_events.py:431: RuntimeWarning: coroutine 'WebSocketProtocol13.write_message.<locals>.wrapper' was never awaited
    task = tasks.Task(coro, loop=self, name=name)

tests/unit/bokeh/util/test_version.py::Test__base_version_helper::test_rc_version_stripped
  /home/arion/anaconda3/miniconda/envs/bokeh-dev/lib/python3.8/site-packages/_pytest/fixtures.py:871: DeprecationWarning: URLopener style of invoking requests is deprecated. Use newer urlopen functions/methods
    func()

-- Docs: https://docs.pytest.org/en/latest/warnings.html
===Flaky Test Report===

test_ping passed 1 out of the required 1 times. Success!
test_next_tick_runs passed 1 out of the required 1 times. Success!
test_timeout_runs passed 1 out of the required 1 times. Success!
test_periodic_runs passed 1 out of the required 1 times. Success!
test_next_tick_does_not_run_if_removed_immediately passed 1 out of the required 1 times. Success!
test_timeout_does_not_run_if_removed_immediately passed 1 out of the required 1 times. Success!
test_periodic_does_not_run_if_removed_immediately passed 1 out of the required 1 times. Success!
test_same_callback_as_all_three_types passed 1 out of the required 1 times. Success!
test_adding_next_tick_twice passed 1 out of the required 1 times. Success!
test_adding_timeout_twice passed 1 out of the required 1 times. Success!
test_adding_periodic_twice passed 1 out of the required 1 times. Success!
test_remove_all_callbacks passed 1 out of the required 1 times. Success!
test_removing_next_tick_twice passed 1 out of the required 1 times. Success!
test_removing_timeout_twice passed 1 out of the required 1 times. Success!
test_removing_periodic_twice passed 1 out of the required 1 times. Success!
test_adding_next_tick_from_another_thread passed 1 out of the required 1 times. Success!
test__lifecycle_hooks passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
============================================================================================= short test summary info =============================================================================================
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_dev_cdn[1.4.0rc1] - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_dev_cdn[2.0.0dev3] - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_release_cdn - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_release_dev_cdn - assert 0 == 4
FAILED tests/unit/bokeh/embed/test_standalone.py::Test_autoload_static::test_js_release_server - assert 0 == 4
======================================================================= 5 failed, 4310 passed, 6 skipped, 158 warnings in 273.79s (0:04:33) =======================================================================

Also it’ll be of great help if a short guide / link to a guide is given in the docs to set up Selenium.

Will it be all right if I skip Selenium tests when running the tests locally

I don’t really understand the question. We always run all CI tests (which include many more than what you are running above, e.g. JS tests, full integration tests, docs build, codebase checks, and a suite of hundreds of image diff tests) on every push to GH. It’s up to decide what you run locally as as your routine. If you make a PR introduces a real test failure, the automated CI will catch it, and it would just have to be fixed before merge.

What are the Test_autoload_static tests ? is it all right if they fail

They test that the autoload_static function performs as expected. It’s definitely not OK for them to fail for real, but AFAIK they are not failing for real here. Unless you are doing work near autoload_static then it would be unlikely they you could break or affect them which is why I suggest that you, personally, might want to just skip them for now as the simplest, quickest solution.

Also it’ll be of great help if a short guide / link to a guide is given in the docs to set up Selenium.

We suggest people use conda in dev guide. That’s what the CI automation uses.