Can someone explaing like I’m 5 and step by step how to setup Firefox and geckodriver in order to simply export plots to png? I don’t have conda nor apt, but virtualenv and dnf.
I tried putting the gecko webdriver downloaded from the github page on my /usr/bin and /usr/local/bin paths but still it doesn’t work. I have the default Firefox version installed with Fedora Linux 35, nothing else. I tried every single python package/module available on pip to auto-download/sync/setup the geckodriver or chromedriver and guess what? None of them work.
I just need to export some png, why it has to be so difficult to do so? This is so incredibly frustrating and user unfriendly, this is a really basic function for a plotting lib and still the user has to setup tons of things that he should not care about.
First off: it is typically important that the webdriver version to be compatible to the installed browser version. So perhaps you just need to install (manually or otherwise) a different version of the webdriver than you tried already. I would expect there to be some error output if this was the case. It might certainly be helpful if you provide more information about what exactly “doesn’t work” actually means in detail (logs, error messages, etc).
@ilpomo Bokeh is, first and foremost, a browser plotting library. It was created, specifically and intentionally and primarily, to afford interactive plots in browsers. That opens a lot of rich new possibilities, but it is also not without tradeoffs, as you are discovering. A main one being that before a plot can be saved as a PNG, it has to be rendered in a browser and then somehow extracted from the running browser process. [1] That means: headless browsers and Selenium webdrivers like Firefox and geckodriver (or Chrome and chromedriver).
Believe me, I would love for there to be a better UX, but this is literally the best solution that has been possible in nearly ten years, after much gnashing of teeth. There has been an idea floated of utilizing Chrome’s dev API directly instead. That would eliminate the need for a separate web driver. However:
it’s a non-trivial amount of work, and there is no funding to prioritize it at present
some people will refuse to use Chrome on principle
So a webdriver option will always need to be available.
Anyway, options for you specifically:
Install miniconda, and use it only to bootstrap a successful geckodriver and FF install
Try with chrome/chromedriver instead (manual install, or miniconda)
Use a different tool.
If your workflow is more centered around static images, rather than interactive, then perhaps MPL is a simpler or better option for you.
Unfortunately I don’t use fedora, and have never used dnf, so I don’t have any experience to offer you specific guidance about those. Perhaps others here can chime in.
The alternative is to replicate the entire BokehJS plotting machinery in some non-browser backend, e.g. using Cairo or using MPL or something else. Various ideas along that axis have been considered and all rejected. It would be a gargantuan effort, doubling the maintenance burden, that is just ultimately far, far, far outside the envelope of our available resources, and at the end of the day we wouldn’t even be able to guarantee identical output. ↩︎
Just for the sake of record, other ideas/possibilities here, re: “why is this so hard”:
Docker image
I suppose we could create and publish Docker image that should be guaranteed to function. I am sure many people would not be happy with a docker-based workflow either, especially when it is a huge image with an entire python+browser stack pre-installed. But maybe it’s a good secondary or “fallback” option. cc @Ian_Thomas
NodeJS
It would be great if BokehJS could just render inside a node process, e.g. using node-canvas but AFAIK this is still not a practical reality cc @mateusz for more comment Regardless some people would inevitably complain about having to install NodeJS in order to use a “Python” library.
Run a service
We could run a service (“Imgur for Bokeh plots”) to generate the images, and in fact I even created a proof of concept at one point. Obvious problems:
Won’t work in airgap, some people would object on privacy grounds
Money concerns: Bokeh is OSS project, not a company, who pays for the servers!
Legal concerns: very serious in my mind, DMCA safe harbor status etc, people abusing to host illegal content, etc. I would never consider this without getting real, actual legal advice.
TLDR; There really, truly is no universally perfect solution here that will satisfy everyone. I promise lots of people have given it extensive thought over many years.
@ilpomo It it usually a bad idea to ask for people to explain things to you as though you were 5 years old, as the answer is likely to be “you are not yet old enough to understand complicated software”. That is not a good starting point if you really want people to help you.
As @Bryan has said, you don’t need to use Bokeh if you want to generate a PNG file. Bokeh is complicated software that produced high-quality high-performance interactive graphics in a web browser. You do not need all of that complexity if you just want the equivalent of a static screenshot of the final output.
If you really want to use Bokeh and want specific advice:
Use conda not virtualenv.
Don’t download binaries and put them in system directories and expect them to work. Use your distro’s package management system instead.
To reply to @Bryan’s question about the docker image approach, I don’t think we could call it a “good secondary or fallback option”. At best maybe an “acceptable secondary or fallback option” for some users, but it requires them to become at least slightly proficient in use of docker which is more esoteric than say conda or linux package management systems. It is still worth pursuing at it may be the least worst alternative to the current approach.
@Ian_Thomas Right, by “good” I really just meant “at least one option that is foolproof if all else fails”. The current need to match webdriver and browser versions really is an unpleasant burden for users, but also something that is completely outside our control. There is just nothing at all we can do on our end to make that situation better for users, because installation process can vary wildly across platforms. (Therefore: we can only help if a platform can fixed that we can control, ala Docker)
What about a python binding of Cairo Language bindings? Something like Pycairo, we could do something similar to the SVG or WebGL backend.
Or going back to the node-canvas option we could use Deno that compiles to a single executable file with the only dependency being Cairo. https://deno.land/manual/tools/compiler#compiling-executables
Or a browser automation library that includes the browsers like playwright, which requires node but would only be needed for exporting images and maybe tests.
I mentioned that in my footnote above. Rewriting almost all of BokehJS rendering in Python to use Cairo (or Qt or whatever) is an enormous task. It would require an incredible amount of tests, but also entirely new kinds of testing infrastructure to support. Any feature we wanted to ever now would become much harder because it would have to be added in an equivalent way on both backends. There are probably lots of corner cases that would simply not have a technical solution at all to maintain parity (font handling especially likely). Those differences lead to documentation and support burden. We (the volunteer Bokeh team) simply do not have the organizational capacity or this. Not even close.
similar to the SVG or WebGL backend.
Both those already also happen in the browser, it it is a much, much shorter leap to support them. Also WebGL support has no bearing on image exports, really, and in any case is only supported on a small subset of glyphs. SVG works because someone else built a canvas-to-svg library we can leverage as a drop-in replacement to canvas (and BTW still also requires using a headless browser, no different from PNG export).
Or going back to the node-canvas option we could use Deno that compiles to a single executable file with the only dependency being Cairo. Compiling Executables | Deno Docs
It think a node-canvas solution would be an improvement. @mateusz will have to comment on how close to feasible it is today. Last I heard there were multiple blockers to running BokehJS in node.
Further investigation on Getting started | Playwright Python shows that it does not need NodeJS at all. It downloads and install browser binaries for Chromium, Firefox and WebKit on demand.
@IuryPiva That project is new to me. If it can handle all the details of installing and driving a browser via a programmatic API, that’s definitely worth looking into. AFAIK the main requirement is an ability to scale the plot/image according to user-specified dimensions before grabbing the screenshot.
It still won’t satisfy everyone and all cases, though, just to be the ever-downer:
“I’m on an airgapped network and can’t download anything”
“I can’t install new un-vetted software due to organizational prohibitions”
“I already have Chrome, why is Bokeh downloading a huge old Chromium?”
Edit: well, it seems very geared toward CI usage, e.g. this “feature” runs completely opposite to our needs I think:
Playwright keeps track of the clients that use its browsers. When there are no more clients that require particular version of the browser, that version is deleted from the system.