Some unsolicited feedback

Hi all,

I just got into developing web dashboards, and I’ve been spending the last few weeks conducting a trade study of several frameworks for developing Python codes into browser-based GUIs. I spent a few days writing up a prototype application in Dash and also a few days writing up the same prototype application in Bokeh. I thought that this community might be interested in hearing my thoughts.

I actually loved using Bokeh, and I consider it to be the framework better suited to my application. I like attaching callbacks to the object that started the callback, as opposed to Dash, where I need to attach callbacks to the object that will be changing. I also like not having to worry about saving/loading the current state of the application.

Nonetheless, I’ve decided to go forward with Dash, and this is for one reason only: it’s because the documentation for Bokeh is simply too sparse to support my needs. Not only are there not examples for most of the functions in the Reference section, even the Reference section itself poorly documents functions. Many class methods don’t state the arguments required when calling the method or what the optional arguments are. Sometimes I need to go to the parent of the parent of the class to figure out how to use it.

I think that this is likely the main reason that Dash has overtaken Bokeh as the most commonly used framework for creating online dashboards. The documentation for Dash (while still not fantastic) is leaps and bounds better than Bokeh. This has led to Dash having a larger user base, which further helps its documentation, as StackOverflow has far more questions related to common uses of Dash than Bokeh has. I expect this feedback loop to continue to grow, which will cause Dash to increase its lead over Bokeh in the coming years.

I really like this project, and I feel that the project developers and maintainers need to hear this information. I know that improving the project documentation is in your project’s roadmap, and you’re probably thinking, “Who’s this guy who hasn’t contributed anything to our project and isn’t willing to do so to tell us how to do our volunteer work?” and you’re right. Nonetheless, I urge you to set aside further development of this development and to sink as many resources as possible into documentation lest your past and present efforts be in vain as users and developers abandon a great tool that’s simply too hard to learn how to use.

Thank you.

Hi @efremdan1,

Firstly, I want to emphasize how much we appreciate your candid and helpful feedback. You’re quite right that this is the sort of user experience that project maintainers need to hear. We really like Bokeh too, obviously, and are motivated to improve in ways that are meaningful to our user base.

You mention that improving docs is on the Bokeh roadmap. It’s actually actively being worked on-- we have an excellent technical writer through Google Season of Docs who has made some strides with our User Guide, breaking sections down into more elemental how-to steps with minimal examples, and we expect the first round of changes to be published with 2.3. This does not affect the Reference yet, though, which sounds like it was your primary resource.

We generally don’t consider the Reference to be a learning guide, so it’s not geared towards beginning users or a how-to-do-xyz mindset. In your exploration, did you happen to make use of the User Guide, Gallery examples, or tutorial notebooks? If so, did you find these to be any more or less useful than the Reference? (If not, could these options be better publicized or promoted?)

We’re really interested in knowing more about how your experience with the docs went, and any specific examples you can provide to illustrate shortcomings. There are a few examples in the Reference, but I think it’s fair to say there are many opportunities for improvement there; the point you make about class methods is interesting, and if you are able to point to specific instances, we’d like to look into how that might be improved.

Again, thank you for taking the time to provide feedback. I came to Bokeh as a data scientist, and some of what you mention is familiar to me, but having user feedback is so valuable because it illustrates what the larger community finds important-- which is not always obvious to the core developers who are so close to the project. If you’re interested, we also invite you to the Bokeh developer Slack to talk further.

–Carolyn

Definitely strong to everything @carolyn says above.

The only other thing I’d like to note is that Bokeh is a very general tool with wide application, and a large number of Bokeh users are not in the business of building dashboards. They are doing things like: exploratory analysis in notebooks, creating basic engineering GUIs, or embedding standalone charts in websites in simpler ways than desribed by the word “dashboard”. I think is is fair to say that our docs currently cater more towards these use other cases than dashboarding, mostly because our personal backgrounds are more close to those use cases.

We’d certainly like to improve the docs in ways that help more with the dashboard case, but I’d also like to point out an additional option that may be worth exploring, which is Panel. The Panel project is built on top of Bokeh to leverage all of its best features, by folks who are more specifically invested in the “dashboard use-case”. I’d encourage anyone looking to use Bokeh to create dashboards to start out there. (And in fact we are already updating our docs to point users more directly towards Panel for dashboarding)

Thank you for responding so cordially. It’s lovely to hear that you’re open to constructive criticism; I was a little worried about how my remarks were going to be taken.

@carolyn: Yes, I went through the full User Guide, and I looked through most of the Gallery examples and tutorial notebooks. They were indeed the first things I went through as I was becoming acquainted to Bokeh. What they have is well-written, though in my opinion I think they should be more friendly to new users who need their hands held a built more.

To illustrate: I’m at the stage in my prototype applications that I want to deploy them so I can see how timing is affected by their being on a remote server. Dash’s documentation (Deploy your Dash App | Dash for Python Documentation | Plotly) actually walks the user through an example of using Heroku to host their application. It’s simple to copy/paste their example and get my application online to show my team. Bokeh’s documentation (Running a Bokeh server — Bokeh 2.4.2 Documentation) assumes the user knows how to do all that, and it’s really just an FAQ for advanced developers. There aren’t even links to tutorials on how or where to host. Personally, I’m a Python developer making my first foray into web development and trying to get my prototype up and running fast so I can choose which framework works best and then learn that one in detail; Bokeh’s docs don’t help me with this.

(Splitting the reply into 2 posts to get around not being allowed to use more than 2 links per post.)

And as a specific example of the Reference section needing improvement: I wanted to make a Spinner widget that accepted floats. bokeh.models.widgets.inputs — Bokeh 2.3.3 Documentation did not help me figure out what I needed to do to do this; I figured that just using the defaults would allow the user to enter any value desired, but my application kept on casting the value entered into an integer. I actually had to look at the source code to see that the default “step” is set to 1, which is what was causing the integer cast. This took me the better part of an hour to figure out, and when I figured it out, I became very frustrated that default function values aren’t given in the documentation. Dash does this properly, e.g., at Input | Dash for Python Documentation | Plotly, you can see that the default for “step” is “any.”

@Bryan: I was not aware of Panel. I’ll be looking into this. Thank you.

Hi

I also fully understand and have experienced the pains above. Bokeh and Panel are really, really powerful tools but it can be hard to see, understand and learn.

Dash and Streamlit have commercial backing and resources on another planet than Bokeh and Panel.

I created awesome-panel.org to show case what is possible. I’ve only scratched the surface so far.

@efremdan1 Thank for your elaborations. As @MarcSkovMadsen notes, Bokeh does not have any commercial backing or the level or resources that Plotly does (though FWIW I think we have held our own against an entire 20+ person funded company fairly well for eight years, and and am very proud of all the work every contributor has put in to Bokeh!)

About the specific points:

• The defaults should definitely be in the reference guide, I am not sure why it is not. The information readily available to the docs build system we use. I’ve made an issue to add default values for properties here; Reference guide should display property defaults · Issue #10628 · bokeh/bokeh · GitHub and expect to have this done for the next release.

• Specifically re: Spinner, Fust as FYI, the live code sample in the User’s Guide here:

  Adding widgets — Bokeh 2.4.2 Documentation

  does demonstrate a non-integer step, but the live code samples on that page are quite new, if you are using any even slightly older docs version you would not have seen that.

• The lack of elaboration on on server deployment is purely due to a lack of depth on the team in these areas. (I’ve never used Heroku, e.g.) I’ve long asked the community for help with developing docs for different deployment scenarios, but to date have not gotten much help in that regard. Today, I think the way forward here is to promote Panel directly for users with these sorts of deployment needs, since the Holoviz team is able to focus more directly on exactly these scenarios. We just added a reference to Panel at the top of our “Other Tools” user guide section here: #10539 compat.rst by Konntroll · Pull Request #10605 · bokeh/bokeh · GitHub and I intend to promote Panel for dashboarding/app deployment in other places like the tutorial notebooks as well.

Regarding deployment. There are so many Streamlit deployment tutorials out there that the community can use on of those directly or to contribute Bokeh/ Panel deployment guides.

Just noting you can see the change deployed in the preview dev docs:

Screen Shot 2020-11-03 at 11.13.18 AM1396×1782 306 KB

@Bryan: So happy to see!

Regarding deployment: The Panel docs (Server Deployment — Panel 0.12.5 documentation) has all the information I was looking for. It would be good to make this more visible from the Bokeh website.

Great discussion and thanks for sharing your thoughts @efremdan1. I myself am still on the fence, having done two medium sized projects now, one with Dash and one with Bokeh.

For my next project I am looking into redash, which uses Dash. Although I prefer the architecture and overall philosophy of Bokeh more, having complete frameworks for dashboarding significantly lightens the development load in my daily work.

Will have a look at Panel to see how that has evolved over the past years.

Looks like I’m sticking around with bokeh a bit longer. The websockets allowing for a session to retain its state is just too wonderful

I figured it might be helpful to catalog some other issues that I think can be fixed with the documentation:

The other day, I was looking into using the tabs widget. I knew to find the user guide section on tabs here: Adding widgets — Bokeh 2.4.2 Documentation. However, I wanted to see what options were available for the widget, which requires looking at the references section. This was very tricky. Going to Reference — Bokeh 2.4.2 Documentation, I recalled that the widgets are all documented under bokeh.models. Then, I opened each of the bokeh.models.widgets options and Ctrl+f’ed for “tabs”. After not finding the widget, I had to resort to the search bar, which eventually showed me that the tabs widget is actually under bokeh.models.layouts.

I think it would be very helpful for the User Guide to have a link that brings you to the corresponding Reference section.

Another:

I’m embedding the Bokeh server in a standalone document. Per Running a Bokeh server — Bokeh 2.4.2 Documentation, “Also note that most every command line argument for bokeh serve has a corresponding keyword argument to Server . For instance, setting the –allow-websocket-origin command line argument is equivalent to passing allow_websocket_origin as a parameter.”

I wanted to pass in --check-unused-sessions, which I guessed would be possible. However, per the Reference on Server (bokeh.server.server — Bokeh 2.4.2 Documentation), this was not an option. I tried anyway, and no warning was passed, but my passing in this argument had no effect on the program.

I dug deeper. I looked at the Reference on Serve (serve — Bokeh 2.4.2 Documentation), and then I looked into the source code on Serve (bokeh.command.subcommands.serve — Bokeh 2.4.2 Documentation). I found that the “check_unused_sessions_milliseconds” was the actual name of the argument I needed to pass in. I tried this, and indeed it worked.

Kind of crazy needing to look at the source code on “Serve” to figure out the argument needed to pass into “Server”…

@efremdan1 I’m glad to hear that! And thanks also for the continued comments. Just purely as an FYI check_unused_sessions_milliseconds is actually an argument to the Bokeh Tornado application, and is documented here:

bokeh.server.tornado — Bokeh 3.3.2 Documentation

Technically the Server docs cover this at the bottom where they state:

Any remaining keyword arguments will be passed as-is to BokehTornado .

But I note that the reference link to BokehTornado in that sentence is not currently working, and that certainly does not help with navigation. It would be better of course if those arguments could be listed explicitly right there, but I think that would require some code and automation to keep in sync (I personally would regard it as worse if there was a list there, but it was not accurate).

Both of the above comments do motivate me to probe another question about your usage of the docs. Namely, are you attempting to utilize the search function as a primary means to navigate? For both of the above, a search on Tabs or your original thought of check_unused_sessions, the search results return the desired page at the top:

Screen Shot 2020-11-12 at 9.12.46 AM2308×612 71.5 KB

Screen Shot 2020-11-12 at 9.11.33 AM1136×678 68.2 KB

For me if I need anything at all from the ref-guide, my very first action is to use search for it because that typically has the desired page as the first result. But it’s not a given that everyone uses (or knows to use) the docs on this manner. I notice you mentioned “resorting” to search which makes me wonder if there is a dis-motivation to using search as a first choice.

In any case, I do agree:

• The docs need much more cross-referencing in general. Tthere is ageneral issue for this somewhere, but I’ve made a specific issue for this particular case here.
• The bokeh.models are broken out by submodule for historical reasons ^[1], which provides a taxonomy (sort-of) But at this point it would probably be better to just have everything in one pile, alphabetized.

That said, Bokeh (and it’s documentation) is large. I don’t believe there is a single information architecture that would not involve negative trade-offs for some users or usage patterns. For this reason I do think the search is absolutely the easiest and fastest way to navigate the docs and I’d like to find ways to encourage folks to think of search as the tool they should reach for first.

1. Early on we encouraged importing from submodules, e.g. from bokeh.models.layouts import Row and so the reference was also arranged along these line. But we stopped doing that and changed to only encouraging importing everything from top-level bokeh.models ↩︎

I was looking through the Reference directly due to the User Guide on the widgets page (Adding widgets — Bokeh 2.4.2 Documentation) saying “For more information about the attributes to watch using .on_change , see the Reference. (Information about widgets is found under bokeh.models.” This led me to click on the bokeh.models links, and whenever I wanted to find out more information about a widget, that was my go-to.

Also, I am quite used to using search for most other large packages, such as NumPy. However, I generally do this search from my browser, by searching for something like “numpy interpolate”. I did try this for Bokeh, but all the top google results are for old versions of the documentation. For example, if I google right now “bokeh tabs”, the top result is from Bokeh 1.0.2, and the next result is from Bokeh 0.12.6. I don’t see any result for the current documentation.

I think this is a more typical use case, since most users won’t want to first have to navigate to the Bokeh Reference page and THEN conduct the search.

Hi @efremdan1 sorry I should have been more specific: I was not talking about Google search for the Bokeh docs, I meant have you tried the built-in search that is directly on the Bokeh docs site? Those results are locked to version you are currently looking at and will be generally much better than a Google search (for that reason).

Screen Shot 2020-11-12 at 5.21.46 PM1588×1148 170 KB

I understood. I was saying that it’s quite unusual for me (and I think most people) to use the build-in search function. It is much more common for me (and again, most people) to google search documentation, since that eliminates the step of first having to navigate to the Bokeh Reference page prior to conducting the search.

Other packages usually have the most recent version as their first google hit (try searching for “numpy interpolate” and you’ll get a link to the current docs). Packages that don’t have this usually have a link at the top of the old docs that warn that the documentation page you’re looking at is out-of-date, and they have a link to that page in the current version of the docs (e.g., see 7. Input and Output — Python 2.7.18 documentation, which links at the very top to 7. Input and Output — Python 3.10.0 documentation).

It is much more common for me (and again, most people) to google search

I will have to take “most people” as speculation. We were actually missing site search for a few years after Google shuttered GSS, and an issue to bring the site search feature back was one of the most discussed and voted issues we ever had. My own experience is that many people do want to use site-search and I’d like to find ways to encourage it even more because it is effective and efficient.

Other packages usually have the most recent version as their first google hit

This is controlled by Google, I am afraid. I am not sure what we can do. We publish canonicial URLs pointing to “latest” on every page, which AFAIK should direct google results to “latest”, but does not seem to have desired effect. If there is anyone who is an expert at optimizing GSE results who would like to work on this, I would love help, because I have tried everything I know to try, and am frankly past a point of frustration with it.

Packages that don’t have this usually have a link at the top of the old docs that warn that the documentation page you’re looking at is out-of-date,

I am confused about this point. We have long had such a banner at the top on old versions (and also on pre-release versions). Are you saying you are not seeing the bright banner like in the screnshot below on your system/browser?

Screen Shot 2020-11-12 at 9.21.37 PM2570×1070 273 KB

Edit: older versions are even more stern:

Screen Shot 2020-11-12 at 10.30.36 PM2408×734 115 KB

I do see those banners, but the link is just to the generic documentation page for the current release, so then I need to conduct my search query again. With the Python link I sent earlier (7. Input and Output — Python 2.7.18 documentation), the banner link goes directly to the current doc page on that same topic.

Sorry about your frustration on the Google results. Python seems to be in a similar situation, though Numpy seems to have figured it out. I just posted to the Numpy message board (http://numpy-discussion.10968.n7.nabble.com/How-did-Numpy-get-its-latest-version-of-the-documentation-to-appear-at-the-top-of-Google-search-resu-td48668.html) to see if one of their developers is willing to lend a hand.

I should mention: Thank you very much for all your work on this wonderful package. Hopefully I’m not pushing the boundary between giving constructive feedback and straight-out complaining, which I know is a fine line.

Our document structure can and has changed over the years, so it’s not always the case that a corresponding link exists in newer versions. Here I think we have a difference in value judgment: I would deem it better to omit links entirely, if there is the possibility that they can be wrong, rather than always auto-generate a link to “latest” that may or may not work (even if it usually works).

But that’s why discussion is good. Although it’s not my preference, if someone feels strongly and wants to implement some simple regex to substitute “latest” in the current page URL, the code to update is here:

https://github.com/bokeh/bokeh/blob/branch-2.3/sphinx/source/bokeh/scripts.html

Regarding searching again, this is another good reason to go to https://docs.bokeh.org as the first step and then utilize the site search from the front page. The site search results are locked to the version you are viewing, so you will never get mis-directed to different versions.

I don’t mean to harp on the site search, but there are so many different ways that different people approach documentation ^[1] that there will never be a single information architecture that “works” for everyone simultaneously. But I think site search a shortcut around the site organization that can work for everyone, which is why I keep bringing it up as something to promote more or better.

Thanks for the kind words and continued discussion (and for reaching out to Numpy!)

1. As a specific example: we had complaints both that the reference docs were not complete enough because they did not include base class members in subclasses, and then once we started doing that, complaints that the reference docs are now too bloated and hard to find a needle in a haystack. There is truly never going to be a site organization that will satisfy everyone. ↩︎

The Numpy devs responded rather quickly. Per Remove numpy documentation from google searches · Issue #39 · scipy/docs.scipy.org · GitHub, the issue was fixed by modifying their robots.txt file. Another person mentioned adding a “canonical” link to the pages with <link rel="canonical" href="https://docs.bokeh.org/en/latest/{MODULE}/{SUBMODULE}/{FUNCTION}.html" />. Not sure if that’s helpful…

Unfortunately we already have the canonical links in place (and have for a long time). E.g. if you look at the page source for

Plotting with Basic Glyphs — Bokeh 1.4.0 documentation

There is

<link rel="canonical" href="https://docs.bokeh.org/en/latest/docs/user_guide/plotting.html" />

in the document <head> We don’t have a robots.txt (or rather, it is empty) so we are not putting up any barriers to any crawlers. I have even on occasion kicked off new crawls explicitly from the search console tool for the property. So I am not really sure what else we can/should do.

I guess we could consider engaging project funds to pay someone to sort it out, but I am not sure how to find such a person, or whether this kind of work would fit in our meager budget (our entire account balance is a few thousand USD)

Edit: and looking that the search console tool, Google shows that there were crawls in the last week, and also shows that 11k user-specified canonical pages were understood. We publish an updated sitemap with every docs push too. AFAIK we are doing everything we are supposed to do

For NumPy, the robots.txt file is only for the old dcumentation site under scipy.org. The new site https://numpy.org/doc/1.17/user/c-info.html does not have a robots.txt. It does have a snippet of javascript that provides the red text at the top of older versions, which searches the “stable” docs for the matching doc there. If it finds it, the link is shown. If not the javscript links to the search page, and pre-seeds it with the page title.
It also adds the canonical link to all pages. It was added in this PR.

That’s essentially the exact same setup we have (sorry if I sound frustrated—I am frustrated, though not with anyone here)

I’m late to the party here, but I’d like to make some points and make some requests.

First off, a big thank you to Bryan and the team for building Bokeh. Yes it has bugs, yes it has shortcomings, but it’s a huge help to me in my work and I’m very glad the project exists. My life would be much harder without it.

I’d like to contribute in two ways, but I don’t know how. Perhaps you could give me some guidance on the way forward?

I’m a regular user, so I know where the weaknesses are in the documentation. I know where examples are needed. What I don’t know is how to raise what I’ve found as an issue. When I find shortcomings, should I raise them as an issue in Github?

I occasionally build Bokeh-based apps for side-projects (meaning, I ‘own’ the IP and I can do what I like with it). I would be happy to submit projects for the gallery, but I don’t know what criteria you use for selection. I don’t know what kind of apps would be helpful either. If I had a list of what you’re looking for, I could build my next project using your guidelines and submit it to you. As a note, all of my side-projects are more detailed than anything currently in the gallery.

I would love to help you guys build Bokeh better, but I can’t commit time to code. I can commit time for demos and documentation (tutorials?), but I need clear guidance on what you want.