Are you using components or otherwise have a template that explicitly loads BokehJS? The most likely explanation is that you are loading a too-old version of BokehJS and need to update it to match the python version you are using now. Otherwise, it is not possible to speculate more without a complete Minimal Reproducible Example to actually investigate.
However, I see this section has fallen away in the latest documentation (2.4.3). I do hope support for LatexLabel will continue or that it will be replaced with the ability to use latex in Label as is the case with axis labels.
And will be supported in even more places (e.g. built in Label) will be present in Bokeh 3.0. That section was removed because we do not want to encourage any users to go down the complicated (and fragile) custom extension path if it is not necessary.