I made a quick example with panel but it should not be hard to port on pure bokeh:
raw code
import panel as pn
css = """
.btn-custom > .bk-btn-group > .bk-btn {
background-image: url(https://previews.123rf.com/images/hemantraval/hemantraval1206/hemantraval120600022/13975128-glassy-aqua-blue-next-icon-button.jpg);
background-repeat: no-repeat;
background-size: contain;
background-position: center;
}
"""
pn.extension(raw_css=[css])
pn.widgets.Button(name='', css_classes=['btn-custom'], width=50, height=50)