So what’s happening here is that the first time you click the button, it calls the plot_change callback, which updates the text of callback_holder to be “Alert!!!”; this is the condition that fires the CustomJS callback, and so you see your alert. But once you’ve done that, every subsequent time it sets the text as “Alert!!!” again; this doesn’t count as a text change, so you’ll never see that popup again.
I realize this is probably a minimal example, and there may be context I don’t know about, but strictly for what you’re trying to do here, it could be simpler:
from bokeh.io import curdoc
from bokeh.models.widgets import Button
from bokeh.layouts import column
from bokeh.models.callbacks import CustomJS
button_classify = Button(label="Create Pop-up alert")
callback = CustomJS(args={}, code='alert("hello!");')
button_classify.js_on_click(callback)
layout = column(button_classify)
curdoc().add_root(layout)
curdoc().title = "Pop-up Alert"