The modify_doc code is only executed once, when the session is created. Once the session is displayed in a browser, the only code that is subsequently executed in the callbacks that you set up initially. It follows, then, that for the call to remove_callback to have any effect, it needs to be in a callback:
def update():
global index
if index == N:
doc.remove_periodic_callback(callback_id)
return
new_data = dict(x=[X[index]], y=[Y[index]])
ypts.stream(new_data)
index += 1
callback_id = doc.add_periodic_callback(update, 40)
This is the same structure (remove inside the callback) as the Gaminder example you initially reference.
As an aside: I’m not trying to harp on the complete, minimal repropducer thing, but there is literally nothing better that users can do to help others, help them (this answer was evident 10 seconds after actually running the complete code you posted on SO—if you’d posted the complete code I asked for here two days ago, I would have had you an answer two days ago)