I had issues with complex layouts with many plots a few years ago in scenarios where I did not expect redraw operations to occur.
See my following discourse topic and the solution I implemented to workaround it as the last post in the thread. No guarantee it will address your particular use-case, but it is at least passingly relevant.