I am successfully using export_png() to generate png instances of bokeh graphs for roll up into video sequences via ffmpeg
i have two questions on my experience
Firstly
The following warning is given,
UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
warnings.warn('Selenium support for PhantomJS has been deprecated, please use headless ’
however, it was my understanding that PhantomJs via selenium is the only supported mechanism of Bokeh at this time, or is there an actual way to use Chrome with as standard a bokeh / selenium release?
Where should I be looking to resolve that Selenium usage? Happy to work through reasonable instructions if I can find them.
Secondly, frame generation is taking about 30 ms, but frame export_png() is taking ~1.5 sec
Where animations can easily be 1000s of frames this becomes a painful overhead.
i assume this is the substatiation of the headless webkit browser instance as an overhead on each frame.
Is there any way to use a single substantiation across multiple calls into export_png to keep this export primitive fast in multiple call use cases, or other tips on performance?
Thanks greatly for bokeh as a whole and consideration of my question.
It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.
So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.
Anyone know if there examples out there showing such usage?
Or even better establishing chrome as the webdriver rather than phantomjs
Hi I have experimented with a chrome driver. I just followed the instructions in the selenium docs for creating a chrome driver and passed that to get_screenshot_as_png. It did not quite work on a retina display, because the pile ratio was not getting set correctly, and only one quarter of the image was captured. However if you are on linux or any system where the pixel ratio is 1 then it might work out of the box today.
It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.
So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.
Anyone know if there examples out there showing such usage?
Or even better establishing chrome as the webdriver rather than phantomjs
Was digging in to see if I could get the phantomJS driver up first.
Unfortunately in the python code in the latest bokeh instal the create drvier function is a private api within export.py, so not quite as simple as I hoped to just call it directly
This is no doubt my lack of experience causing me to reinvent the wheel.
Indeed when I get to chrome, its all ubuntu boxes.
Do you happen to have your example lying around?
···
On Thursday, July 19, 2018 at 1:03:43 AM UTC-4, Bryan Van de ven wrote:
Hi I have experimented with a chrome driver. I just followed the instructions in the selenium docs for creating a chrome driver and passed that to get_screenshot_as_png. It did not quite work on a retina display, because the pile ratio was not getting set correctly, and only one quarter of the image was captured. However if you are on linux or any system where the pixel ratio is 1 then it might work out of the box today.
It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.
So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.
Anyone know if there examples out there showing such usage?
Or even better establishing chrome as the webdriver rather than phantomjs
–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Was digging in to see if I could get the phantomJS driver up first.
Unfortunately in the python code in the latest bokeh instal the create drvier function is a private api within export.py, so not quite as simple as I hoped to just call it directly
This is no doubt my lack of experience causing me to reinvent the wheel.
Indeed when I get to chrome, its all ubuntu boxes.
Do you happen to have your example lying around?
On Thursday, July 19, 2018 at 1:03:43 AM UTC-4, Bryan Van de ven wrote:
Hi I have experimented with a chrome driver. I just followed the instructions in the selenium docs for creating a chrome driver and passed that to get_screenshot_as_png. It did not quite work on a retina display, because the pile ratio was not getting set correctly, and only one quarter of the image was captured. However if you are on linux or any system where the pixel ratio is 1 then it might work out of the box today.
It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.
So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.
Anyone know if there examples out there showing such usage?
Or even better establishing chrome as the webdriver rather than phantomjs
–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
On Jul 18, 2018, at 22:03, Bryan Van de ven [email protected] wrote:
Hi I have experimented with a chrome driver. I just followed the instructions in the selenium docs for creating a chrome driver and passed that to get_screenshot_as_png. It did not quite work on a retina display, because the pile ratio was not getting set correctly, and only one quarter of the image was captured. However if you are on linux or any system where the pixel ratio is 1 then it might work out of the box today.
It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.
So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.
Anyone know if there examples out there showing such usage?
Or even better establishing chrome as the webdriver rather than phantomjs
I need to dig into what version of bokeh i have in my stack as the version of create driver is private but as you point out, that simply does not match the source code as per the link you provided.
Ok this is the change that moved it from private to public.
So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
Not at PC right now, so scratching around on phone.
So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
Not at PC right now, so scratching around on phone.
Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.
Bryan
···
On Jul 19, 2018, at 07:57, Bryan Van de ven <[email protected]> wrote:
Hi,
I'm not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:
So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
Not at PC right now, so scratching around on phone.
<details class='elided'>
<summary title='Show trimmed content'>···</summary>
On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:
> Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.
>
>
> Bryan
>
>
>
> > On Jul 19, 2018, at 07:57, Bryan Van de ven <[email protected]> wrote:
>
> >
> > Hi,
>
> >
> > I'm not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:
>
> >
> >
> > driver = webdriver.Chrome(...)
>
> >
> > p1 = figure(...)
>
> > get_screenshot_as_png(p1, driver=driver)
>
> >
> > p2 = figure(...)
>
> > get_screenshot_as_png(p2, driver=driver)
>
> >
> > Thanks,
>
> >
> > Bryan
> >
> >
> >> On Jul 18, 2018, at 22:44, Anthony Tod <[email protected]> wrote:
>
> >>
> >> Ok this is the change that moved it from private to public.
>
> >>
> >> [https://github.com/bokeh/bokeh/commit/dec89a73dbf20c7dc2a8358864981de3080c84ed](https://github.com/bokeh/bokeh/commit/dec89a73dbf20c7dc2a8358864981de3080c84ed)
>
> >>
> >> So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
>
> >>
> >> Not at PC right now, so scratching around on phone.
>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
>
> >> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
>
> >> To post to this group, send email to [email protected].
>
> >> To view this discussion on the web visit [https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io](https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io).
>
> >> For more options, visit [https://groups.google.com/a/continuum.io/d/optout](https://groups.google.com/a/continuum.io/d/optout).
>
> >
>
>
</details>
I'm glad to hear that's working and that there is a 73% reduction in time. I'd like to think there are some performance improvements still to make, but in truth I am somewhat skeptical that there is much else we can do on our end. However, if you do find ay way to reduce the times further please let us know.
On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:
Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.
Bryan
> On Jul 19, 2018, at 07:57, Bryan Van de ven <[email protected]> wrote:
>
> Hi,
>
> I'm not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:
>
>
> driver = webdriver.Chrome(...)
>
> p1 = figure(...)
> get_screenshot_as_png(p1, driver=driver)
>
> p2 = figure(...)
> get_screenshot_as_png(p2, driver=driver)
>
> Thanks,
>
> Bryan
>
>
>> On Jul 18, 2018, at 22:44, Anthony Tod <[email protected]> wrote:
>>
>> Ok this is the change that moved it from private to public.
>>
>> Silence deprecation warning from selenium.webdriver.phantomjs (#7780) · bokeh/bokeh@dec89a7 · GitHub
>>
>> So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
>>
>> Not at PC right now, so scratching around on phone.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
>> To post to this group, send email to bo...@continuum.io.
>> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io\.
>> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.
>
All this runtime is indeed in the selenium / webkit pipeline.
Will tickle at multithreading, but my expectation is a mutexed pipeline through selenium / webkit.
Will update if I succeed
···
On Thursday, July 19, 2018 at 1:35:16 PM UTC-4, Bryan Van de ven wrote:
I’m glad to hear that’s working and that there is a 73% reduction in time. I’d like to think there are some performance improvements still to make, but in truth I am somewhat skeptical that there is much else we can do on our end. However, if you do find ay way to reduce the times further please let us know.
On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:
Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.
Bryan
On Jul 19, 2018, at 07:57, Bryan Van de ven [email protected] wrote:
Hi,
I’m not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:
So I must be on an older version, it’s all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
Not at PC right now, so scratching around on phone.
I now have a concurrent.futures implementation capable of N worker processes, so that if my overall video has 1000 frames these can be divided into N chunks of work and farmed out to run in parrallel.
I am currently using N as 4.
I farm into 4 chunks rather than 1000 jobs in a queue, as it is important not to have to recreate the webdriver each time. Much better having four parrallel engines creating one webdriver each for lifetime and working through the 250 frames before destroy.
So full job tme to complete is now limited by the processor availability on the host, and I can assign a beefy VM if I really want the full video generation to run in as little time as possible.
i don’t get a true N times acceleration, but for example on my notebook with four cores, and N workers configured.
Will update with some A B timings
I do however have an oddity noticed, during development.
Sometimes, if I allow the system to run with no logging overhead, I can hit a random failure during a random save to png.
I believe this is coming from with the selenium, chrome pipeline, called via export_png(…) but I am at a loss how to hardend. If I run with logging enabled, it simply does not happen.
2018-07-22 12:28:02,299 - ERROR - Plugin ‘anime’ threw an exception on call to stop. Exception - Message: unknown error: Cannot read property ‘is_idle’ of undefined
On Thursday, July 19, 2018 at 4:14:49 PM UTC-4, Anthony Tod wrote:
All this runtime is indeed in the selenium / webkit pipeline.
Will tickle at multithreading, but my expectation is a mutexed pipeline through selenium / webkit.
Will update if I succeed
On Thursday, July 19, 2018 at 1:35:16 PM UTC-4, Bryan Van de ven wrote:
I’m glad to hear that’s working and that there is a 73% reduction in time. I’d like to think there are some performance improvements still to make, but in truth I am somewhat skeptical that there is much else we can do on our end. However, if you do find ay way to reduce the times further please let us know.
On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:
Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.
Bryan
On Jul 19, 2018, at 07:57, Bryan Van de ven [email protected] wrote:
Hi,
I’m not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:
So I must be on an older version, it’s all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
Not at PC right now, so scratching around on phone.
I am not seeing benefit above 3 workers as my 4 core notebook is running max CPU at that point. Will test on an 8 core VM later today.
The is_idle crash problem is nasty, it can occur in any frame during processing, and only if the system is not loaded.
So on a single worker with no logging it is very likely to occur.
For multiple workers it will only occur sometimes in final frames when one of the workers is finished.
If I have logging enabled, it does not occur
···
On Sunday, July 22, 2018 at 12:49:34 PM UTC-4, Anthony Tod wrote:
I now have a concurrent.futures implementation capable of N worker processes, so that if my overall video has 1000 frames these can be divided into N chunks of work and farmed out to run in parrallel.
I am currently using N as 4.
I farm into 4 chunks rather than 1000 jobs in a queue, as it is important not to have to recreate the webdriver each time. Much better having four parrallel engines creating one webdriver each for lifetime and working through the 250 frames before destroy.
So full job tme to complete is now limited by the processor availability on the host, and I can assign a beefy VM if I really want the full video generation to run in as little time as possible.
i don’t get a true N times acceleration, but for example on my notebook with four cores, and N workers configured.
Will update with some A B timings
I do however have an oddity noticed, during development.
Sometimes, if I allow the system to run with no logging overhead, I can hit a random failure during a random save to png.
I believe this is coming from with the selenium, chrome pipeline, called via export_png(…) but I am at a loss how to hardend. If I run with logging enabled, it simply does not happen.
2018-07-22 12:28:02,299 - ERROR - Plugin ‘anime’ threw an exception on call to stop. Exception - Message: unknown error: Cannot read property ‘is_idle’ of undefined
On Thursday, July 19, 2018 at 4:14:49 PM UTC-4, Anthony Tod wrote:
All this runtime is indeed in the selenium / webkit pipeline.
Will tickle at multithreading, but my expectation is a mutexed pipeline through selenium / webkit.
Will update if I succeed
On Thursday, July 19, 2018 at 1:35:16 PM UTC-4, Bryan Van de ven wrote:
I’m glad to hear that’s working and that there is a 73% reduction in time. I’d like to think there are some performance improvements still to make, but in truth I am somewhat skeptical that there is much else we can do on our end. However, if you do find ay way to reduce the times further please let us know.
On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:
Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.
Bryan
On Jul 19, 2018, at 07:57, Bryan Van de ven [email protected] wrote:
Hi,
I’m not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:
So I must be on an older version, it’s all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
Not at PC right now, so scratching around on phone.
