export_png warning and performance question

On Jul 18, 2018, at 21:52, Anthony Tod <anthony.tod@gmail.com> wrote:

I have been looking at underlying code in

bokeh.io.export — Bokeh 3.3.2 Documentation

It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.

So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.

Anyone know if there examples out there showing such usage?

Or even better establishing chrome as the webdriver rather than phantomjs

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+unsubscribe@continuum.io.
To post to this group, send email to bokeh@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/22428aeb-db18-48a7-9efe-6b8354a35784%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

On Thursday, July 19, 2018 at 1:03:43 AM UTC-4, Bryan Van de ven wrote:

Hi I have experimented with a chrome driver. I just followed the instructions in the selenium docs for creating a chrome driver and passed that to get_screenshot_as_png. It did not quite work on a retina display, because the pile ratio was not getting set correctly, and only one quarter of the image was captured. However if you are on linux or any system where the pixel ratio is 1 then it might work out of the box today.

Bryan

On Jul 18, 2018, at 21:52, Anthony Tod antho...@gmail.com wrote:

I have been looking at underlying code in

https://bokeh.pydata.org/en/latest/_modules/bokeh/io/export.html

It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.

So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.

Anyone know if there examples out there showing such usage?

Or even better establishing chrome as the webdriver rather than phantomjs

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.

To post to this group, send email to bo...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/22428aeb-db18-48a7-9efe-6b8354a35784%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

On Jul 18, 2018, at 22:12, Anthony Tod anthony.tod@gmail.com wrote:

Thanks for the response.

Was digging in to see if I could get the phantomJS driver up first.

Unfortunately in the python code in the latest bokeh instal the create drvier function is a private api within export.py, so not quite as simple as I hoped to just call it directly

This is no doubt my lack of experience causing me to reinvent the wheel.

Indeed when I get to chrome, its all ubuntu boxes.

Do you happen to have your example lying around?

On Thursday, July 19, 2018 at 1:03:43 AM UTC-4, Bryan Van de ven wrote:

Hi I have experimented with a chrome driver. I just followed the instructions in the selenium docs for creating a chrome driver and passed that to get_screenshot_as_png. It did not quite work on a retina display, because the pile ratio was not getting set correctly, and only one quarter of the image was captured. However if you are on linux or any system where the pixel ratio is 1 then it might work out of the box today.

Bryan

On Jul 18, 2018, at 21:52, Anthony Tod antho...@gmail.com wrote:

I have been looking at underlying code in

https://bokeh.pydata.org/en/latest/_modules/bokeh/io/export.html

It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.

So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.

Anyone know if there examples out there showing such usage?

Or even better establishing chrome as the webdriver rather than phantomjs

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.

To post to this group, send email to bo...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/22428aeb-db18-48a7-9efe-6b8354a35784%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

–

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+unsubscribe@continuum.io.

To post to this group, send email to bokeh@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/82978b0d-8696-4024-b535-fea7069814f0%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

On Jul 18, 2018, at 22:03, Bryan Van de ven bryanv@anaconda.com wrote:

Hi I have experimented with a chrome driver. I just followed the instructions in the selenium docs for creating a chrome driver and passed that to get_screenshot_as_png. It did not quite work on a retina display, because the pile ratio was not getting set correctly, and only one quarter of the image was captured. However if you are on linux or any system where the pixel ratio is 1 then it might work out of the box today.

Bryan

On Jul 18, 2018, at 21:52, Anthony Tod anthony.tod@gmail.com wrote:

I have been looking at underlying code in

https://bokeh.pydata.org/en/latest/_modules/bokeh/io/export.html

It looks like a create destroy lifecycle is enforced if I dont pass in a web driver.

So I can likely call the creation code for the web driver that is run when no parameter is passed create_webdriver() and create a sustained instance from outside my frame loop, then hopefully just pass that driver reference in, and I would then get no create / destroy overhead.

Anyone know if there examples out there showing such usage?

Or even better establishing chrome as the webdriver rather than phantomjs

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+unsubscribe@continuum.io.
To post to this group, send email to bokeh@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/22428aeb-db18-48a7-9efe-6b8354a35784%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

On Jul 18, 2018, at 22:44, Anthony Tod <anthony.tod@gmail.com> wrote:

Ok this is the change that moved it from private to public.

Silence deprecation warning from selenium.webdriver.phantomjs (#7780) · bokeh/bokeh@dec89a7 · GitHub

So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.

Not at PC right now, so scratching around on phone.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+unsubscribe@continuum.io.
To post to this group, send email to bokeh@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

On Jul 19, 2018, at 07:57, Bryan Van de ven <bryan@bokeh.org> wrote:

Hi,

I'm not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:

   driver = webdriver.Chrome(...)

   p1 = figure(...)
   get_screenshot_as_png(p1, driver=driver)

   p2 = figure(...)
   get_screenshot_as_png(p2, driver=driver)

Thanks,

Bryan

On Jul 18, 2018, at 22:44, Anthony Tod <anthony.tod@gmail.com> wrote:

Ok this is the change that moved it from private to public.

Silence deprecation warning from selenium.webdriver.phantomjs (#7780) · bokeh/bokeh@dec89a7 · GitHub

So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.

Not at PC right now, so scratching around on phone.

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+unsubscribe@continuum.io.
To post to this group, send email to bokeh@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

On Jul 19, 2018, at 10:17, Anthony Tod <anthony.tod@gmail.com> wrote:

all working now against google chrome headerless, seeing 400 ms frame export rather than 1.5 sec, so good enough first pass improvement.

had to grab an instance fo chromedriver as per

ChromeDriver - WebDriver for Chrome - Downloads

placed it in /usr/local/share for now, will deal with that further when I go to full stack intergration.

then in my code...

from selenium import webdriver
...
options = webdriver.ChromeOptions()
options.binary_location = find_executable('google-chrome')
options.add_argument('headless')
self.webdriver = webdriver.Chrome(executable_path=find_executable('chromedriver'),
                                  chrome_options=options)
...
export_png(self.p_base, filename=full_path, webdriver=self.webdriver)

On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:
Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.

Bryan

> On Jul 19, 2018, at 07:57, Bryan Van de ven <bry...@anaconda.com> wrote:
>
> Hi,
>
> I'm not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:
>
>
> driver = webdriver.Chrome(...)
>
> p1 = figure(...)
> get_screenshot_as_png(p1, driver=driver)
>
> p2 = figure(...)
> get_screenshot_as_png(p2, driver=driver)
>
> Thanks,
>
> Bryan
>
>
>> On Jul 18, 2018, at 22:44, Anthony Tod <antho...@gmail.com> wrote:
>>
>> Ok this is the change that moved it from private to public.
>>
>> Silence deprecation warning from selenium.webdriver.phantomjs (#7780) · bokeh/bokeh@dec89a7 · GitHub
>>
>> So I must be on an older version, it's all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.
>>
>> Not at PC right now, so scratching around on phone.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
>> To post to this group, send email to bo...@continuum.io.
>> To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io\.
>> For more options, visit https://groups.google.com/a/continuum.io/d/optout\.
>

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+unsubscribe@continuum.io.
To post to this group, send email to bokeh@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/7d701c39-30d4-4470-9efc-a83aaae05dff%40continuum.io\.
For more options, visit https://groups.google.com/a/continuum.io/d/optout\.

On Thursday, July 19, 2018 at 1:35:16 PM UTC-4, Bryan Van de ven wrote:

I’m glad to hear that’s working and that there is a 73% reduction in time. I’d like to think there are some performance improvements still to make, but in truth I am somewhat skeptical that there is much else we can do on our end. However, if you do find ay way to reduce the times further please let us know.

Thanks,

Bryan

On Jul 19, 2018, at 10:17, Anthony Tod antho...@gmail.com wrote:

all working now against google chrome headerless, seeing 400 ms frame export rather than 1.5 sec, so good enough first pass improvement.

had to grab an instance fo chromedriver as per

http://chromedriver.chromium.org/downloads

placed it in /usr/local/share for now, will deal with that further when I go to full stack intergration.

then in my code…

from selenium import webdriver

…

options = webdriver.ChromeOptions()

options.binary_location = find_executable(‘google-chrome’)

options.add_argument(‘headless’)

self.webdriver = webdriver.Chrome(executable_path=find_executable(‘chromedriver’),

                              chrome_options=options)

…

export_png(self.p_base, filename=full_path, webdriver=self.webdriver)

On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:

Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.

Bryan

On Jul 19, 2018, at 07:57, Bryan Van de ven bry...@anaconda.com wrote:

Hi,

I’m not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:

driver = webdriver.Chrome(…)

p1 = figure(…)
get_screenshot_as_png(p1, driver=driver)

p2 = figure(…)
get_screenshot_as_png(p2, driver=driver)

Thanks,

Bryan

On Jul 18, 2018, at 22:44, Anthony Tod antho...@gmail.com wrote:

Ok this is the change that moved it from private to public.

https://github.com/bokeh/bokeh/commit/dec89a73dbf20c7dc2a8358864981de3080c84ed

So I must be on an older version, it’s all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.

Not at PC right now, so scratching around on phone.

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
To post to this group, send email to bo...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.

To post to this group, send email to bo...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/7d701c39-30d4-4470-9efc-a83aaae05dff%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

On Thursday, July 19, 2018 at 4:14:49 PM UTC-4, Anthony Tod wrote:

All this runtime is indeed in the selenium / webkit pipeline.

Will tickle at multithreading, but my expectation is a mutexed pipeline through selenium / webkit.

Will update if I succeed

On Thursday, July 19, 2018 at 1:35:16 PM UTC-4, Bryan Van de ven wrote:

I’m glad to hear that’s working and that there is a 73% reduction in time. I’d like to think there are some performance improvements still to make, but in truth I am somewhat skeptical that there is much else we can do on our end. However, if you do find ay way to reduce the times further please let us know.

Thanks,

Bryan

On Jul 19, 2018, at 10:17, Anthony Tod antho...@gmail.com wrote:

all working now against google chrome headerless, seeing 400 ms frame export rather than 1.5 sec, so good enough first pass improvement.

had to grab an instance fo chromedriver as per

http://chromedriver.chromium.org/downloads

placed it in /usr/local/share for now, will deal with that further when I go to full stack intergration.

then in my code…

from selenium import webdriver

…

options = webdriver.ChromeOptions()

options.binary_location = find_executable(‘google-chrome’)

options.add_argument(‘headless’)

self.webdriver = webdriver.Chrome(executable_path=find_executable(‘chromedriver’),

                              chrome_options=options)

…

export_png(self.p_base, filename=full_path, webdriver=self.webdriver)

On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:

Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.

Bryan

On Jul 19, 2018, at 07:57, Bryan Van de ven bry...@anaconda.com wrote:

Hi,

I’m not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:

driver = webdriver.Chrome(…)

p1 = figure(…)
get_screenshot_as_png(p1, driver=driver)

p2 = figure(…)
get_screenshot_as_png(p2, driver=driver)

Thanks,

Bryan

On Jul 18, 2018, at 22:44, Anthony Tod antho...@gmail.com wrote:

Ok this is the change that moved it from private to public.

https://github.com/bokeh/bokeh/commit/dec89a73dbf20c7dc2a8358864981de3080c84ed

So I must be on an older version, it’s all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.

Not at PC right now, so scratching around on phone.

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
To post to this group, send email to bo...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.

To post to this group, send email to bo...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/7d701c39-30d4-4470-9efc-a83aaae05dff%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

On Sunday, July 22, 2018 at 12:49:34 PM UTC-4, Anthony Tod wrote:

I now have a concurrent.futures implementation capable of N worker processes, so that if my overall video has 1000 frames these can be divided into N chunks of work and farmed out to run in parrallel.
I am currently using N as 4.
I farm into 4 chunks rather than 1000 jobs in a queue, as it is important not to have to recreate the webdriver each time. Much better having four parrallel engines creating one webdriver each for lifetime and working through the 250 frames before destroy.

So full job tme to complete is now limited by the processor availability on the host, and I can assign a beefy VM if I really want the full video generation to run in as little time as possible.

i don’t get a true N times acceleration, but for example on my notebook with four cores, and N workers configured.

Will update with some A B timings

I do however have an oddity noticed, during development.

Sometimes, if I allow the system to run with no logging overhead, I can hit a random failure during a random save to png.

I believe this is coming from with the selenium, chrome pipeline, called via export_png(…) but I am at a loss how to hardend. If I run with logging enabled, it simply does not happen.

2018-07-22 12:28:02,299 - ERROR - Plugin ‘anime’ threw an exception on call to stop. Exception - Message: unknown error: Cannot read property ‘is_idle’ of undefined

(Session info: headless chrome=67.0.3396.99)

(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee976b2c3e3b9f335cde7),platform=Linux 4.4.0-130-generic x86_64)

On Thursday, July 19, 2018 at 4:14:49 PM UTC-4, Anthony Tod wrote:

All this runtime is indeed in the selenium / webkit pipeline.

Will tickle at multithreading, but my expectation is a mutexed pipeline through selenium / webkit.

Will update if I succeed

On Thursday, July 19, 2018 at 1:35:16 PM UTC-4, Bryan Van de ven wrote:

I’m glad to hear that’s working and that there is a 73% reduction in time. I’d like to think there are some performance improvements still to make, but in truth I am somewhat skeptical that there is much else we can do on our end. However, if you do find ay way to reduce the times further please let us know.

Thanks,

Bryan

On Jul 19, 2018, at 10:17, Anthony Tod antho...@gmail.com wrote:

all working now against google chrome headerless, seeing 400 ms frame export rather than 1.5 sec, so good enough first pass improvement.

had to grab an instance fo chromedriver as per

http://chromedriver.chromium.org/downloads

placed it in /usr/local/share for now, will deal with that further when I go to full stack intergration.

then in my code…

from selenium import webdriver

…

options = webdriver.ChromeOptions()

options.binary_location = find_executable(‘google-chrome’)

options.add_argument(‘headless’)

self.webdriver = webdriver.Chrome(executable_path=find_executable(‘chromedriver’),

                              chrome_options=options)

…

export_png(self.p_base, filename=full_path, webdriver=self.webdriver)

On Thursday, July 19, 2018 at 10:58:42 AM UTC-4, Bryan Van de ven wrote:

Unless you are referring to fixing any issues with pixel ratio, that would require digging into the library code. But just using existing APIs should not.

Bryan

On Jul 19, 2018, at 07:57, Bryan Van de ven bry...@anaconda.com wrote:

Hi,

I’m not sure you might not be overthinking things. The code to re-use your own driver should not be much more complicated than:

driver = webdriver.Chrome(…)

p1 = figure(…)
get_screenshot_as_png(p1, driver=driver)

p2 = figure(…)
get_screenshot_as_png(p2, driver=driver)

Thanks,

Bryan

On Jul 18, 2018, at 22:44, Anthony Tod antho...@gmail.com wrote:

Ok this is the change that moved it from private to public.

https://github.com/bokeh/bokeh/commit/dec89a73dbf20c7dc2a8358864981de3080c84ed

So I must be on an older version, it’s all packaged controlled against a deep stack, so I just have some do some digging tomorrow to get the bokeh package moved forward.

Not at PC right now, so scratching around on phone.

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.
To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.
To post to this group, send email to bo...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/df1affcf-205c-4643-9a99-0711c1bf189b%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

–
You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to bokeh+un...@continuum.io.

To post to this group, send email to bo...@continuum.io.

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/7d701c39-30d4-4470-9efc-a83aaae05dff%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.