Plotting large rasters on map

Hello everybody,

I am new to the group and to Bokeh, so I apologize in advance for the newbie question. I’ve been using Matplotlib basemap to plot rasters (e.g. temperature data) over vector maps for a while. However, I am hitting some memory issues now that I am handling very large and more detailed rasters. Does Bokeh have a similar functionality? If so, would it handle very large rasters better than basemap? A code snippet would be greatly appreciated!

Thanks,

George

Hi George,

Can you be more specific about data sizes, etc, as well as what you have tried? There are various different possibilities, from using your own tile source with bokeh's tile renderers, to a Bokeh app that does downsampling based on the current viewable range, to a standalone plot with an AjazDataSource if the data happens to be available via some REST API, to the separate bokeh/datashader project if you have, e.g. billions of points to visualize.

Bryan

···

On Feb 20, 2016, at 3:24 PM, George Azzari <[email protected]> wrote:

Hello everybody,

I am new to the group and to Bokeh, so I apologize in advance for the newbie question. I've been using Matplotlib basemap to plot rasters (e.g. temperature data) over vector maps for a while. However, I am hitting some memory issues now that I am handling very large and more detailed rasters. Does Bokeh have a similar functionality? If so, would it handle very large rasters better than basemap? A code snippet would be greatly appreciated!

Thanks,
George

--
You received this message because you are subscribed to the Google Groups "Bokeh Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/21a75768-d99d-47f4-b41e-6175669baae8%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Hi Bryan,

Apologies for my delayed reply, and many thanks for your response. My final goal is to visualize on a map a bunch of geotiffs that extent over the entire US corn belt. Such rasters are generated by state, and their size varies on a yearly basis; for example, Illinois has an average of 342 million points @ 30m resolution for one year (~ 140 MB). It is very likely that the overall number of points will come very close to a billion in some years. I am interested in both dynamic visualizations for our lab website (perhaps the downsampling app or the bokeh/datashader project you mentioned?), and static plots for journal publications.

So far I only had to deal with smaller datasets, so I’ve been using Matplotlib’s basemap for a while. Unfortunately basemap does not seem to deal well with so much data, and that’s what brought me here. I have not tried anything on Bokeh yet, I was first curious to know if Bokeh deals well with large datasets. The options you laid out seem all very interesting; if you could point me to some docs and/or code examples I’d be very grateful :slight_smile:

Cheers,

George

···

On Thu, Mar 3, 2016 at 5:50 PM, Bryan Van de Ven [email protected] wrote:

Hi George,

Can you be more specific about data sizes, etc, as well as what you have tried? There are various different possibilities, from using your own tile source with bokeh’s tile renderers, to a Bokeh app that does downsampling based on the current viewable range, to a standalone plot with an AjazDataSource if the data happens to be available via some REST API, to the separate bokeh/datashader project if you have, e.g. billions of points to visualize.

Bryan

On Feb 20, 2016, at 3:24 PM, George Azzari [email protected] wrote:

Hello everybody,

I am new to the group and to Bokeh, so I apologize in advance for the newbie question. I’ve been using Matplotlib basemap to plot rasters (e.g. temperature data) over vector maps for a while. However, I am hitting some memory issues now that I am handling very large and more detailed rasters. Does Bokeh have a similar functionality? If so, would it handle very large rasters better than basemap? A code snippet would be greatly appreciated!

Thanks,

George

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/21a75768-d99d-47f4-b41e-6175669baae8%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4952DA5D-73CC-428C-88A7-DAD2B5A9CFF0%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

George

Hi George,

It sounds like a fascinating project and, at first glance one that datashader would be ripe for.

Check out the docs which have links to an intro video and examples on the home page: http://datashader.readthedocs.org/en/latest/

Datashader currently only works with points and lines but i’m guessing u could transform geotiff data in pointwise data.

Sarah Bird
[email protected]

···

On Thu, Mar 3, 2016 at 5:50 PM, Bryan Van de Ven [email protected] wrote:

Hi George,

Can you be more specific about data sizes, etc, as well as what you have tried? There are various different possibilities, from using your own tile source with bokeh’s tile renderers, to a Bokeh app that does downsampling based on the current viewable range, to a standalone plot with an AjazDataSource if the data happens to be available via some REST API, to the separate bokeh/datashader project if you have, e.g. billions of points to visualize.

Bryan

On Feb 20, 2016, at 3:24 PM, George Azzari [email protected] wrote:

Hello everybody,

I am new to the group and to Bokeh, so I apologize in advance for the newbie question. I’ve been using Matplotlib basemap to plot rasters (e.g. temperature data) over vector maps for a while. However, I am hitting some memory issues now that I am handling very large and more detailed rasters. Does Bokeh have a similar functionality? If so, would it handle very large rasters better than basemap? A code snippet would be greatly appreciated!

Thanks,

George

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/21a75768-d99d-47f4-b41e-6175669baae8%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/4952DA5D-73CC-428C-88A7-DAD2B5A9CFF0%40continuum.io.
For more options, visit https://groups.google.com/a/continuum.io/d/optout.

George

HI George,
As Sarah mentioned datashader would be perfect for you . I have also been using it to plot points which were not possible using matplotlib.

Since you mentioned temp data you probably can reduce the resolution of your data to a Km from 30m and reduce data size by 1000x.

This operation should be very easy using pandas/dask.

Sameer

···

Hi Sarah,

Thanks for the pointers; I just watched the video you shared, and I am looking forward to test out datashader for this project. I’ll keep you guys posted on how it goes.

Cheers,

George

···

On Wed, Mar 16, 2016 at 11:18 AM, Sam [email protected] wrote:

HI George,
As Sarah mentioned datashader would be perfect for you . I have also been using it to plot points which were not possible using matplotlib.

Since you mentioned temp data you probably can reduce the resolution of your data to a Km from 30m and reduce data size by 1000x.

This operation should be very easy using pandas/dask.

Sameer

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/abe96ebc-7a15-476b-9f04-24333f26d6c0%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

Hi Sameer,

Thanks for your suggestion; I agree with you, for certain variables (such as temperature) downscaling raster data is probably the best visualization option. I think though that in my specific case datashader could be a better option, as we are also testing ways to expose large scale crop yield maps with some very interesting fine-scale variability.

Cheers,

George

···

On Wed, Mar 16, 2016 at 11:18 AM, Sam [email protected] wrote:

HI George,
As Sarah mentioned datashader would be perfect for you . I have also been using it to plot points which were not possible using matplotlib.

Since you mentioned temp data you probably can reduce the resolution of your data to a Km from 30m and reduce data size by 1000x.

This operation should be very easy using pandas/dask.

Sameer

You received this message because you are subscribed to the Google Groups “Bokeh Discussion - Public” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/bokeh/abe96ebc-7a15-476b-9f04-24333f26d6c0%40continuum.io.

For more options, visit https://groups.google.com/a/continuum.io/d/optout.

George