Hi all,
I’m relatively new to this and have never asked for help on an issue from an open source project. I think that this might be a bug report but I thought I’d start here. (I have text formatting issues down below. Sorry if it’s annoying.)
I’m interested in loading data from a pandas dataframe into a bokeh DataTable widget. The data includes a column of dates. I know that the dates are all relative to UTC, and I want them to be loaded into the DataTable exactly as they are, but bokeh will convert them to my local timezone. This was the topic of two different Issues (i.e. bug reports) #4139 and #4223. In #4223 there is code that shows that bokeh should be sending dates to DataTables without adjusting for the local timezone. This works fine for the dates when they are in the form of a datetime.date object. However, if the dates are datetime64 (as used in pandas) the timezone conversion still seems to occur.
Before providing code, here are the versions I’m using:
Python version : 3.5.1 |Anaconda 2.4.1 (64-bit)| (default, Feb 16 2016, 09:49:46) [MSC v.1900 64 bit (AMD64)]
IPython version : 4.0.1
Bokeh version : 0.12.1
pandas : 0.17.1
Firstly, the code in Issue #4223 works as expected- no timezone conversion when data is sent to the DataTable. It uses a dict() structure and the datetime.date() function to create a list of dates. It does it like this:
from datetime import date
from random import randint
data = dict(
dates=[date(2014, 3, i+1) for i in range(10)],
downloads=[randint(0, 100) for i in range(10)],
)
However, I’m interested in using pandas and its datetime64 datatype. So instead of using a dict() structure to define the data, I will use a pandas dataframe.
from datetime import date
from random import randint
import pandas as pd
from bokeh.models import ColumnDataSource
from bokeh.models.widgets import DataTable, DateFormatter, TableColumn
from bokeh.io import output_file, show
output_file('data_table.html')
#Define the data
datadf = pd.DataFrame({'dates': [date(2014, 3, i+1) for i in range(10)], \
'downloads':[randint(0, 100) for i in range(10)]})
#Convert the datetime.date to datetime64
datadf['dates'] = pd.to_datetime(datadf['dates'])
source = ColumnDataSource(datadf)
#define the columns and the DataTable
columns = [
TableColumn(field="dates", title="Date", formatter=DateFormatter()),
TableColumn(field="downloads", title="Downloads"),
]
data_table = DataTable(source=source, columns=columns, width=400, height=280)
show(data_table)
<img src="//bokeh-discourse-uploads.s3.dualstack.us-east-1.amazonaws.com/original/1X/3309b63de4ce4fb2a288b393687d22d1acf265b9.png" alt="">
The resulting DataTable has Feb 28, 2014 as the first date element NOT March 1 2014. This is how the timezone conversion manifests itself. If you look at the dataframe (datadf),
it has March 1, 2014 as the first element, so I think the issue is in bokeh.
I also tried defining the data structure using the dict() as in the code from Issue #4223, but I converted the datetime.date into a numpy datetime64
#All other imports are the same
import numpy as np
data = dict(
dates=[np.datetime64(date(2014, 3, i+1)) for i in range(10)],
downloads=[randint(0, 100) for i in range(10)],
)
source = ColumnDataSource(data)
columns = [
TableColumn(field="dates", title="Date", formatter=DateFormatter()),
TableColumn(field="downloads", title="Downloads"),
]
data_table = DataTable(source=source, columns=columns)
show(data_table)
This also resulted in Feb 28, 2014 in the first date cell which seems to indicate that there is an issue with how bokeh handles that datatype.
My work-around is to convert all the dates in the pandas dataframes to datetime.date. This works fine, but it seems like I should be able to load datetime64's "as-is" into the
DataTable.
Thanks for any help!!
Mark