Plot with rect and multiple color mapping

I’m trying to create a heatmap-like plot where every column has its own color mapping.

It works fine when I use numeric range for the x-axis (fig1), but if I try to use the factor which would be the more intuitive way (fig2) it won’t plot anything. Is there a reason for it?

I also tried the with CategoricalColorMapper and ColorSpec but the same occurs, so I guess is not related to the coloring itself. Follow minimal example:

from bokeh.io import show
from bokeh.plotting import figure
from bokeh.palettes import Category20
from bokeh.layouts import layout
from bokeh.transform import factor_cmap

data = {"index": ["row1","row2","row3","row4","row5"],
        "colA": ["a","aa","aaa","aaa","aaa"], 
        "colB": ["b","bb","bbb","bbbb","bbbb"], 
        "colC": ["c","cc","cccc","ccccc","cccccc"]}

data_rows = data["index"]
data_cols = list(data.keys())[1:]

fig1 = figure(x_range=data_cols, y_range=data_rows)
fig2 = figure(x_range=data_cols, y_range=data_rows)

for i,col in enumerate(data_cols):
	unique_factors = list(set(data[col]))

	fig1.rect(x=i+.5, y="index", 
		width=1, height=1,
		source=data,
		fill_color=factor_cmap(field_name=col, palette=Category20[len(unique_factors)], factors=unique_factors))

	fig2.rect(x=col, y="index",
		width=1, height=1,
		source=data,
		fill_color=factor_cmap(field_name=col, palette=Category20[len(unique_factors)], factors=unique_factors))

show(layout([fig1,fig2]))