Bokeh image format standards and interoperability

Hello,

I’m trying to get small image icons (with transparency) to rotate about their image x,y centers dynamically in a bokeh serve periodic callback. I’d like to use an outside package like opencv or imutils.rotate_bound(img, angle) or similar but cannot find a way to access the bokeh image in a standard format.

I have no preference on image methods and can use image(), image_url(), or image_rgba().

This post from bigreddot explains to get from cv2 to bokeh with image_rgba() but it’s not entirely clear what standard Bokeh uses.

Do bokeh images adhere to a certain standard with .to() and .from() methods to OpenCV or any other format?

Marc Compere

From the linked SO post:

What Bokeh expects is a Numpy array of shape (M, N) 32-bit integers representing RGBA values.

Just to add more: there is nothing built in to Bokeh to generate those 32 bit RGBA arrays from any other formats or sources. Seems like it could be a reasonable feature to add some convenience functions for those sorts of conversions (behind optional dependencies).

1 Like

Maybe this will be useful:

from base64 import b64decode

import cv2
import numpy as np
from bokeh.io import curdoc
from bokeh.layouts import column
from bokeh.models import ColumnDataSource, Button
from bokeh.plotting import figure

img_jpeg = b64decode(
    'iVBORw0KGgoAAAANSUhEUgAAADIAAAAOBAMAAACfqVJUAAAAHlBMVEUAAAAAAwCKLY3jF17xWyICq65Qlc8js1T2qhmnzTbxGUeMAAAAAXRSTlMAQObYZgAAAAFiS0dEAIgFHUgAAACVSURBVBjTpY89DsIwDIWfRRF087sB6sCMyhHIAbp078TMgroy9gq5LXYSooqOfYqsF3+Jf0BU6c9IcjvI6bUlJiDO78EJIXYVJRXptHEeRyP22PPC7Cy2EUcnKQOmkPrwHIHnpNQNWT5OvLi1oa7/LMB1ouZqqES0CUCXm6+Iz9aER9/5bKyYZZ9w7y91n0KKDjf86QvVrhIPSsvLCwAAAABJRU5ErkJggg==')
arr = np.fromstring(img_jpeg, np.uint8)
frame = cv2.imdecode(arr, cv2.IMREAD_UNCHANGED)
orig_img = frame[::-1].copy().view(dtype=np.uint32)

pw = ph = 100
current_angle = 0
p = figure(aspect_ratio=1, x_range=(0, pw), y_range=(0, ph))


def mk_data(image):
    ih, iw, _ = image.shape
    return dict(image=[image], dw=[iw], dh=[ih], x=[(pw - iw) / 2], y=[(ph - ih) / 2])


src = ColumnDataSource(data=mk_data(orig_img))
p.image_rgba(image='image', x='x', y='y', dw='dw', dh='dh', source=src)


# From https://github.com/jrosebr1/imutils/blob/master/imutils/convenience.py#L41.
def rotate_bound(image, angle):
    (h, w) = image.shape[:2]
    (cX, cY) = (w // 2, h // 2)
    M = cv2.getRotationMatrix2D((cX, cY), -angle, 1.0)
    cos = np.abs(M[0, 0])
    sin = np.abs(M[0, 1])
    nW = int((h * sin) + (w * cos))
    nH = int((h * cos) + (w * sin))
    M[0, 2] += (nW / 2) - cX
    M[1, 2] += (nH / 2) - cY
    return cv2.warpAffine(image, M, (nW, nH))


b = Button(label='Rotate!')


def rotate_img_by_15_deg():
    global current_angle
    current_angle += 15
    img = rotate_bound(orig_img.view(dtype=np.uint8), current_angle)
    src.data = mk_data(img.view(dtype=np.uint32))


b.on_click(rotate_img_by_15_deg)

curdoc().add_root(column(p, b))
3 Likes

@p-himik, indeed, it seems that there are a number of standard image formats and no single style is dominant.

It seems Bokeh uses something close to the “Truecolor and alpha” 4-channel, 32-bit PNG described in the table here.

Indeed, indeed, this example was quite useful. It explained a few things.

First, the approach for updating pixel-level data in the image using the ColumnDataSource is quite instructive. I’ve seen this in bits and pieces frequently but was unable to pull that example together with the image data. Other examples show how to change the image size dynamically but I had yet to see this.

Second, the mk_data() function is instructive because this was not entirely obvious that all images need these elements. In hindsight, it is clear and understandable but was not easy to puzzle together.

Third, the .view() method is quite simple for changing the datatypes with which the memory location is referenced like a mask or union, somewhat like a datatype cast in c.

I’ve updated your example with another button and a periodic callback.

Thanks for your help - this was extremely helpful.

My slightly longer version with an additional button to turn periodic callbacks on and off for continuous rotations. Both buttons work to increase the image rotation angle.

from base64 import b64decode
import sys

import cv2
import numpy as np
from bokeh.io import curdoc
from bokeh.layouts import column
from bokeh.models import ColumnDataSource, Button
from bokeh.plotting import figure

import imutils # rotate_bound()

T_update = 200 # (ms) browser update rate specified in add_periodic_callback()
current_angle = 0

img_jpeg = b64decode(
    'iVBORw0KGgoAAAANSUhEUgAAADIAAAAOBAMAAACfqVJUAAAAHlBMVEUAAAAAAwCKLY3jF17xWyICq65Qlc8js1T2qhmnzTbxGUeMAAAAAXRSTlMAQObYZgAAAAFiS0dEAIgFHUgAAACVSURBVBjTpY89DsIwDIWfRRF087sB6sCMyhHIAbp078TMgroy9gq5LXYSooqOfYqsF3+Jf0BU6c9IcjvI6bUlJiDO78EJIXYVJRXptHEeRyP22PPC7Cy2EUcnKQOmkPrwHIHnpNQNWT5OvLi1oa7/LMB1ouZqqES0CUCXm6+Iz9aER9/5bKyYZZ9w7y91n0KKDjf86QvVrhIPSsvLCwAAAABJRU5ErkJggg==')
arr = np.fromstring(img_jpeg, np.uint8)
frame = cv2.imdecode(arr, cv2.IMREAD_UNCHANGED)
orig_img = frame[::-1].copy().view(dtype=np.uint32)


pw = 5*orig_img.shape[0]
ph = 5*orig_img.shape[1]
current_angle = 0
p = figure(aspect_ratio=1, x_range=(0, pw), y_range=(0, ph))


def mk_data(image):
    ih, iw, _ = image.shape
    return dict(image=[image], dw=[iw], dh=[ih], x=[(pw - iw) / 2], y=[(ph - ih) / 2])


src = ColumnDataSource(data=mk_data(orig_img))
p.image_rgba(image='image', x='x', y='y', dw='dw', dh='dh', source=src)


b1 = Button(label='Rotate once!')

def rotate_img_by_15_deg():
    global current_angle
    current_angle += 15
    img = imutils.rotate_bound(orig_img.view(dtype=np.uint8), current_angle) # use imutils with cv2 format image
    src.data = mk_data(img.view(dtype=np.uint32)) # re-create attributes for bokeh using updated image

def callback_button():
    global periodic_cb_id
    if b2.label == 'Press to: > Play':
        print('adding add_periodic_callback()')
        b2.label = 'Press to: = Pause'
        periodic_cb_id = curdoc().add_periodic_callback(callback_update_data, T_update) # <-- this controls update frequency
    else:
        print('removing add_periodic_callback()')
        b2.label = 'Press to: > Play'
        curdoc().remove_periodic_callback( periodic_cb_id )

def callback_update_data():
    global current_angle
    current_angle += 1
    sys.stdout.flush()
    img = imutils.rotate_bound(orig_img.view(dtype=np.uint8), current_angle) # use imutils with cv2 formated (.view'ed) image
    src.data = mk_data(img.view(dtype=np.uint32)) # re-create all image attributes for bokeh using updated img


b1.on_click(rotate_img_by_15_deg)

b2 = Button(label='Press to: > Play')
b2.on_click(callback_button)


curdoc().add_root(column(p, b1, b2))

Run with:
bokeh serve --show rotate_img_callbacks.py

Just FYI - if you need it for some animation and nothing that required rotated image data, then you can just rotate an image when drawing it.
It doesn’t require you to always having to pass data back and forth, and it doesn’t even require a server.
I don’t think there’s a built-in way to do that right now but it’s something that easy to implement even without changing Bokeh itself.

Ok, agreed and understood. I am developing an application that uses a bokeh server to bring in network data and continuously update glyphs with location and angle data. It’s a create-once-then-update-frequently approach that animates multiple objects (with image icons) over an OpenStreetmap or ESRI imagery basemap.

The ColumnDataSource(), add_periodic_callback() and image rotation is about the only approach to animation that avoids deleting then re-creating glyphs on the fly.

Thanks for your inputs.

The ColumnDataSource() , add_periodic_callback() and image rotation is about the only approach to animation that avoids deleting then re-creating glyphs on the fly.

That’s not true. As I mentioned - you can just create your own AnimatedImage glyph or something like that by extending the existing ImageRGBA. Or, if the animation state depends on the backend state (not “on/off” but the specific angle), it could be RotatedImage.
It will prevent having to send the same (albeit rotated) images over and over again and it doesn’t require you to recreate any glyphs - you will be able to just switch a flag or change an angle in the relevant data source.

Here’s an example implementation of both. Hope it helps.
Note that it may take a few seconds to load because Bokeh has to compile the TypeScript sources.

from base64 import b64decode

import cv2
import numpy as np
from bokeh.core.property.dataspec import AngleSpec
from bokeh.core.property.primitive import Bool, Int
from bokeh.io import curdoc
from bokeh.layouts import column, row
from bokeh.models import ImageRGBA, Button, ColumnDataSource
from bokeh.plotting import figure
from bokeh.util.compiler import TypeScript


class AnimatedImage(ImageRGBA):
    animating = Bool(default=False)
    delay = Int(default=100, help="Animation delay in milliseconds.")

    __implementation__ = TypeScript("""
import {ImageRGBAData, ImageRGBAView, ImageRGBA} from "models/glyphs/image_rgba"
import {Class} from "core/class"
import * as p from "core/properties"
import {Context2d} from "core/util/canvas"


export interface AnimatedImageData extends ImageRGBAData {}

export interface AnimatedImageView extends ImageRGBAData {}
    
export class AnimatedImageView extends ImageRGBAView {
  model: AnimatedImage
  visuals: AnimatedImage.Visuals
  
  protected _intervalID: number | null = null
  protected _angle: number = 0
  protected _angle_delta: number = 0.1

  initialize(): void {
    super.initialize()
    this.connect(this.model.properties.animating.change, () => this.renderer.request_render())
    this.connect(this.model.properties.delay.change, () => {
      if (this._intervalID != null) {
        clearInterval(this._intervalID)
        this._setup_interval()
      }
    })
  }
  
  protected _setup_interval() {
    this._intervalID = setInterval(() => {
      this._angle += this._angle_delta
      this.renderer.request_render()
    }, this.model.delay)
  }

  protected _render(ctx: Context2d, indices: number[], {image_data, sx, sy, sw, sh}: AnimatedImageData): void {
    const old_smoothing = ctx.getImageSmoothingEnabled()
    ctx.setImageSmoothingEnabled(false)

    ctx.globalAlpha = this.model.global_alpha

    for (const i of indices) {
      if (isNaN(sx[i] + sy[i] + sw[i] + sh[i]))
        continue

      const sxi = sx[i] | 0
      const syi = sy[i] | 0
      const y_offset = syi

      ctx.translate(0, y_offset)
      ctx.scale(1, -1)
      ctx.translate(0, -y_offset)
      
      ctx.translate(sxi + sw[i] / 2, syi + sh[i] / 2)
      // Note that here the sign is different from the `RotatedImage`'s implementation.
      // That's because of the way Bokeh transforms all values in the `AngleSpec` that's
      // used by `RotatedImage`.
      ctx.rotate(-this._angle)
      
      ctx.drawImage(image_data[i], -sw[i] / 2, -sh[i] / 2, sw[i], sh[i])
      
      ctx.rotate(this._angle)
      ctx.translate(-(sxi + sw[i] / 2), -(syi + sh[i] / 2))
      
      
      ctx.translate(0, y_offset)
      ctx.scale(1, -1)
      ctx.translate(0, -y_offset)
    }

    ctx.setImageSmoothingEnabled(old_smoothing)
    
    if (!this.model.animating) {
      if (this._intervalID != null) {
        clearInterval(this._intervalID)
        this._intervalID = null
      }
    } else if (this._intervalID == null) {
      this._setup_interval()
    }
  }
}

export namespace AnimatedImage {
  export type Attrs = p.AttrsOf<Props>

  export type Props = ImageRGBA.Props & {
    animating: p.Property<boolean>
    delay: p.Property<number>
  }

  export type Visuals = ImageRGBA.Visuals
}

export interface AnimatedImage extends AnimatedImage.Attrs {}
    
export class AnimatedImage extends ImageRGBA {
  properties: AnimatedImage.Props
  default_view: Class<AnimatedImageView>

  constructor(attrs?: Partial<AnimatedImage.Attrs>) {
    super(attrs)
  }

  static init_AnimatedImage(): void {
    this.prototype.default_view = AnimatedImageView
    
    this.define<AnimatedImage.Props>({
      animating:   [ p.Boolean, false ],
      delay:       [ p.Int,     100   ],
    })
  }
}
""")


class RotatedImage(ImageRGBA):
    angle = AngleSpec(units_default='deg')

    __implementation__ = TypeScript("""
import {ImageRGBAData, ImageRGBAView, ImageRGBA} from "models/glyphs/image_rgba"
import {Class} from "core/class"
import {Arrayable} from "core/types"
import * as p from "core/properties"
import {Context2d} from "core/util/canvas"


export interface RotatedImageData extends ImageRGBAData {
  _angle: Arrayable<number>
}

export interface RotatedImageView extends ImageRGBAData {}
    
export class RotatedImageView extends ImageRGBAView {
  model: RotatedImage
  visuals: RotatedImage.Visuals
  
  initialize(): void {
    super.initialize()
    this.connect(this.model.properties.angle.change, () => this.renderer.request_render())
  }

  protected _render(ctx: Context2d, indices: number[], {image_data, sx, sy, sw, sh, _angle}: RotatedImageData): void {
    const old_smoothing = ctx.getImageSmoothingEnabled()
    ctx.setImageSmoothingEnabled(false)

    ctx.globalAlpha = this.model.global_alpha

    for (const i of indices) {
      if (isNaN(sx[i] + sy[i] + sw[i] + sh[i]))
        continue

      const sxi = sx[i] | 0
      const syi = sy[i] | 0
      const y_offset = syi

      ctx.translate(0, y_offset)
      ctx.scale(1, -1)
      ctx.translate(0, -y_offset)
      
      ctx.translate(sxi + sw[i] / 2, syi + sh[i] / 2)
      ctx.rotate(_angle[i])
      
      ctx.drawImage(image_data[i], -sw[i] / 2, -sh[i] / 2, sw[i], sh[i])
      
      ctx.rotate(-_angle[i])
      ctx.translate(-(sxi + sw[i] / 2), -(syi + sh[i] / 2))
      
      
      ctx.translate(0, y_offset)
      ctx.scale(1, -1)
      ctx.translate(0, -y_offset)
    }

    ctx.setImageSmoothingEnabled(old_smoothing)
  }
}

export namespace RotatedImage {
  export type Attrs = p.AttrsOf<Props>

  export type Props = ImageRGBA.Props & {
    angle: p.AngleSpec
  }

  export type Visuals = ImageRGBA.Visuals
}

export interface RotatedImage extends RotatedImage.Attrs {}
    
export class RotatedImage extends ImageRGBA {
  properties: RotatedImage.Props
  default_view: Class<RotatedImageView>

  constructor(attrs?: Partial<RotatedImage.Attrs>) {
    super(attrs)
  }

  static init_RotatedImage(): void {
    this.prototype.default_view = RotatedImageView
    
    this.define<RotatedImage.Props>({
      angle: [ p.AngleSpec, 0 ],
    })
  }
}
""")


img_jpeg = b64decode(
    'iVBORw0KGgoAAAANSUhEUgAAADIAAAAOBAMAAACfqVJUAAAAHlBMVEUAAAAAAwCKLY3jF17xWyICq65Qlc8js1T2qhmnzTbxGUeMAAAAAXRSTlMAQObYZgAAAAFiS0dEAIgFHUgAAACVSURBVBjTpY89DsIwDIWfRRF087sB6sCMyhHIAbp078TMgroy9gq5LXYSooqOfYqsF3+Jf0BU6c9IcjvI6bUlJiDO78EJIXYVJRXptHEeRyP22PPC7Cy2EUcnKQOmkPrwHIHnpNQNWT5OvLi1oa7/LMB1ouZqqES0CUCXm6+Iz9aER9/5bKyYZZ9w7y91n0KKDjf86QvVrhIPSsvLCwAAAABJRU5ErkJggg==')
arr = np.fromstring(img_jpeg, np.uint8)
frame = cv2.imdecode(arr, cv2.IMREAD_UNCHANGED)
img = frame[::-1].copy().view(dtype=np.uint32)

pw = ph = 100
ap = figure(aspect_ratio=1, x_range=(0, pw), y_range=(0, ph))
rp = figure(aspect_ratio=1, x_range=(0, pw), y_range=(0, ph))

ih, iw, _ = img.shape
ds = ColumnDataSource(data=dict(image=[img], dw=[iw], dh=[ih], x=[(pw - iw) / 2], y=[(ph - ih) / 2], angle=[0]))

ai = AnimatedImage(image='image', x='x', y='x', dw='dw', dh='dh', delay=10)
ap.add_glyph(ds, ai)
rp.add_glyph(ds, RotatedImage(image='image', x='x', y='x', dw='dw', dh='dh', angle='angle'))

angle_delta = 15
ab = Button(label='Start animation')
rb = Button(label=f'Rotate {angle_delta} degrees')


def toggle_animation():
    # It is trivial to move this whole callback to the JS side by using `Button.js_on_click`.
    ai.animating = not ai.animating
    if ai.animating:
        ab.label = 'Stop animation'
    else:
        ab.label = 'Start animation'


ab.on_click(toggle_animation)


def increate_rotation_angle():
    # Same here - it could be moved completely to the JS side.
    angle = [a + angle_delta for a in ds.data['angle']]
    ds.patch(dict(angle=[(slice(len(angle)), angle)]))


rb.on_click(increate_rotation_angle)

curdoc().add_root(row(column(ap, ab), column(rp, rb)))

umm… :joy: :crazy_face: :stuck_out_tongue_closed_eyes: :weary: :scream: :sob: :flushed: :astonished: :cold_sweat: :cry: :disappointed_relieved: :weary: :tired_face:

The video is awesome. I like it a lot.

The Javacsript… um… ok, I’m probably going to stick with Python. :sob: :sob: :sob: :joy: :joy: :joy:

@comperem FWIW most of that TypeScript is lifted from Bokeh itself, and lightly modified. I agree that creating custom models currently requires way too much obsure boilerplate. Helping to streamline the process is a priority of mine this year. Regardless, you could just take @p-himik’s code above and use it as-is!

Ok, thank you both for the inputs. The work is appreciated both from a support standpoint and also a coding standpoint.

I’m developing my own open-source software that includes Bokeh for visualization and the python-only solution in @p-himik’s first script above is probably the best approach for me to keep. I don’t use TypeScript and, although it may be boilerplate and only slightly modified, it’s more than I care to support and include.

It certainly represents a higher performance approach rather than moving between python and TypeScript memory locations. The improved performance, in this particular instance, does not outweigh the simplicity and readability of the pythonic approach. This is subjective and will vary as-needed, certainly.

I’ll try to post a screenshot or video of the end-result I’m working on when it’s ready.

Thank you both again. This discussion and both code examples have enabled a quite useful feature that I was having a hard time coding.

1 Like

@p-himik, I can’t absorb the large chunks of TypeScript or JavaScript.

The solution you posted earlier works very well and is entirely python-based.

I’ve got an application that has a changing x- and y-axis scales and @Bryan kindly suggested migrating the image’s glyph units to screen units with the dw_units and dh_units options like this:

p.image_rgba(image='image', x='x', y='y', dw='dw', dh='dh', dw_units='screen', dh_units='screen', source=src)

But doing that changes the math somehow so the rotation translates.

Do you have a suggestion on how to get the image to stay still using screen units?

Not sure what you mean by “entirely Python based”. 90% of the code is TypeScript.

In order to adjust the code so it respects d[wh]_units, I will have to spend some time. Time that I don’t have right now, sorry.

@p-himik, ok, that’s fine. It’s a geometry issue. the center is moving because it’s not accounted for in the rotation. There’s a simple solution and I can continue to investigate. It’s not all that urgent but it seems you’re quite familiar.

Certainly, the TypeScript is what executes, but the API I’m most familiar with is python.

Thanks

there’s some non-standard calculations being performed in the TypeScript.

the python interface to image height and width appears straightforward but produces unexpected results when attempting to subtract 1/2 the image height and width from the position.