Use.GPU has DOM-like canvas events, with native support for GPU-driven picking. This supports the full set of pointer behaviors, including pointer capture and motion. It follows the familiar DOM event API, but papers over browser/platform differences similar to React.

@use-gpu/interact provides some prefab controls that leverage this, but also exposes hooks for custom constraints and dragging.

Additionally, there are declarative hooks as a light-way mechanism, e.g. to pass UI state into shaders:

When using the hooks, the consuming component should ensure it only responds when the relevant hook state changes, as a component may be re-rendered for other reasons.

Pointer Events

GPU Picking

Use the <Pick> wrapper to make pickable objects. You must also enable picking on the surrounding <Pass>.

<Pick> will generate a unique id which you pass on to shapes and layers inside:

<Pick
  onPointerDown={(e: PointerEvent, index: number) => {...}}
  onPointerUp={(e: PointerEvent, index: number) => {...}}
>{
  ({ id, hovered }) => <>
    {hovered ? <Cursor cursor="move" /> : null}
    <PointLayer id={id} {...} />
  </>
}</Pick>

All shapes and layers with the same id will act as one pickable surface. This allows you to decouple interaction from rendering.

Pass a lookup/lookups attribute to most layers to distinguish different data points or sub-surfaces. This is returned as the event's picking index.

While onPointerEnter/onPointerLeave only fires when entering/leaving the pickable surface, onPointerOver/onPointerOut fires whenever the index changes.

Use <Pick all ...> to create an event handler for the canvas background.

Ray Picking

To project a given PointerEvent into world space, use pick(event) from useViewContext. This will return an [origin, ray] pair.

  const {pick} = useViewContext();
  const handlePointerDown = (e: PointerEvent) => {
    const [origin, ray] = pick(e);
    // ...
  };

The useDrag hook wraps this mechanism to implement arbitrary 3D dragging, with e.g. a lineConstraint or planeConstraint.

Pointer Capture and Lock

During a pointer drag, events should target the originally clicked element, even if the pointer momentarily exits.

Use.GPU automatically performs pointer capture on the HTML canvas to ensure this. <Pick> will then capture on a per-id level. Use usePointerCapture to control capturing by hand.

Use usePointerLock to lock the mouse to the canvas, as used e.g. by <FPSControls>.

Event Type

The interface for events is DOM-like, with e.g. e.preventDefault() and e.stopPropagation().

Similar to React, the Event is actually a synthetic event, which wraps the native DOM canvas event as e.nativeEvent.

There are a few minor differences with the DOM, intended to provide more convenience:

Both x/y and u/v coordinates for the canvas are directly provided
Buttons are labeled as left, right, etc. instead of numeric flags
Mouse wheel handling shenanigans are unified, as moveX/moveY/spinX/spinY
XY motion is provided as a standard moveX / moveY in logical pixels

If an event has multiple target handlers, they will be dispatched in reverse tree-order:

Later siblings go before earlier siblings
Children go before parents

This follows the logic that the last rendered layers should go first.

Event Provider

<Pick> is merely a convenience wrapper around the useCanvasEvents hook. This in turn just quote-yeets the event handlers to the global event reconciler, tagged with their id.

As such, it's possible to skip <Pick> entirely and pass handlers to the event reconciler directly:

// In a component
const handlers = useCanvasEvents(id, {
  pointerDown: (e) => {...},
});
return [handlers, ...];

This is how <OrbitControls> and <PanControls> are implemented, despite not having a pickable surface.

Sections

Overview