Skip to content
Babylon.js GUI & Interaction — Build Interactive 3D Applications

Babylon.js GUI & Interaction — Build Interactive 3D Applications

DodaTech Updated Jun 6, 2026 13 min read

Babylon.js includes a full 2D GUI system that renders on top of your 3D scene — buttons, text, images, panels — alongside powerful mesh interaction through picking events and the ActionManager. Together, they turn a static 3D view into an interactive application.

What You’ll Learn

By the end of this tutorial, you’ll create full-screen UI overlays with buttons, text, images, and panels using AdvancedDynamicTexture, lay out UI with StackPanel and Grid containers, handle pointer events (click, hover, drag) on GUI controls, use ActionManager to detect clicks and hovers on 3D meshes, raycast from mouse position into the 3D scene with scene.pick, render interactive UI on 3D mesh surfaces, and sync HTML/CSS overlays with 3D world positions.

Why GUI and Interaction Matter

A 3D scene without interaction is a screensaver. Adding clickable objects, hover effects, UI panels, and floating labels transforms it into a tool — a product configurator where clicking a car body changes its color, a data dashboard where hovering over a data node shows metrics, or a game with a health bar and score display.

Babylon.js provides two complementary systems:

  1. GUI system — 2D controls rendered as an overlay (like HTML, but GPU-accelerated)
  2. ActionManager — detect clicks, hovers, and keys on 3D meshes

Real-world use: Durga Antivirus Pro uses Babylon.js GUI for its 3D threat dashboard. Clickable server nodes show popup panels with system metrics. A floating GUI panel lists active threats with color-coded severity buttons. Hovering over a threat connection highlights the attack path.

Where This Fits in Your Learning Path

    flowchart LR
    A["Babylon.js Getting Started"] --> B["Materials & Textures"]
    B --> C["Animation & Physics"]
    C --> D["**GUI & Interaction**"]
    D --> E["Importing & Assets"]
    style D fill:#f97316,stroke:#c2410c,color:#fff
    style A fill:#e5e7eb,stroke:#9ca3af,color:#374151
    style E fill:#22c55e,stroke:#16a34a,color:#22c55e
  

AdvancedDynamicTexture — The GUI Entry Point

Think of AdvancedDynamicTexture as the canvas where you place all your UI controls. It can render either:

  • Full-screen overlay — UI floats on top of the 3D scene (like a game HUD)
  • On a 3D mesh — UI renders on a surface in 3D space (like a computer screen in a scene)
// Full-screen UI (most common)
const adt = BABYLON.GUI.AdvancedDynamicTexture.CreateFullscreenUI('UI')
adt.idealWidth = 1920   // Design at 1920x1080 — scales automatically
adt.idealHeight = 1080

// Or render UI on a 3D mesh surface
const adt3D = BABYLON.GUI.AdvancedDynamicTexture.CreateForMesh(mesh, 1024, 1024)

Why idealWidth / idealHeight? You design the UI at a fixed resolution (1920x1080), and Babylon.js automatically scales it to fit any screen. This saves you from writing responsive layouts.


Basic GUI Controls

TextBlock — Displaying Text

const title = new BABYLON.GUI.TextBlock()
title.text = 'Hello Babylon!'
title.color = 'white'
title.fontSize = 32
title.fontWeight = 'bold'
title.textHorizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER
title.top = '20px'
adt.addControl(title)

Button — Clickable Action

const btn = BABYLON.GUI.Button.CreateSimpleButton('btn', 'Click Me')
btn.width = '160px'
btn.height = '48px'
btn.color = 'white'
btn.background = '#4a6cf7'
btn.cornerRadius = 8
btn.fontSize = 16
btn.top = '80px'
btn.horizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER
btn.verticalAlignment = BABYLON.GUI.Control.VERTICAL_ALIGNMENT_TOP
btn.onPointerClickObservable.add(() => {
  console.log('Button clicked!')
})
adt.addControl(btn)

Image

const img = new BABYLON.GUI.Image('logo', 'logo.png')
img.width = '200px'
img.height = 'auto'
img.stretch = BABYLON.GUI.Image.STRETCH_UNIFORM
img.top = '140px'
adt.addControl(img)

Common Control Properties

PropertyExampleDescription
width / height'160px', '50%', 'auto'Size with units
top / left'20px', '-10px'Position offset from alignment anchor
horizontalAlignmentLEFT, CENTER, RIGHTHorizontal anchor
verticalAlignmentTOP, CENTER, BOTTOMVertical anchor
color'white', '#ff0000'Text color
background'#4a6cf7', 'rgba(0,0,0,0.5)'Background color
alpha0.5Opacity (0-1)
isVisiblefalseShow/hide without removing

Layout Containers

Laying out UI manually with absolute positions is tedious. Use containers for automatic layout.

StackPanel — Vertical or Horizontal Stack

const panel = new BABYLON.GUI.StackPanel()
panel.width = '300px'
panel.height = '100%'
panel.horizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_RIGHT
panel.isVertical = true
panel.top = '20px'
panel.left = '-20px'

const label = new BABYLON.GUI.TextBlock()
label.text = 'Controls'; label.height = '40px'; label.color = 'white'
panel.addControl(label)

const btn1 = BABYLON.GUI.Button.CreateSimpleButton('btn1', 'Option A')
btn1.height = '40px'
panel.addControl(btn1)

adt.addControl(panel)

Grid — Row/Column Layout

const grid = new BABYLON.GUI.Grid()
grid.addColumnDefinition(100)   // first column: 100px fixed
grid.addColumnDefinition(1)     // second column: remaining space
grid.addRowDefinition(40)       // row 1: 40px
grid.addRowDefinition(40)       // row 2: 40px

const labelA = new BABYLON.GUI.TextBlock()
labelA.text = 'Label A'
grid.addControl(labelA, 0, 0)  // row 0, column 0

const inputA = new BABYLON.GUI.TextBlock()
inputA.text = 'Value A'
grid.addControl(inputA, 0, 1)  // row 0, column 1

adt.addControl(grid)

Pointer Events on GUI Controls

Every GUI control supports mouse/touch events through observables:

btn.onPointerClickObservable.add((info) => { })
btn.onPointerDownObservable.add(() => { })
btn.onPointerUpObservable.add(() => { })
btn.onPointerEnterObservable.add(() => {
  btn.background = '#5f7cf7'  // hover effect
})
btn.onPointerOutObservable.add(() => {
  btn.background = '#4a6cf7'
})

Mesh Interaction with ActionManager

GUI controls are for 2D overlays. For clicking on 3D objects themselves, use ActionManager.

// Enable interaction on a mesh
mesh.actionManager = new BABYLON.ActionManager(scene)

// Click to select
mesh.actionManager.registerAction(
  new BABYLON.ExecuteCodeAction(
    BABYLON.ActionManager.OnPickTrigger,
    (evt) => {
      console.log('Clicked:', evt.meshUnderPointer.name)
      evt.sourceMesh.material.emissiveColor = new BABYLON.Color3(0.5, 0, 0)
    }
  )
)

// Hover highlight
mesh.actionManager.registerAction(
  new BABYLON.ExecuteCodeAction(
    BABYLON.ActionManager.OnPointerOverTrigger,
    () => { mesh.material.emissiveColor = new BABYLON.Color3(0.2, 0.2, 0.2) }
  )
)

mesh.actionManager.registerAction(
  new BABYLON.ExecuteCodeAction(
    BABYLON.ActionManager.OnPointerOutTrigger,
    () => { mesh.material.emissiveColor = BABYLON.Color3.Black() }
  )
)

ActionManager Triggers

TriggerFires When
OnPickTriggerMesh is clicked/tapped
OnDoublePickTriggerDouble-click
OnPointerOverTriggerPointer enters the mesh
OnPointerOutTriggerPointer leaves the mesh
OnKeyUpTrigger / OnKeyDownTriggerKey pressed while mesh is “focused”

Scene Picking (Raycasting)

For more control than ActionManager, use scene.pick() — it casts a ray from the camera through the mouse position into the scene:

canvas.addEventListener('click', (evt) => {
  const pickResult = scene.pick(evt.clientX, evt.clientY)

  if (pickResult.hit) {
    console.log('Hit mesh:', pickResult.pickedMesh.name)
    console.log('World position:', pickResult.pickedPoint)
    console.log('Face normal:', pickResult.getNormal(true))
    console.log('UV coordinates:', pickResult.textureCoordinates)
    console.log('Distance from camera:', pickResult.distance)
  }
})

// Pick with filter
scene.pick(evt.clientX, evt.clientY, (mesh) => {
  return mesh.isPickable && mesh.name !== 'ground'
})

scene.pick() is useful for tooltips, context menus, and any interaction where you need detailed information about what was clicked.


3D GUI on Mesh Surfaces

You can render UI directly on a 3D surface — like a floating holographic panel or an in-world computer screen:

const panelMesh = BABYLON.MeshBuilder.CreatePlane('screen', { width: 3, height: 2 }, scene)
panelMesh.position.set(0, 2, 3)

const gui3D = BABYLON.GUI.AdvancedDynamicTexture.CreateForMesh(panelMesh, 600, 400)

const header = new BABYLON.GUI.TextBlock()
header.text = '3D Panel'
header.fontSize = 28
header.color = 'white'
gui3D.addControl(header)

const btn3D = BABYLON.GUI.Button.CreateSimpleButton('btn3d', 'Press')
btn3D.width = '200px'
btn3D.height = '50px'
btn3D.background = '#e04040'
btn3D.color = 'white'
btn3D.onPointerClickObservable.add(() => {
  header.text = 'Pressed!'
})
gui3D.addControl(btn3D)

// Make the panel always face the camera
panelMesh.billboardMode = BABYLON.Mesh.BILLBOARDMODE_ALL

HTML/CSS Overlay Synced with 3D

For complex UIs (forms, rich text, custom styling), layer HTML on top of the canvas and sync positions using Babylon’s world-to-screen projection:

function worldToScreen(worldPos, camera, canvas) {
  const pos = BABYLON.Vector3.Project(
    worldPos,
    BABYLON.Matrix.Identity(),
    scene.getTransformMatrix(),
    camera.getViewMatrix(),
    camera.getProjectionMatrix(),
    canvas.width / canvas.height
  )
  return { x: pos.x, y: pos.y }
}

// In render loop
engine.runRenderLoop(() => {
  const screenPos = worldToScreen(mesh.position, camera, canvas)
  document.getElementById('label').style.transform =
    `translate(${screenPos.x}px, ${screenPos.y}px)`
  scene.render()
})

This technique is how tooltips follow 3D objects, or how labels appear above characters in games.


Common Mistakes Beginners Make

1. GUI Controls Not Visible

You created the control but forgot to add it to the texture:

const btn = BABYLON.GUI.Button.CreateSimpleButton(...)
// Missing:
adt.addControl(btn)

2. Click Events Not Firing on GUI Controls

If a control is behind another control, it won’t receive clicks. Check isPointerBlocker (default true). Set control.isPointerBlocker = false on background controls to let clicks pass through.

3. Mesh OnPickTrigger Doesn’t Work

The mesh needs both isPickable = true (default) AND an actionManager:

mesh.actionManager = new BABYLON.ActionManager(scene)

4. Picking Returns Wrong Mesh

Other transparent meshes might be in front. Use the predicate parameter in scene.pick() to filter:

scene.pick(x, y, (mesh) => mesh.isPickable && mesh.name !== 'ground')

5. GUI Controls Wrong Size

Without idealWidth / idealHeight, controls may appear tiny on 4K screens or huge on phones. Always set them:

adt.idealWidth = 1920
adt.idealHeight = 1080

Practice Questions

  1. What is the difference between full-screen GUI and 3D mesh GUI? Full-screen renders in screen space like an overlay. 3D mesh GUI renders in world space on a 3D surface and can be occluded by other objects.

  2. What must you do to make a mesh clickable with ActionManager? Create a new ActionManager for the mesh: mesh.actionManager = new BABYLON.ActionManager(scene), then register actions with registerAction.

  3. How do you make a GUI control ignore pointer events (let clicks pass through)? Set control.isPointerBlocker = false.

  4. What does scene.pick(x, y) return? A PickInfo object with hit, pickedMesh, pickedPoint, getNormal(), textureCoordinates, and distance.

  5. Why should you set idealWidth and idealHeight on a full-screen UI? It makes the UI resolution-independent — you design at one resolution and it auto-scales to any screen size.

Challenge

Build an interactive 3D dashboard with:

  • Five clickable 3D objects (box, sphere, cylinder, torus, torus knot) each with a name and color
  • Clicking an object highlights it and shows a floating HTML popup with its name, type, and position
  • A GUI panel (using Babylon GUI) with scale and rotation sliders for the selected object
  • A “Reset” button that deselects the current object
  • Hover effects (emissive color change) on all clickable meshes

FAQ

Can I mix HTML overlays with Babylon GUI?

Yes. Many projects use Babylon GUI for game HUD and HTML for complex forms. Use scene.pick to get 3D positions and sync HTML elements via CSS transforms.

How do I make a drag-and-drop UI control?

Use onPointerDownObservable to track the start, onPointerMoveObservable during drag, and onPointerUpObservable to finalize. Update the control’s top/left properties during drag.

What’s the best way to create a scrollable list?

Wrap controls in a StackPanel inside a Rectangle with clipContent = true. Use onPointerWheelObservable on the panel to adjust top for scrolling.

Can I use custom fonts in GUI controls?

Yes. Load the font via CSS @font-face, then set control.fontFamily = 'Your Font'. For 3D GUI meshes, ensure the font loads before creating the GUI.

How do I handle multi-touch?

Babylon GUI handles multi-touch natively. Each touch is a separate pointer. Use pointerId in observable callbacks to track individual touches.

Try It Yourself: Interactive 3D Dashboard

Click 3D objects to select them, see a floating info popup, and control the selected object with sliders.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Babylon.js — Interactive 3D Dashboard</title>
  <style>
    * { margin: 0; padding: 0; box-sizing: border-box; }
    html, body { width: 100%; height: 100%; overflow: hidden; font-family: system-ui, sans-serif; }
    #renderCanvas { width: 100%; height: 100%; display: block; }
    #htmlPopup {
      position: absolute; display: none;
      background: rgba(0,0,0,0.85); color: #fff;
      padding: 16px 20px; border-radius: 10px; font-size: 14px;
      backdrop-filter: blur(6px); pointer-events: none;
      border: 1px solid rgba(255,255,255,0.1);
      min-width: 160px; transform: translate(-50%, -120%);
    }
    #htmlPopup h4 { margin-bottom: 4px; font-size: 16px; }
    #htmlPopup p { margin: 2px 0; opacity: 0.8; font-size: 13px; }
    #infoBar {
      position: absolute; bottom: 24px; left: 50%; transform: translateX(-50%);
      background: rgba(0,0,0,0.7); color: #fff;
      padding: 10px 20px; border-radius: 8px; font-size: 14px;
      backdrop-filter: blur(4px); pointer-events: none;
    }
    #guiContainer {
      position: absolute; top: 16px; right: 16px;
      background: rgba(0,0,0,0.75); color: #fff;
      padding: 14px 18px; border-radius: 8px; font-size: 13px;
      backdrop-filter: blur(4px); user-select: none; min-width: 180px;
      display: flex; flex-direction: column; gap: 8px;
    }
    #guiContainer label { display: flex; justify-content: space-between; align-items: center; gap: 10px; }
    #guiContainer input[type="range"] { width: 100px; }
    #guiContainer button { background: #4a6cf7; color: #fff; border: none; padding: 5px 10px; border-radius: 4px; cursor: pointer; font-size: 12px; }
    #guiContainer button:hover { background: #5f7cf7; }
    #guiContainer hr { border: none; border-top: 1px solid #444; margin: 4px 0; }
  </style>
</head>
<body>
  <canvas id="renderCanvas"></canvas>
  <div id="htmlPopup"><h4 id="popupTitle">Object</h4><p id="popupBody">Details</p></div>
  <div id="guiContainer">
    <strong>Dashboard Controls</strong>
    <label>Scale <input type="range" id="scaleSlider" min="0.5" max="2.5" step="0.05" value="1" /></label>
    <label>Rotation <input type="range" id="rotSlider" min="0" max="360" step="1" value="0" /></label>
    <hr /><button id="resetBtn">Reset Selection</button>
  </div>
  <div id="infoBar">Click a 3D object to select it</div>
  <script src="https://cdn.babylonjs.com/babylon.js"></script>
  <script src="https://cdn.babylonjs.com/gui/babylon.gui.min.js"></script>
  <script>
    const canvas = document.getElementById('renderCanvas')
    const engine = new BABYLON.Engine(canvas, true, { stencil: true })
    const scene = new BABYLON.Scene(engine)
    scene.clearColor = new BABYLON.Color3(0.07, 0.07, 0.12)

    const camera = new BABYLON.ArcRotateCamera('cam', -Math.PI / 2.8, Math.PI / 3.2, 14, new BABYLON.Vector3(0, 2, 0), scene)
    camera.lowerRadiusLimit = 5; camera.upperRadiusLimit = 30; camera.attachControl(canvas, true)

    const hemi = new BABYLON.HemisphericLight('hemi', new BABYLON.Vector3(0, 1, 0), scene); hemi.intensity = 0.35
    const dir = new BABYLON.DirectionalLight('dir', new BABYLON.Vector3(-1, -2, -1), scene); dir.position = new BABYLON.Vector3(5, 15, 10)

    const ground = BABYLON.MeshBuilder.CreateGround('ground', { width: 16, height: 16 }, scene)
    const gMat = new BABYLON.StandardMaterial('gMat', scene); gMat.diffuseColor = new BABYLON.Color3(0.18, 0.18, 0.22); gMat.specularColor = BABYLON.Color3.Black(); ground.material = gMat

    const objects = []; let selectedMesh = null; const originalColors = new Map()
    const configs = [
      { name: 'Ruby Box', color: new BABYLON.Color3(0.9, 0.15, 0.15), pos: new BABYLON.Vector3(-4, 0.5, -2), shape: 'box' },
      { name: 'Sapphire Sphere', color: new BABYLON.Color3(0.15, 0.4, 0.95), pos: new BABYLON.Vector3(0, 0.7, -2), shape: 'sphere' },
      { name: 'Emerald Cylinder', color: new BABYLON.Color3(0.1, 0.8, 0.3), pos: new BABYLON.Vector3(4, 0.6, -2), shape: 'cylinder' },
      { name: 'Amber Torus', color: new BABYLON.Color3(0.95, 0.6, 0.1), pos: new BABYLON.Vector3(-3, 1.2, 2), shape: 'torus' },
      { name: 'Platinum Ring', color: new BABYLON.Color3(0.75, 0.45, 0.9), pos: new BABYLON.Vector3(3, 0.9, 2), shape: 'torusKnot' },
    ]

    configs.forEach(cfg => {
      let mesh
      switch (cfg.shape) {
        case 'box': mesh = BABYLON.MeshBuilder.CreateBox(cfg.name, { size: 1.4 }, scene); break
        case 'sphere': mesh = BABYLON.MeshBuilder.CreateSphere(cfg.name, { diameter: 1.4, segments: 32 }, scene); break
        case 'cylinder': mesh = BABYLON.MeshBuilder.CreateCylinder(cfg.name, { height: 1.4, diameter: 1 }, scene); break
        case 'torus': mesh = BABYLON.MeshBuilder.CreateTorus(cfg.name, { diameter: 1.4, thickness: 0.35 }, scene); break
        case 'torusKnot': mesh = BABYLON.MeshBuilder.CreateTorusKnot(cfg.name, { radius: 0.7 }, scene); break
      }
      mesh.position.copyFrom(cfg.pos)
      const mat = new BABYLON.StandardMaterial(cfg.name + 'Mat', scene)
      mat.diffuseColor = cfg.color.clone(); mat.specularColor = new BABYLON.Color3(0.3, 0.3, 0.3); mat.specularPower = 32
      mesh.material = mat; originalColors.set(mesh, cfg.color.clone())

      mesh.actionManager = new BABYLON.ActionManager(scene)
      mesh.actionManager.registerAction(new BABYLON.ExecuteCodeAction(BABYLON.ActionManager.OnPickTrigger, (evt) => selectMesh(evt.meshUnderPointer)))
      mesh.actionManager.registerAction(new BABYLON.ExecuteCodeAction(BABYLON.ActionManager.OnPointerOverTrigger, () => { if (mesh !== selectedMesh) mesh.material.emissiveColor = new BABYLON.Color3(0.15, 0.15, 0.25) }))
      mesh.actionManager.registerAction(new BABYLON.ExecuteCodeAction(BABYLON.ActionManager.OnPointerOutTrigger, () => { if (mesh !== selectedMesh) mesh.material.emissiveColor = BABYLON.Color3.Black() }))
      objects.push(mesh)
    })

    const popup = document.getElementById('htmlPopup')
    const popupTitle = document.getElementById('popupTitle')
    const popupBody = document.getElementById('popupBody')
    const infoBar = document.getElementById('infoBar')

    function selectMesh(mesh) {
      if (selectedMesh) { selectedMesh.material.emissiveColor = BABYLON.Color3.Black(); selectedMesh.material.diffuseColor.copyFrom(originalColors.get(selectedMesh)) }
      selectedMesh = mesh
      mesh.material.emissiveColor = new BABYLON.Color3(0.3, 0.3, 0); mesh.material.diffuseColor = new BABYLON.Color3(1, 1, 0.6)
      popupTitle.textContent = mesh.name; popupBody.textContent = 'Type: ' + mesh.getClassName() + ' | Pos: ' + mesh.position.x.toFixed(1) + ', ' + mesh.position.y.toFixed(1) + ', ' + mesh.position.z.toFixed(1)
      infoBar.textContent = 'Selected: ' + mesh.name
      updatePopupPosition(mesh); popup.style.display = 'block'
      document.getElementById('scaleSlider').value = mesh.scaling.x
      document.getElementById('rotSlider').value = BABYLON.Tools.ToDegrees(mesh.rotation.y).toFixed(0)
    }

    function updatePopupPosition(mesh) {
      const pos = BABYLON.Vector3.Project(new BABYLON.Vector3(0, 1.2, 0).add(mesh.position), BABYLON.Matrix.Identity(), scene.getTransformMatrix(), camera.getViewMatrix(), camera.getProjectionMatrix(), canvas.width / canvas.height)
      popup.style.left = pos.x + 'px'; popup.style.top = pos.y + 'px'
    }

    document.getElementById('scaleSlider').addEventListener('input', function () { if (!selectedMesh) return; const s = parseFloat(this.value); selectedMesh.scaling.set(s, s, s) })
    document.getElementById('rotSlider').addEventListener('input', function () { if (!selectedMesh) return; selectedMesh.rotation.y = BABYLON.Tools.ToRadians(parseFloat(this.value)) })
    document.getElementById('resetBtn').addEventListener('click', () => { if (selectedMesh) { selectedMesh.material.emissiveColor = BABYLON.Color3.Black(); selectedMesh.material.diffuseColor.copyFrom(originalColors.get(selectedMesh)); selectedMesh.scaling.set(1, 1, 1); selectedMesh.rotation.y = 0; selectedMesh = null; popup.style.display = 'none'; infoBar.textContent = 'Click a 3D object to select it' } })

    engine.runRenderLoop(() => {
      objects.forEach((m, i) => { if (m !== selectedMesh) m.rotation.y += 0.005 * (i + 1) * 0.5 })
      if (selectedMesh) updatePopupPosition(selectedMesh)
      scene.render()
    })
    window.addEventListener('resize', () => engine.resize())
  </script>
</body>
</html>

What to expect: Five clickable 3D objects with different shapes and colors. Click to select — a floating popup appears with details, and sliders control the selected object’s scale and rotation. Unselected objects gently rotate automatically.


What’s Next

TutorialWhat You’ll Learn
Importing & AssetsglTF/GLB import, asset containers, optimization

Related topics: Babylon.js Animation & Physics, Babylon.js Materials & Textures.

What’s Next

Congratulations on completing this Babylonjs Gui Interaction tutorial! Here’s where to go from here:

  • Practice daily — Consistency is more important than long study sessions
  • Build a project — Apply what you learned by building something real
  • Explore related topics — Check out other tutorials in the same category
  • Join the community — Discuss with other learners and share your progress

Remember: every expert was once a beginner. Keep coding!

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro