Babylon.js GUI & Interaction — Build Interactive 3D Applications
Babylon.js includes a full 2D GUI system that renders on top of your 3D scene — buttons, text, images, panels — alongside powerful mesh interaction through picking events and the ActionManager. Together, they turn a static 3D view into an interactive application.
What You’ll Learn
By the end of this tutorial, you’ll create full-screen UI overlays with buttons, text, images, and panels using AdvancedDynamicTexture, lay out UI with StackPanel and Grid containers, handle pointer events (click, hover, drag) on GUI controls, use ActionManager to detect clicks and hovers on 3D meshes, raycast from mouse position into the 3D scene with scene.pick, render interactive UI on 3D mesh surfaces, and sync HTML/CSS overlays with 3D world positions.
Why GUI and Interaction Matter
A 3D scene without interaction is a screensaver. Adding clickable objects, hover effects, UI panels, and floating labels transforms it into a tool — a product configurator where clicking a car body changes its color, a data dashboard where hovering over a data node shows metrics, or a game with a health bar and score display.
Babylon.js provides two complementary systems:
- GUI system — 2D controls rendered as an overlay (like HTML, but GPU-accelerated)
- ActionManager — detect clicks, hovers, and keys on 3D meshes
Real-world use: Durga Antivirus Pro uses Babylon.js GUI for its 3D threat dashboard. Clickable server nodes show popup panels with system metrics. A floating GUI panel lists active threats with color-coded severity buttons. Hovering over a threat connection highlights the attack path.
Where This Fits in Your Learning Path
flowchart LR
A["Babylon.js Getting Started"] --> B["Materials & Textures"]
B --> C["Animation & Physics"]
C --> D["**GUI & Interaction**"]
D --> E["Importing & Assets"]
style D fill:#f97316,stroke:#c2410c,color:#fff
style A fill:#e5e7eb,stroke:#9ca3af,color:#374151
style E fill:#22c55e,stroke:#16a34a,color:#22c55e
AdvancedDynamicTexture — The GUI Entry Point
Think of AdvancedDynamicTexture as the canvas where you place all your UI controls. It can render either:
- Full-screen overlay — UI floats on top of the 3D scene (like a game HUD)
- On a 3D mesh — UI renders on a surface in 3D space (like a computer screen in a scene)
// Full-screen UI (most common)
const adt = BABYLON.GUI.AdvancedDynamicTexture.CreateFullscreenUI('UI')
adt.idealWidth = 1920 // Design at 1920x1080 — scales automatically
adt.idealHeight = 1080
// Or render UI on a 3D mesh surface
const adt3D = BABYLON.GUI.AdvancedDynamicTexture.CreateForMesh(mesh, 1024, 1024)Why idealWidth / idealHeight? You design the UI at a fixed resolution (1920x1080), and Babylon.js automatically scales it to fit any screen. This saves you from writing responsive layouts.
Basic GUI Controls
TextBlock — Displaying Text
const title = new BABYLON.GUI.TextBlock()
title.text = 'Hello Babylon!'
title.color = 'white'
title.fontSize = 32
title.fontWeight = 'bold'
title.textHorizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER
title.top = '20px'
adt.addControl(title)Button — Clickable Action
const btn = BABYLON.GUI.Button.CreateSimpleButton('btn', 'Click Me')
btn.width = '160px'
btn.height = '48px'
btn.color = 'white'
btn.background = '#4a6cf7'
btn.cornerRadius = 8
btn.fontSize = 16
btn.top = '80px'
btn.horizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER
btn.verticalAlignment = BABYLON.GUI.Control.VERTICAL_ALIGNMENT_TOP
btn.onPointerClickObservable.add(() => {
console.log('Button clicked!')
})
adt.addControl(btn)Image
const img = new BABYLON.GUI.Image('logo', 'logo.png')
img.width = '200px'
img.height = 'auto'
img.stretch = BABYLON.GUI.Image.STRETCH_UNIFORM
img.top = '140px'
adt.addControl(img)Common Control Properties
| Property | Example | Description |
|---|---|---|
width / height | '160px', '50%', 'auto' | Size with units |
top / left | '20px', '-10px' | Position offset from alignment anchor |
horizontalAlignment | LEFT, CENTER, RIGHT | Horizontal anchor |
verticalAlignment | TOP, CENTER, BOTTOM | Vertical anchor |
color | 'white', '#ff0000' | Text color |
background | '#4a6cf7', 'rgba(0,0,0,0.5)' | Background color |
alpha | 0.5 | Opacity (0-1) |
isVisible | false | Show/hide without removing |
Layout Containers
Laying out UI manually with absolute positions is tedious. Use containers for automatic layout.
StackPanel — Vertical or Horizontal Stack
const panel = new BABYLON.GUI.StackPanel()
panel.width = '300px'
panel.height = '100%'
panel.horizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_RIGHT
panel.isVertical = true
panel.top = '20px'
panel.left = '-20px'
const label = new BABYLON.GUI.TextBlock()
label.text = 'Controls'; label.height = '40px'; label.color = 'white'
panel.addControl(label)
const btn1 = BABYLON.GUI.Button.CreateSimpleButton('btn1', 'Option A')
btn1.height = '40px'
panel.addControl(btn1)
adt.addControl(panel)Grid — Row/Column Layout
const grid = new BABYLON.GUI.Grid()
grid.addColumnDefinition(100) // first column: 100px fixed
grid.addColumnDefinition(1) // second column: remaining space
grid.addRowDefinition(40) // row 1: 40px
grid.addRowDefinition(40) // row 2: 40px
const labelA = new BABYLON.GUI.TextBlock()
labelA.text = 'Label A'
grid.addControl(labelA, 0, 0) // row 0, column 0
const inputA = new BABYLON.GUI.TextBlock()
inputA.text = 'Value A'
grid.addControl(inputA, 0, 1) // row 0, column 1
adt.addControl(grid)Pointer Events on GUI Controls
Every GUI control supports mouse/touch events through observables:
btn.onPointerClickObservable.add((info) => { })
btn.onPointerDownObservable.add(() => { })
btn.onPointerUpObservable.add(() => { })
btn.onPointerEnterObservable.add(() => {
btn.background = '#5f7cf7' // hover effect
})
btn.onPointerOutObservable.add(() => {
btn.background = '#4a6cf7'
})Mesh Interaction with ActionManager
GUI controls are for 2D overlays. For clicking on 3D objects themselves, use ActionManager.
// Enable interaction on a mesh
mesh.actionManager = new BABYLON.ActionManager(scene)
// Click to select
mesh.actionManager.registerAction(
new BABYLON.ExecuteCodeAction(
BABYLON.ActionManager.OnPickTrigger,
(evt) => {
console.log('Clicked:', evt.meshUnderPointer.name)
evt.sourceMesh.material.emissiveColor = new BABYLON.Color3(0.5, 0, 0)
}
)
)
// Hover highlight
mesh.actionManager.registerAction(
new BABYLON.ExecuteCodeAction(
BABYLON.ActionManager.OnPointerOverTrigger,
() => { mesh.material.emissiveColor = new BABYLON.Color3(0.2, 0.2, 0.2) }
)
)
mesh.actionManager.registerAction(
new BABYLON.ExecuteCodeAction(
BABYLON.ActionManager.OnPointerOutTrigger,
() => { mesh.material.emissiveColor = BABYLON.Color3.Black() }
)
)ActionManager Triggers
| Trigger | Fires When |
|---|---|
OnPickTrigger | Mesh is clicked/tapped |
OnDoublePickTrigger | Double-click |
OnPointerOverTrigger | Pointer enters the mesh |
OnPointerOutTrigger | Pointer leaves the mesh |
OnKeyUpTrigger / OnKeyDownTrigger | Key pressed while mesh is “focused” |
Scene Picking (Raycasting)
For more control than ActionManager, use scene.pick() — it casts a ray from the camera through the mouse position into the scene:
canvas.addEventListener('click', (evt) => {
const pickResult = scene.pick(evt.clientX, evt.clientY)
if (pickResult.hit) {
console.log('Hit mesh:', pickResult.pickedMesh.name)
console.log('World position:', pickResult.pickedPoint)
console.log('Face normal:', pickResult.getNormal(true))
console.log('UV coordinates:', pickResult.textureCoordinates)
console.log('Distance from camera:', pickResult.distance)
}
})
// Pick with filter
scene.pick(evt.clientX, evt.clientY, (mesh) => {
return mesh.isPickable && mesh.name !== 'ground'
})scene.pick() is useful for tooltips, context menus, and any interaction where you need detailed information about what was clicked.
3D GUI on Mesh Surfaces
You can render UI directly on a 3D surface — like a floating holographic panel or an in-world computer screen:
const panelMesh = BABYLON.MeshBuilder.CreatePlane('screen', { width: 3, height: 2 }, scene)
panelMesh.position.set(0, 2, 3)
const gui3D = BABYLON.GUI.AdvancedDynamicTexture.CreateForMesh(panelMesh, 600, 400)
const header = new BABYLON.GUI.TextBlock()
header.text = '3D Panel'
header.fontSize = 28
header.color = 'white'
gui3D.addControl(header)
const btn3D = BABYLON.GUI.Button.CreateSimpleButton('btn3d', 'Press')
btn3D.width = '200px'
btn3D.height = '50px'
btn3D.background = '#e04040'
btn3D.color = 'white'
btn3D.onPointerClickObservable.add(() => {
header.text = 'Pressed!'
})
gui3D.addControl(btn3D)
// Make the panel always face the camera
panelMesh.billboardMode = BABYLON.Mesh.BILLBOARDMODE_ALLHTML/CSS Overlay Synced with 3D
For complex UIs (forms, rich text, custom styling), layer HTML on top of the canvas and sync positions using Babylon’s world-to-screen projection:
function worldToScreen(worldPos, camera, canvas) {
const pos = BABYLON.Vector3.Project(
worldPos,
BABYLON.Matrix.Identity(),
scene.getTransformMatrix(),
camera.getViewMatrix(),
camera.getProjectionMatrix(),
canvas.width / canvas.height
)
return { x: pos.x, y: pos.y }
}
// In render loop
engine.runRenderLoop(() => {
const screenPos = worldToScreen(mesh.position, camera, canvas)
document.getElementById('label').style.transform =
`translate(${screenPos.x}px, ${screenPos.y}px)`
scene.render()
})This technique is how tooltips follow 3D objects, or how labels appear above characters in games.
Common Mistakes Beginners Make
1. GUI Controls Not Visible
You created the control but forgot to add it to the texture:
const btn = BABYLON.GUI.Button.CreateSimpleButton(...)
// Missing:
adt.addControl(btn)2. Click Events Not Firing on GUI Controls
If a control is behind another control, it won’t receive clicks. Check isPointerBlocker (default true). Set control.isPointerBlocker = false on background controls to let clicks pass through.
3. Mesh OnPickTrigger Doesn’t Work
The mesh needs both isPickable = true (default) AND an actionManager:
mesh.actionManager = new BABYLON.ActionManager(scene)4. Picking Returns Wrong Mesh
Other transparent meshes might be in front. Use the predicate parameter in scene.pick() to filter:
scene.pick(x, y, (mesh) => mesh.isPickable && mesh.name !== 'ground')5. GUI Controls Wrong Size
Without idealWidth / idealHeight, controls may appear tiny on 4K screens or huge on phones. Always set them:
adt.idealWidth = 1920
adt.idealHeight = 1080Practice Questions
What is the difference between full-screen GUI and 3D mesh GUI? Full-screen renders in screen space like an overlay. 3D mesh GUI renders in world space on a 3D surface and can be occluded by other objects.
What must you do to make a mesh clickable with ActionManager? Create a new
ActionManagerfor the mesh:mesh.actionManager = new BABYLON.ActionManager(scene), then register actions withregisterAction.How do you make a GUI control ignore pointer events (let clicks pass through)? Set
control.isPointerBlocker = false.What does
scene.pick(x, y)return? APickInfoobject withhit,pickedMesh,pickedPoint,getNormal(),textureCoordinates, anddistance.Why should you set
idealWidthandidealHeighton a full-screen UI? It makes the UI resolution-independent — you design at one resolution and it auto-scales to any screen size.
Challenge
Build an interactive 3D dashboard with:
- Five clickable 3D objects (box, sphere, cylinder, torus, torus knot) each with a name and color
- Clicking an object highlights it and shows a floating HTML popup with its name, type, and position
- A GUI panel (using Babylon GUI) with scale and rotation sliders for the selected object
- A “Reset” button that deselects the current object
- Hover effects (emissive color change) on all clickable meshes
FAQ
Try It Yourself: Interactive 3D Dashboard
Click 3D objects to select them, see a floating info popup, and control the selected object with sliders.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Babylon.js — Interactive 3D Dashboard</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
html, body { width: 100%; height: 100%; overflow: hidden; font-family: system-ui, sans-serif; }
#renderCanvas { width: 100%; height: 100%; display: block; }
#htmlPopup {
position: absolute; display: none;
background: rgba(0,0,0,0.85); color: #fff;
padding: 16px 20px; border-radius: 10px; font-size: 14px;
backdrop-filter: blur(6px); pointer-events: none;
border: 1px solid rgba(255,255,255,0.1);
min-width: 160px; transform: translate(-50%, -120%);
}
#htmlPopup h4 { margin-bottom: 4px; font-size: 16px; }
#htmlPopup p { margin: 2px 0; opacity: 0.8; font-size: 13px; }
#infoBar {
position: absolute; bottom: 24px; left: 50%; transform: translateX(-50%);
background: rgba(0,0,0,0.7); color: #fff;
padding: 10px 20px; border-radius: 8px; font-size: 14px;
backdrop-filter: blur(4px); pointer-events: none;
}
#guiContainer {
position: absolute; top: 16px; right: 16px;
background: rgba(0,0,0,0.75); color: #fff;
padding: 14px 18px; border-radius: 8px; font-size: 13px;
backdrop-filter: blur(4px); user-select: none; min-width: 180px;
display: flex; flex-direction: column; gap: 8px;
}
#guiContainer label { display: flex; justify-content: space-between; align-items: center; gap: 10px; }
#guiContainer input[type="range"] { width: 100px; }
#guiContainer button { background: #4a6cf7; color: #fff; border: none; padding: 5px 10px; border-radius: 4px; cursor: pointer; font-size: 12px; }
#guiContainer button:hover { background: #5f7cf7; }
#guiContainer hr { border: none; border-top: 1px solid #444; margin: 4px 0; }
</style>
</head>
<body>
<canvas id="renderCanvas"></canvas>
<div id="htmlPopup"><h4 id="popupTitle">Object</h4><p id="popupBody">Details</p></div>
<div id="guiContainer">
<strong>Dashboard Controls</strong>
<label>Scale <input type="range" id="scaleSlider" min="0.5" max="2.5" step="0.05" value="1" /></label>
<label>Rotation <input type="range" id="rotSlider" min="0" max="360" step="1" value="0" /></label>
<hr /><button id="resetBtn">Reset Selection</button>
</div>
<div id="infoBar">Click a 3D object to select it</div>
<script src="https://cdn.babylonjs.com/babylon.js"></script>
<script src="https://cdn.babylonjs.com/gui/babylon.gui.min.js"></script>
<script>
const canvas = document.getElementById('renderCanvas')
const engine = new BABYLON.Engine(canvas, true, { stencil: true })
const scene = new BABYLON.Scene(engine)
scene.clearColor = new BABYLON.Color3(0.07, 0.07, 0.12)
const camera = new BABYLON.ArcRotateCamera('cam', -Math.PI / 2.8, Math.PI / 3.2, 14, new BABYLON.Vector3(0, 2, 0), scene)
camera.lowerRadiusLimit = 5; camera.upperRadiusLimit = 30; camera.attachControl(canvas, true)
const hemi = new BABYLON.HemisphericLight('hemi', new BABYLON.Vector3(0, 1, 0), scene); hemi.intensity = 0.35
const dir = new BABYLON.DirectionalLight('dir', new BABYLON.Vector3(-1, -2, -1), scene); dir.position = new BABYLON.Vector3(5, 15, 10)
const ground = BABYLON.MeshBuilder.CreateGround('ground', { width: 16, height: 16 }, scene)
const gMat = new BABYLON.StandardMaterial('gMat', scene); gMat.diffuseColor = new BABYLON.Color3(0.18, 0.18, 0.22); gMat.specularColor = BABYLON.Color3.Black(); ground.material = gMat
const objects = []; let selectedMesh = null; const originalColors = new Map()
const configs = [
{ name: 'Ruby Box', color: new BABYLON.Color3(0.9, 0.15, 0.15), pos: new BABYLON.Vector3(-4, 0.5, -2), shape: 'box' },
{ name: 'Sapphire Sphere', color: new BABYLON.Color3(0.15, 0.4, 0.95), pos: new BABYLON.Vector3(0, 0.7, -2), shape: 'sphere' },
{ name: 'Emerald Cylinder', color: new BABYLON.Color3(0.1, 0.8, 0.3), pos: new BABYLON.Vector3(4, 0.6, -2), shape: 'cylinder' },
{ name: 'Amber Torus', color: new BABYLON.Color3(0.95, 0.6, 0.1), pos: new BABYLON.Vector3(-3, 1.2, 2), shape: 'torus' },
{ name: 'Platinum Ring', color: new BABYLON.Color3(0.75, 0.45, 0.9), pos: new BABYLON.Vector3(3, 0.9, 2), shape: 'torusKnot' },
]
configs.forEach(cfg => {
let mesh
switch (cfg.shape) {
case 'box': mesh = BABYLON.MeshBuilder.CreateBox(cfg.name, { size: 1.4 }, scene); break
case 'sphere': mesh = BABYLON.MeshBuilder.CreateSphere(cfg.name, { diameter: 1.4, segments: 32 }, scene); break
case 'cylinder': mesh = BABYLON.MeshBuilder.CreateCylinder(cfg.name, { height: 1.4, diameter: 1 }, scene); break
case 'torus': mesh = BABYLON.MeshBuilder.CreateTorus(cfg.name, { diameter: 1.4, thickness: 0.35 }, scene); break
case 'torusKnot': mesh = BABYLON.MeshBuilder.CreateTorusKnot(cfg.name, { radius: 0.7 }, scene); break
}
mesh.position.copyFrom(cfg.pos)
const mat = new BABYLON.StandardMaterial(cfg.name + 'Mat', scene)
mat.diffuseColor = cfg.color.clone(); mat.specularColor = new BABYLON.Color3(0.3, 0.3, 0.3); mat.specularPower = 32
mesh.material = mat; originalColors.set(mesh, cfg.color.clone())
mesh.actionManager = new BABYLON.ActionManager(scene)
mesh.actionManager.registerAction(new BABYLON.ExecuteCodeAction(BABYLON.ActionManager.OnPickTrigger, (evt) => selectMesh(evt.meshUnderPointer)))
mesh.actionManager.registerAction(new BABYLON.ExecuteCodeAction(BABYLON.ActionManager.OnPointerOverTrigger, () => { if (mesh !== selectedMesh) mesh.material.emissiveColor = new BABYLON.Color3(0.15, 0.15, 0.25) }))
mesh.actionManager.registerAction(new BABYLON.ExecuteCodeAction(BABYLON.ActionManager.OnPointerOutTrigger, () => { if (mesh !== selectedMesh) mesh.material.emissiveColor = BABYLON.Color3.Black() }))
objects.push(mesh)
})
const popup = document.getElementById('htmlPopup')
const popupTitle = document.getElementById('popupTitle')
const popupBody = document.getElementById('popupBody')
const infoBar = document.getElementById('infoBar')
function selectMesh(mesh) {
if (selectedMesh) { selectedMesh.material.emissiveColor = BABYLON.Color3.Black(); selectedMesh.material.diffuseColor.copyFrom(originalColors.get(selectedMesh)) }
selectedMesh = mesh
mesh.material.emissiveColor = new BABYLON.Color3(0.3, 0.3, 0); mesh.material.diffuseColor = new BABYLON.Color3(1, 1, 0.6)
popupTitle.textContent = mesh.name; popupBody.textContent = 'Type: ' + mesh.getClassName() + ' | Pos: ' + mesh.position.x.toFixed(1) + ', ' + mesh.position.y.toFixed(1) + ', ' + mesh.position.z.toFixed(1)
infoBar.textContent = 'Selected: ' + mesh.name
updatePopupPosition(mesh); popup.style.display = 'block'
document.getElementById('scaleSlider').value = mesh.scaling.x
document.getElementById('rotSlider').value = BABYLON.Tools.ToDegrees(mesh.rotation.y).toFixed(0)
}
function updatePopupPosition(mesh) {
const pos = BABYLON.Vector3.Project(new BABYLON.Vector3(0, 1.2, 0).add(mesh.position), BABYLON.Matrix.Identity(), scene.getTransformMatrix(), camera.getViewMatrix(), camera.getProjectionMatrix(), canvas.width / canvas.height)
popup.style.left = pos.x + 'px'; popup.style.top = pos.y + 'px'
}
document.getElementById('scaleSlider').addEventListener('input', function () { if (!selectedMesh) return; const s = parseFloat(this.value); selectedMesh.scaling.set(s, s, s) })
document.getElementById('rotSlider').addEventListener('input', function () { if (!selectedMesh) return; selectedMesh.rotation.y = BABYLON.Tools.ToRadians(parseFloat(this.value)) })
document.getElementById('resetBtn').addEventListener('click', () => { if (selectedMesh) { selectedMesh.material.emissiveColor = BABYLON.Color3.Black(); selectedMesh.material.diffuseColor.copyFrom(originalColors.get(selectedMesh)); selectedMesh.scaling.set(1, 1, 1); selectedMesh.rotation.y = 0; selectedMesh = null; popup.style.display = 'none'; infoBar.textContent = 'Click a 3D object to select it' } })
engine.runRenderLoop(() => {
objects.forEach((m, i) => { if (m !== selectedMesh) m.rotation.y += 0.005 * (i + 1) * 0.5 })
if (selectedMesh) updatePopupPosition(selectedMesh)
scene.render()
})
window.addEventListener('resize', () => engine.resize())
</script>
</body>
</html>What to expect: Five clickable 3D objects with different shapes and colors. Click to select — a floating popup appears with details, and sliders control the selected object’s scale and rotation. Unselected objects gently rotate automatically.
What’s Next
| Tutorial | What You’ll Learn |
|---|---|
| Importing & Assets | glTF/GLB import, asset containers, optimization |
Related topics: Babylon.js Animation & Physics, Babylon.js Materials & Textures.
What’s Next
Congratulations on completing this Babylonjs Gui Interaction tutorial! Here’s where to go from here:
- Practice daily — Consistency is more important than long study sessions
- Build a project — Apply what you learned by building something real
- Explore related topics — Check out other tutorials in the same category
- Join the community — Discuss with other learners and share your progress
Remember: every expert was once a beginner. Keep coding!
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro