From capture and rigging to projection geometry and show control, this guide breaks down the technical pipeline behind a convincing holographic performance.
The most convincing holographic performances look deceptively simple. A human figure appears onstage, addresses an audience, moves with believable weight, and seems to occupy the same space as a live performer.
Behind that moment is a chain of disciplines: rights clearance, 3D capture, character modeling, facial rigging, body motion capture, hair and cloth work, lighting, rendering, projection geometry, audio, networking, and show control.
For developers and technical artists, the interesting question is not whether the result looks futuristic. It is how the complete system preserves identity, timing, depth, and performance intent from the first scan to the final pixel.
First, what are holograms?
In strict optical terms, holography records information about a light wavefront and reconstructs it through interference and diffraction. Dennis Gabor received the 1971 Nobel Prize in Physics for the invention and development of the holographic method. His work describes a process in which object light and a reference beam create an interference pattern that can later reconstruct an image with depth.
In entertainment and experiential production, the word is used more broadly. Many displays described as holographic use reflection, projection, transparent media, LED surfaces, or volumetric techniques rather than classical wavefront holography.
That distinction is useful, not pedantic. It affects the asset format, camera setup, viewing zone, latency target, lighting plan, and whether the content can react in real time. A detailed introduction to what are holograms is a useful starting point, but a production team must still identify the exact display technology before building the character.
A believable holographic figure begins as a coordinated character, performance, rendering, and display pipeline.
The word “hologram” covers several display systems
A production brief may say “hologram” while referring to one of several very different systems.
True optical holography
Classical holography records and reconstructs light through interference and diffraction. Modern computer-generated holography calculates wavefront patterns for spatial light modulators or related optical systems. This is the technically strict use of the term.
The engineering challenge is optical. The system must reconstruct light with the right phase and direction so the viewer receives depth cues from the intended viewing zone.
Stage projection and reflection illusions
Large entertainment productions often create a human figure using a bright rendered image, carefully controlled black levels, and a transparent or reflective surface positioned relative to the audience. The content may be pre-rendered video or a live engine output.
The engineering challenge is compositional. Camera perspective, stage geometry, background darkness, brightness, reflection angle, and performer blocking have to work together. The audience is not viewing a free-floating object from every position. It is viewing a highly controlled illusion from a planned zone.
Volumetric and light-field displays
Volumetric systems generate visible samples within a physical volume, while light-field displays deliver different rays or views in different directions. These approaches can support more natural parallax, but they introduce constraints around resolution, capture format, compute, viewing area, and display size.
Before production begins, identify which display method the project actually uses.
A studio should therefore ask one question before discussing character quality: What does the display need from the content?
Why human holograms are difficult
A floating logo or abstract product can tolerate simplification. A human face cannot.
Viewers are highly sensitive to facial identity, eye direction, speech timing, posture, weight shifts, and emotional coherence. A few frames of lip-sync drift may be more damaging than a moderate reduction in texture resolution. A perfectly modeled face can still feel wrong if the smile shape, blink rhythm, or head motion does not match the performance.
The term human holograms therefore describes more than a display effect. It implies a digital-human pipeline capable of preserving a recognizable person or creating a believable new performer.
That pipeline has to solve four linked problems:
- Identity: Does the model preserve the face, proportions, age, and defining traits?
- Deformation: Does the face and body move correctly from every required angle?
- Performance: Do speech, gesture, expression, and timing feel intentional?
- Display translation: Does the final image survive the brightness, contrast, perspective, and viewing constraints of the venue?
The scan-to-stage production pipeline
1. Define rights, intent, and display constraints
Before scanning or modeling, define who controls the likeness, which performances are authorized, where the asset may appear, and how long the rights remain valid.
The technical brief should also capture:
• display type and physical dimensions
• expected viewing distance and audience zone
• frame rate, resolution, and color format
• pre-rendered or interactive behavior
• live performer interaction
• network and compute limits
• fallback behavior if the system fails
A beautiful character built for the wrong camera and viewing geometry is not production-ready.
2. Capture or construct the digital human
A real person may be captured using photogrammetry, facial scanning, structured light, calibrated photography, or a combination of these methods. Historical or unavailable subjects may require reference-based reconstruction from photographs, footage, casts, and expert review.
The resulting geometry is usually too dense or inconsistent for animation. Artists retopologize it into a mesh with deformation-friendly edge flow, create UVs, and build texture sets for skin color, roughness, normal detail, displacement, eyes, teeth, and clothing.
3. Build facial and body rigs
The body rig needs stable shoulders, hips, spine, hands, and feet. The facial rig needs enough control to reproduce speech and identity-specific expressions without turning the face into a generic puppet.
Common facial approaches include:
• blend shapes or pose-space deformations
• joint-based controls
• muscle-inspired solvers
• scan-derived expression shapes
• hybrid rigs that combine several methods
The rig should be tested with the target performance range early. A rig that looks correct in a neutral pose may fail under a wide smile, compressed lips, cheek movement, or rapid speech.
4. Capture and direct the performance
Body motion capture provides timing, weight, and gesture. Facial capture records expressions, mouth movement, head motion, and sometimes eye behavior. Voice may be recorded simultaneously or replaced later.
Raw capture is not the finish line. It needs cleanup, retargeting, contact correction, timing adjustments, and artistic direction. The performance must also fit the display. A subtle gesture that works in a close-up may disappear when the figure is ten meters from the audience.
5. Simulate hair and clothing
Hair and fabric communicate scale and motion. They also create expensive simulation and rendering problems.
A pre-rendered show can use higher-cost strand hair and cloth simulation. A real-time system may need groom simplification, cards, bone-driven secondary motion, reduced collision, or baked caches. The correct method depends on whether the character must react live.
6. Light for the destination, not only the render
A holographic stage image often needs deeper blacks, carefully shaped rim light, and stronger silhouette separation than a normal film render. Brightness and contrast have to survive the display medium and ambient venue lighting.
The team should test on the actual hardware as early as possible. A standard monitor cannot reveal all projection artifacts, viewing-angle limits, reflected light, ghosting, or black-level problems.
7. Adapt perspective and framing
The virtual camera must correspond to the physical stage and audience. A mismatch can make the figure appear tilted, stretched, floating too high, or disconnected from the floor.
For a fixed-view illusion, lock the camera to the intended audience perspective. For multi-view displays, prepare the scene and capture volume for the required parallax range.
8. Integrate show control
The final character must synchronize with audio, lighting, stage automation, live performers, video servers, or interactive systems. Timecode may be sufficient for a pre-rendered piece. Real-time experiences need events, state management, monitoring, and cancellation behavior.
The pipeline has multiple handoffs. Identity and timing can be lost at any one of them.
Pre-rendered or real-time?
The choice changes almost every downstream decision.
Use pre-rendered content when:
• the performance is fixed
• maximum image quality matters more than interaction
• the venue requires deterministic playback
• the show has tightly rehearsed timing
• compute or networking is limited
Pre-rendering allows heavier simulation, more samples, complex compositing, and careful frame-by-frame review.
Use real-time content when:
• the character must respond to an operator or audience
• dialogue changes between performances
• the system reacts to sensors or live data
• the character shares a stage with unscripted participants
• personalization is central to the experience
Real-time work trades some visual headroom for responsiveness. Asset budgets, shader complexity, hair, cloth, lighting, and animation systems must fit a stable frame budget.
A hybrid design is often best. Render the visually expensive sequences in advance, then switch to a real-time character for interactive sections.
Synchronizing a real-time holographic character
A live digital performer is a distributed system. Audio, dialogue, facial animation, gestures, rendering, stage cues, and display output may run in different processes or machines.
The safest design uses a shared clock, unique response IDs, and explicit cancellation.
Here is a simplified event model:
type HologramEvent =
| { type: "speech.started"; responseId: string; atMs: number }
| { type: "audio.chunk"; responseId: string; atMs: number; url: string }
| { type: "viseme"; responseId: string; atMs: number; value: string }
| { type: "gesture"; responseId: string; atMs: number; name: string }
| { type: "lighting.cue"; responseId: string; atMs: number; cue: number }
| { type: "response.cancel"; responseId: string; atMs: number }
| { type: "response.completed"; responseId: string; atMs: number };
Every component should reject events whose responseId is no longer active. That prevents late audio, lip-sync, or gestures from a previous turn from reaching the stage.
A cancellation path might look like this:
function cancelActivePerformance(responseId: string): void {
audioPlayer.stop(responseId);
faceController.clear(responseId);
gestureController.blendToIdle(responseId, 120);
showControl.cancelCues(responseId);
renderer.setListeningState();
}
The important point is architectural: stopping the language response is not enough. The audio, face, body, lights, and stage cues associated with that response must stop together.
Common failure modes
Where hologram studios add value
Specialized hologram studios sit between character production, VFX, animation, display engineering, and live show delivery. Their value is not simply creating a 3D person. It is translating that person into the visual and operational constraints of a particular holographic system.
That includes decisions such as:
• whether digital face replacement is preferable to a full digital body
• how much facial detail the final display can preserve
• which hair and cloth methods will survive the frame budget
• how to light for transparency and black levels
• how to calibrate camera perspective to the venue
• how to monitor a live performance and recover from failure
Mimic Productions, for example, describes a workflow that combines 3D scanning, facial rigging, animation, motion capture, digital face replacement, and holographic production for entertainment, exhibitions, and brand experiences.
For readers who want a wider overview of applications, types, and limitations, this guide to holograms covers entertainment, museums, corporate events, retail, education, medical visualization, and industrial use.
Rights, disclosure, and digital resurrection
Reconstructing a living or deceased person is not only a technical project. It raises questions about consent, estate approval, performer rights, voice rights, cultural context, and the limits of the authorized performance.
A responsible production should document:
• who authorized the likeness and performance
• which source materials may be used
• whether synthetic voice or dialogue is permitted
• how the character will be labeled to audiences
• which edits require additional approval
• where and for how long the asset may appear
• how the files and training data will be secured
The more lifelike the result, the more important the disclosure. Audience members should be able to understand whether they are seeing archival footage, a digital recreation, a live actor with face replacement, or an interactive synthetic performance.
A practical preflight checklist
Before final delivery, test the complete experience in venue-like conditions.
Character
• facial identity holds across the required expression range
• eyes, teeth, tongue, and lips survive close views
• body proportions and floor contact are believable
• hair and clothing remain stable for the complete show
Display
• camera and perspective match the physical setup
• black levels and edge treatment are clean
• the audience viewing zone is documented
• ambient lighting does not destroy the illusion
• color and brightness are tested on final hardware
Performance
• dialogue, lips, gesture, and body timing stay synchronized
• transitions into and out of the figure are rehearsed
• live performers know the figure’s apparent position
• interruption and operator controls are tested
Reliability
• playback or engine health is monitored
• a fallback state exists for network or model failure
• the show can recover without restarting the venue system
• logs identify which cue, response, or asset failed
Rights and safety
• likeness, voice, music, and source-material rights are cleared
• the audience receives appropriate disclosure
• interactive dialogue has a defined scope
• a human operator can intervene when needed
Final thoughts
The phrase “human hologram” makes the result sound like a display trick. In practice, the display is only the final layer.
A convincing performance depends on an end-to-end system that protects identity, deformation, timing, visual quality, and rights across many handoffs. The best projects begin with the display constraints, build the digital human for those constraints, and test the complete experience early.
For developers, the main lesson is familiar: optimize the system, not only the component. A stronger face model cannot fix bad time synchronization. A faster dialogue model cannot fix poor stage calibration. A brighter projector cannot fix an unconvincing performance.
When character craft, real-time engineering, and show control are designed together, the technology becomes less visible. What remains is the intended effect, a believable presence sharing the room with the audience.
About Mimic Productions: Mimic Productions is a Berlin-based 3D studio specializing in character creation, scanning, motion capture, animation, real-time assets, and digital-human production.
