3D/4D Digital Twins Utilizing 3D Gaussian Splatting

Talk · Yoshiharu (Josh) Sato · 3DISE 2025 · 25 April 2025 · 10:10 - 10:30 · 19 min

Chapters

What the talk is about

Josh Sato, founder of Waldek Technologies, explains how 3D Gaussian splatting works, from rasterization and alpha blending to gradient-descent optimization, and why it can beat photogrammetry for fast, immersive digital twins. Josh then shows Gauzilla Pro, a browser-based splat editor used for 4D construction time lapses (including a proof of concept for Tesla), AI object segmentation and splat editing.

Key ideas

  • Gaussian splatting builds a scene from millions of semi-transparent 3D Gaussians that are rasterized and optimized against the photos by minimizing photometric loss, so by default it involves no neural network.
  • According to Josh, splats offer low capture cost (phones, 3D cameras, drones), reuse of existing photogrammetry datasets, and convincing reconstruction of water, glass, foliage and surroundings.
  • Weekly drone videos of Tesla's Gigafactory expansion in Austin were turned into splats, aligned and combined into an interactive 4D time lapse of the construction progress.
  • Josh's team aligns the separate scans in RealityCapture with ground control points before stitching them into 4D models; overlaying or splitting scenes helps owners, architects and contractors track progress and avoid rework.
  • Gauzilla Pro runs AI segmentation, a real-time sharpness filter and object import (including generative AI splats) in the browser via WebGPU and WebAssembly, so users install nothing and no GPU servers are needed.
“So there is actually no AI per se, uh, in Gaussian splatting, at least by, by default.”
— Yoshiharu (Josh) Sato ·
“construction companies lose billions of dollars every year due to rework, so it's vital for them to reduce such costly rework.”
— Yoshiharu (Josh) Sato ·
“there is no expensive GPU server costs because all the GPU processing, uh, takes place on the, uh, on the client side.”
— Yoshiharu (Josh) Sato ·

Questions answered

Does Gaussian splatting use AI or neural networks?

Not by default, according to Josh Sato: splats are fitted with classic mathematical optimization (gradient descent on photometric loss), which is why scenes can be rendered and edited in real time.

Why use Gaussian splatting instead of photogrammetry for digital twins?

Josh Sato lists low capture cost with consumer devices, reuse of photogrammetry datasets, faster capture and processing for 4D models, and better reconstruction of water, glass, foliage and the surroundings.

How are 4D construction time lapses made from Gaussian splats?

Josh Sato's team turns each weekly drone video into a splat, aligns the scans in RealityCapture using ground control points, and combines them in Gauzilla Pro into an interactive time lapse.

Full transcriptabout 12 min readShowHide

Transcribed automatically (ElevenLabs Scribe) and lightly edited. It may contain small errors. Click a time to play the video from there.

Yoshiharu (Josh) Sato: Thank you. Uh, thank you everyone for being here today. Uh, my name is Yoshi Sato, and I'm the founder of Waldek Technologies. My company is specialized in radiance field technologies, uh, mainly Gaussian splats. So, um, this is our Gaussian splats in 4D dynamic time lapse, uh, which was generated using our Gauzilla Pro, an AI-powered Gaussian splat web editor, uh, for 3D and 4D digital twins.

Yoshiharu (Josh) Sato: So you can visualize all these dynamic construction progresses, uh, in fully instructive Gaussian splats. And you can see all the details, not just in the building, but also, uh, those beautiful surroundings. Uh, by the way, this is a, a construction project in downtown Chicago. Uh, it's a Skender project, and you can see all these, uh, yeah, beautiful, uh, Chicago skyline, uh, in the, in the background. So just in case you were wondering, uh, who this, uh, bearded guy is, um, my name is Yoshi Sato, and again, I'm a founder of Waldek Technologies.

Yoshiharu (Josh) Sato: Um, my company specializes in radiance fields, uh, Gaussian splats, as well as AI, uh, including, uh, transformers, reinforcement learning, and spatial intelligence for embodied, uh, AI. Um, my company is also a Techstars, uh, startup. Techstars being one of the, uh, uh, world's largest and most active, um, venture capitalists, uh, in the world, investing in startups, uh, globally. And I got two decades of experience, uh, in 3D graphics and AI machine learning. And I was a, I was formerly a quant, uh, working in the financial industry for hedge funds and trading firms, applying statistical and machine learning techniques, uh, on, um, peta-- gigabytes and terabytes of, um, multi-dimensional, uh, time series data.

Yoshiharu (Josh) Sato: And Gauzilla Pro is what my company has been working on for some time now, and it utilizes Gaussian splatting for instructive and high-fidelity 3D and 4D rendering and editing. It's a fully web-based platform, uh, making it, uh, installation-free for users. So, uh, as a user, you will not have to install anything like Python library, PyTorch, NVIDIA CUDA, all these different versions. So it's, it's, uh, installation-free and hassle-free. Uh, it's also got, um, edge AI models, uh, that run directly inside web browsers.

Yoshiharu (Josh) Sato: So you can do, uh, for instance, seg- 3D segmentation and annotation, uh, of 3D objects. Or there's also real-time sharpness filter, uh, to make splatted image more crisp or sharper. And there's an intuitive splat selector, uh, that you can see on the-- in, in the video, where you can select all these complex geometries, uh, using a very intuitive UI. Uh, by the way, this is a, a, uh, a botanical garden in Germany, uh, captured by Simon Bethke. So yeah, it's a beautiful, uh, splat scene.

Yoshiharu (Josh) Sato: And although my company is very small and, uh, uh, very young, uh, we've done several POCs already, uh, with companies like Tesla, Volkswagen, as well as CoStar Group, uh, which is now the owner of Matterport. And those usage cases include, uh, AC construction, construction projects, also facility management of factories or marketing, sales of real estate, and also GIS, uh, on large scale, uh, terrains. And, uh, I'm going to, uh, talk about some of these, uh, use cases, uh, in a, in a minute. Uh, you might be wondering why Gaussian splatting, uh, why not photogrammetry? Uh, don't we already have a matured ecosystem, uh, for, uh, for doing photo, uh, photogrammetry?

Yoshiharu (Josh) Sato: To answer that question, uh, let me quickly recap what Gaussian splatting is and how it works. So Gaussian splatting is a, a, uh, a novel view synthesis technique for generating new images from unseen camera poses. Uh, it uses explicit rendering primitives, meaning that, uh, uh, yeah, it's a 3D Gaussian ellipsoid. So the idea is we compose a 3D scene using millions of those Gaussians, and then we use, uh, rasterization, uh, and alpha blending, uh, which are, uh, you know, conventional, uh, rendering techniques, uh, and the GPU pipeline to render all these semi-transparent Gaussians. And we project those, uh, uh, Gaussians, uh, on the, onto the screen, so that's why it's called Gaussian splatting.

Yoshiharu (Josh) Sato: Uh, it's not Gaussian, uh, skunking or an-anything. And, uh, yeah, so... And this is the equation. Uh, so if you're a 3D graphics, uh, engineer, you might-- so this equation might look familiar already, uh, because it's, uh, it's almost equivalent to the, uh, the discrete volume rendering, uh, integral. Uh, the only difference here is that we get the term G 2D, uh, which denotes the 2D Gaussian, uh, density at the pixel ax.

Yoshiharu (Josh) Sato: But the difference is that unlike, uh, the, uh, the per-pixel ray casting, uh, in the volume rendering, in 3D Gaussian splatting, we use the traditional, uh, rasterization, uh, and alpha blending. So, uh, it makes it, uh, uh, high performance, and you can achieve, uh, real-time, uh, frame rates. Um, importantly, Gaussian splatting is differen-differentiable, which means that, uh, we can compute the, uh, the derivatives or gradients of the splatted image with respect to all these input, uh, parameters. Uh, you can do, uh, 3D reconstruction, uh, using machine learning on the GPU. So how does the machine learning and Gaussian splatting work?

Yoshiharu (Josh) Sato: The basic idea is we compare the difference or compute the photometric loss between the ground truth image and the splatted image, and then we optimize the, those 3D Gaussian parameters, uh, in order to, uh, minimize those photometric, photometric loss. So those 3D Gaussians parameters includes, uh, 3D position of those Gaussians in the scene or rotation, uh, as a quaternion or the scale or, or the color, uh, represented as spherical harmonics or coefficients, coefficient, coefficient, and also the opacity. So here's the, uh, diagram, uh, of this, uh, op-optimization process. So first, uh, we initially get 3D point clouds generated using SFM like COLMAP, RealityCapture. Then we initialize the scene by placing d- a default 3D Gaussian at each, at the, uh, location of those, uh, each of those Gaussians.

Yoshiharu (Josh) Sato: Then from all the camera poses, uh, which are also estimated using, uh, uh, you know, SFM, we project, uh, those Gaussians onto sc- onto the screen using the, uh, the rasterization algorithm, uh, that we looked at, uh, previously. Then we compare and compute the, uh, photometric loss, uh, between those pairs of images. And then we can compute the, uh, what's called, um, gradient. Then we can pro-propagate, uh, those Ga-gradients and do, uh, what's called, um, gradient descent, which is, uh, uh, traditional machine learning to progressively update those 3D Gaussian parameters in the scenes so that we can minimize the difference, uh, between those, uh, images, the ground truth image, and the, uh, splatted image. So that's how the, uh, machine learning and the Gaussian splatting, uh, works.

Yoshiharu (Josh) Sato: And there's also, uh, uh, one additional, uh, optimization, optimization, uh, mechanism. It's called adaptive density control, density control, or ADC, uh, which dynamically adjusts the, uh, the number of Gaussians in the scene, uh, based on the, uh, the size of the splats, uh, on, on the screen. So back to the question, um, why Gaussian splatting? So first of all, we can do 3D re-reconstruction without neural networks, and as we have seen, uh, already. So there is actually no AI per se, uh, in Gaussian splatting, at least by, by default.

Yoshiharu (Josh) Sato: So i-it's all about traditional mathematical opti-optimization using gradient descent and machine learning. So there is no costly neural networks. So we can do real-time rendering and editing, uh, of those 3D scenes. And there's also a low data ac-acquisition cost, uh, because you can use almost any consumer devices such as smartphone, uh, 3D cameras, or flying drones. And there's also a high data, uh, reusability because you can reuse the dataset that you captured originally for photo-photogrammetry.

Yoshiharu (Josh) Sato: And you can also achieve a high scene fidelity using professional cameras or LiDAR scanners such as XGRIDS', uh, L2 Pro. And Gaussian splats also have expressive visuals, meaning that there is a, a wide variety of materials that we can accurately reconstruct, uh, such as water surfaces, glass windows, or semi-transparent tree leaves, as well as fluffy, uh, animals. And scenes that were reconstructed using Gaussian splats are also holistic and immersive because, uh, you get not just the main objects of interest, uh, at the center of the scene, but also, uh, you get all the, uh, surroundings, uh, in the background. So it's very, uh, immersive and gives you a better scene context. So that, so those are the reasons why, uh, my company is focusing on Gaussian splats and specifically are developing this Gauzilla Pro.

Yoshiharu (Josh) Sato: Um, by the way, this is our Gauzilla guy. Uh, it's a dinosaur, uh, with a back fin shaped as a one-dimensional Gaussian, uh, distribution. Uh, it's a, it's a nerdy joke. So one of the, um, major issues to which Gauzilla Pro offers a solution, uh, is costly and time-consuming 4D visualization. So traditional techniques like photogrammetry requires a large number of high-resolution images in order to generate, uh, uh, all these interactive 4D scenes.

Yoshiharu (Josh) Sato: So Gauzilla Pro, uh, makes it easier to, to make, uh, such as-built 4D models thanks to 3D, uh, Gaussian splatting, which is, uh, which has faster data capturing and processing time. And in fact, these 4D models are already utilized in the AEC or construction projects or for virtual tours, uh, in real estate, e-commerce, and cultural heritage. So you can see this, uh, uh, dynamic 4D, uh, virtual tour of a real estate, uh, apartment. And you can see this, uh, uh, dynamic, uh, 4D time lapse of a construction project. Uh, this one is a, uh, Clayco project, uh, in downtown Chicago.

Yoshiharu (Josh) Sato: So you can see all these dynamic, uh, intertemporal changes of this, uh, construction site. Oops, sorry. Let me go back. Um, oops. Okay.

Yoshiharu (Josh) Sato: So yeah, in fact, uh, we did a POC, uh, for Tesla, uh, last year. This is a, a, uh, as-built 4D model, uh, of their Gigafactory expansion project which took last year. So Tesla expanded this Gigafactory in Austin, Texas, and they documented, or they captured a drone video, uh, on a weekly basis, uh, in order to capture, document, record, analyze all these construction processes. And then we generated 3D Gaussian splats for each of those videos. Then we concatenated and aligned them together, uh, on Gauzilla Pro and generated this fully interactive 4D time lapse where you can observe all these dynamic construction progresses from any position, from any angle, because the underlying is the real-time 3D Gaussian splatting.

Yoshiharu (Josh) Sato: And it's a great way to visualize and a-analyze construction progresses, and also it enhances safety compliance and risk assessments in construction projects because all these 4D models contain valuable information regarding construction projects, such as, uh, those site logistics, lo-logistics, uh, temporary structures, and all these, uh, uh, construction phases that can be reviewed after the fact. So yeah, this is, uh, a fully interactive dynamic 4D. And we've been also doing closed beta with Skender, uh, which is a Chicago-based construction company, and this is a, this is an as-built 4D model, uh, of their construction projects, uh, in Chicago. So as you can see, you can, uh, overlay multiple scenes together, uh, on the same screen, or you can split the screen, uh, in half, uh, for detailed progress tracking and analysis. And it greatly, greatly enhances communications among OAC, uh, the owner, architect, and constructors, and it also helps reduce the risk of costly rework.

Yoshiharu (Josh) Sato: Uh, it's important because construction companies lose billions of dollars every year due to rework, so it's vital for them to reduce such costly rework. And Gauzilla Pro, uh, platform helps them reduce such, uh, re-risk of rework, uh, thanks to the, thanks to all this detailed high-fidelity 4D model. And those models are also useful for future modifications, renovations, as well as, uh, decommission, uh, decommissioning, uh, of those buildings. Uh, because again, these 4D models contain all the, uh, construction phases that can be reviewed, uh, after the fact. And since Capturing Reality, uh, is the, uh, is a tech, uh, tech, tech partner, uh, to 3DISE, uh, I should mention that, uh, we do use Reality Capture for aligning, uh, all these different point clouds together, uh, in order to generate these, uh, these 4D models.

Yoshiharu (Josh) Sato: And we use GCPs or ground control points, uh, to align multiple point clouds, uh, at the same time. And those GCPs are either geo-referenced, uh, with actual, uh, locations or, uh, they have synthetic, uh, Euclidean coordinates. And you can just, uh, uh, yeah, you can, you can, you can align all, all these point clouds together, uh, in a single click on Reality Capture. So it's a great, uh, tool and software, uh, for us to generate, uh, the, uh, that 4D time lapse models. And another major problem, uh, to which Gauzilla Pro offers a solution, uh, is, uh, uh, is scene understanding and 3D reconstruction.

Yoshiharu (Josh) Sato: So 3D reconstruction, uh, inherently lacks scene understanding, whether it be, uh, point clouds for the geometry, 3D Gaussians, uh, because the, uh, the underlying is just a monolithic collection of static points, meshes, or Gaussians. And it makes it difficult to rapidly make informed 3D digital twins, uh, for instance, uh, museums with a lot of exhibitions or retail stores with, uh, uh, with a bunch of products or items. So Gauzilla Pro, uh, utilizes vision transformers or AI models for, uh, dynamic 3D object segmentation. And everything runs inside a web browser, uh, thanks to the latest web technologies such as WebGPU and WebAssembly. And it achieves a, uh, full web-based edge AI inferencing.

Yoshiharu (Josh) Sato: And this means that, uh, the platform is fully installation-free for users. And for us as a service provider, uh, there is no expensive GPU server costs because all the GPU processing, uh, takes place on the, uh, on the client side. And on Gauzilla Pro, you can also do detailed segmentation, uh, of complex geometries. Uh, it's got an intuitive UI for, uh, manual 3D segmentation of objects like MEP pipes and cables at the construction sites or HVAC, uh, uh, uh, equipments, uh, heating, ventilation, air, air, air conditioning. And it can be also utilized in facility management, uh, uh, of complex, uh, machinery and equi-equipment, equ-equipment.

Yoshiharu (Josh) Sato: Uh, you can also, uh, use it for forestry, uh, where you need to segment and keep track of all these, uh, uh, different trees, uh, vegetation or, uh, or plants. So, uh, it's got a intuitive, intuitive UI where you can just need to drag along the object. Then you can segment, uh, Gaussian splats very, very easily. And although Gauzilla Pro is still in early beta, uh, there are, there are other, uh, useful features, uh, already, uh, such as object import and export from other Gaussian splatting scenes or from generative AI. So you can generate, uh, highly detailed 3D Gaussian splats from just a single image, and you can import that 3D Gaussian into your scene, uh, as you can see in the, in the, in the left video.

Yoshiharu (Josh) Sato: So this is a, a, uh, generative model. Uh, it's called Trellis. And you can generate, uh, these highly detailed 3D Gaussians from a single input image. And on Gauzilla Pro, you can segment, uh, you know, for instance, the, the skull of the, uh, triceratops, and you can remove it and then import and replace it with the, uh, the skull of the T-rex. So yeah, that's something you can do, uh, already, uh, on our platform.

Yoshiharu (Josh) Sato: And also we get, uh, a 3D Gaussian sharpness filter, uh, which works in real-time to enhance, uh, the, uh, the splatted image, uh, in order to make the, uh, uh, splatted image more crisp or sharper. Uh, because one of the, um, common complaints of Gaussian splats is actually blurred, uh, images. So this filter, uh, dynamically, um, enhances these, uh, splatted images. And also, uh, yeah, there's a camera animator where you can create, create, uh, smooth camera fly-throughs. And also we are working on large scale, uh, scene rendering, uh, which we are, we are work- we are working on to make a digital twin of, of a city.

Yoshiharu (Josh) Sato: Uh, those are large scale, uh, yeah, city, city scale, uh, scenes or you can also do it, uh, for doing GIS. And, uh, we are also working on, uh, other more useful features. So that's it. Uh, thank you very much everyone for listening. And, uh, if you're- Yeah.

Yoshiharu (Josh) Sato: And, uh, so if you or your company is interested in doing a POC or early beta, yeah, please feel free to reach out, uh, reach out. Thank you.

Host: Yoshi, thank you so much. Let's give it up for Yoshi one more time.

Yoshiharu (Josh) Sato: Thank you.

Host: Thank you so much. Well done. Well done.

Speaker

Yoshiharu (Josh) Sato

Yoshiharu (Josh) SatoLinkedIn

Founder & CTO/CEO at Waldek Technologies

Is a quant and machine learning expert with a background in high-frequency trading, blockchain, and 3D graphics. He holds a Master’s in Quantitative Finance from the University of Warsaw and studied ML at NYU. Josh has advised hedge funds and startups, focusing on deep reinforcement learning and…

Hear the next ones live – 3DISE 2027, 20–21 April, Prague

Buy tickets