Tomas Barnas of Overhead4D shows how Machu Picchu was turned into a multiplayer VR experience for more than a hundred people in one venue. Tomas explains how photogrammetry from about 30,000 images, Gaussian splats and satellite data were combined with historians' reconstructions and heavily optimized for standalone headsets, and how cartoon-style avatars, motion capture and an AI-dubbed guide voice turned the scans into a story.
“But here with Gaussian splats, that's totally changing because you are getting the entire environment”
“we have to sometimes also go back to the past to use the future technologies.”
“the VR industry is here for many years, but it still didn't reach its best years”
Tomas said the data ran into terabytes and the whole project took around a year, with roughly three months of processing.
According to Tomas, clients ask for the best possible reconstruction rather than for splats; Overhead4D uses splats in almost every project, so they are paid for but not requested.
Tomas recommended Unreal Engine, where about 99% of Overhead4D's projects end up, after photogrammetry, Gaussian splatting and modelling software.
Transcribed automatically (ElevenLabs Scribe) and lightly edited. It may contain small errors. Click a time to play the video from there.
Tomas Barnas: Hello again. I was lucky to get a chance to speak as a first. I don't know who was choosing that. Uh yeah, it was a coincidence. So I will be speaking, uh, today about, uh, cultural heritage, as this is something what my heart beats for.
Tomas Barnas: And, uh, we can launch the presentation. Uh, what I will, will be talking about is the digitization, but not only digitization as, as maybe, uh, we get it in mind as a first thing. So 3D scan something and get the data, but also how to use those, those data and, uh, what we can get out of them. How we can bring them back to people, because this is very important topic. And not only how to create a digital copy, but actually how to bring it back to people that they can really learn about it, use it, and, uh, yeah, educate themselves.
Tomas Barnas: What you can see here is, uh, just a quick, uh, pre-process or let's say like a first step of, uh, of a data processing, uh, which is the registration of, uh, s-- around thirty thousand images in RealityScan. And you can see that we have like denser spots, more tighter spots. So that all depends on the idea of how you want to scan something. Because if you are getting like a huge space, large area as Machu Picchu, as, as Peru, as any other places, you always have to think what you need and what you don't need. Because just scanning something that huge in, let's say, one GSD, so let's say one centimeter per pixel, would be crazy.
Tomas Barnas: You would need like hundreds of thousands or millions of images, and not only thirty, uh, thousand as we had here. So you see the preview of the, of the first processing and, uh, our 3D scan, 3D model from the photogrammetry, which we are further enhancing and, uh, designing it to a real experience, which I will show you during the presentation. But here you can see some previews on where we are getting to. Uh, it's not only again about photogrammetry, it's all about combining all the technologies. You can see here also Gaussian splats processed from the same place, from the same images.
Tomas Barnas: So again, we can talk about it that we can use one registration, we can create one set of data, and we create-- we can recreate in two, two ways. Why do we have Gaussian splats? For what we have photogrammetry? That all will be explained later. But as you can see, you have different visuals, different results, different outputs, which you can combine on the end into one, uh, mm, thing, project which works together.
Tomas Barnas: This is a preview of a Gaussian splat, uh, where you can see that we are able to relight it. Those of you who are working with Gaussian splats, you know it's possible. So the big-- the biggest advantage of having something like this is that before, when we were creating photogrammetry model, you were always focused on certain area, on certain model. That means focusing on that spot which you are capturing. But you never got, uh, all the surrounding, all the context.
Tomas Barnas: But here with Gaussian splats, that's totally changing because you are getting the entire environment, and instead of using three-sixty images on the backdrop scene, you are able to recreate them from Gaussian splats. But as I said, like this project was not only about scanning. It was only, uh, uh-- it was about recreating the story. And recreating the story means we need also more context than just that what is there today. And we wanted to show to people, uh, how the place looked like before, how it looked like in the sixteenth century, seventeenth century, when Incas were living there, and what they were doing.
Tomas Barnas: Uh, for doing something like this, you need a research. You need people which know how it looked like. You need the entire reconstruction. How the building looked like inside, outside, what was, what was there, how people lived. Uh, we wouldn't be able to do that without historians which, uh, have a knowledge about these places.
Tomas Barnas: But, uh, if you look at it later, when we combine 3D scans, Gaussian splats, and, uh, new roofs, new buildings, everything how it was reconstructed, we lit it in Unreal Engine. Uh, you are getting visuals, which are nice. You can get close to photorealism if you, if you want. Uh, but what you see here is a video. But just showing videos would be not immersive enough for nowadays time.
Tomas Barnas: And, uh, that's why we needed to push it further. But before that, also what you see here, it's that we have got lighting. Lighting which is referring to sunset, sunrise on that place. And again, we've scanned the top of Machu Picchu, but not the entire mountains. How you get the right lighting, how you get the right shadows, you need a context.
Tomas Barnas: So the context, I was speaking about photogrammetry data, Gaussian splats, and here we have a satellite data. So, uh, we had to get like, uh, Airbus and Maxar data from a satellite. Where you see the circle in the center, that's our area which we have scanned. The rest was used from satellite images, reprocessed, combined together. So if we are doing the sunset or sunrise, you really see the shadows from the mountains around because they are even higher than, uh, uh, than that what we have digitized and scanned.
Tomas Barnas: Then setting up everything in the softwares, getting the lighting, try to simulate everything based on the proper hours. It's a very complex thing to get the right lighting. But that wouldn't be challenging enough if that would be on high performance PC. But the target of this project was to get it on the headset and on the headset only, not streamed experience. And this is something which is, uh, what is bringing us back to '80s, how we were developing games in the past times, because the headsets are actually, uh, phones, Androids.
Tomas Barnas: So they don't have really GPU performance. They don't have a performance where you can just display millions of polygons of data, which we are easily gaining by scanning. So the choice which headset, I don't wanna talk about this, but, uh, it's just about the experience you are planning to do and you are planning to develop. And, uh, I will just, uh, start to talk about it. What was the end result?
Tomas Barnas: So end result of this project was to develop an experience for hundred plus people in one venue synchronized. That means you are getting into the venue, you are picking up the headset and like, instead of sitting in the cine-cinema, you are walking in a cinema. So you are getting headsets, walking around with your fr-friends and not hitting the others. So you need to be correctly synchronized and that was a huge challenge. And therefore I have these two headsets where you have di-different technologies which are suitable for different headsets.
Tomas Barnas: As I said, like I will go further. Uh, as I said, we are here, uh, speaking about extreme optimization, which is bringing us back to past times how we were developing games. And the target here was as well the same, how to get 3D scan data into the headset, how to get a gigabytes of data into one headset, which is impossible. So unfortunately, many thing need to be retopologized, then baked with normal maps, with lighting. So traditional gaming procedures where you are faking.
Tomas Barnas: You are faking the shadows, you are faking the lighting, you are bringing everything into low polygon models. And, uh, if you're doing it correctly, then you can always compare and see that, okay, it is different, but still you are gaining enough of information and it looks good enough for going further, but you are extremely optimized. So going through those places, optimizing it step-by-step. Uh, but it wouldn't be only about optimization the places as they are. But as I mentioned on the beginning, we were reconstructing it into the past.
Tomas Barnas: So first thing you would think it's okay, we brought the roofs on top, we brought some new buildings, objects. But look at that. They built their stairs, platforms, all of that needed to be removed, replaced, and that made it also extremely complex. The same thing happened with the textures. We have things which are there artificially made nowadays needed to be removed.
Tomas Barnas: So a lot of manual work. And on the end, as we are on the headsets, we can't do our nice real-time lighting and everything need to be baked in. I'm going a little bit more technical and out of scanning, which is-- or, uh, Gaussian splats, which is our core topic here. But I also want to bring this, that if we're creating storytelling, sometimes it's not only about our, uh, core tools and core technologies, but we have to combine it and we have to sometimes also go back to the past to use the future technologies. But telling the story wouldn't be big enough without bringing people in.
Tomas Barnas: And as this experience was supposed to be for kids as well, uh, we decided to go with a artistic style, so more cartoon-like figures. And again, not just making any kind of figures, but again, speaking with historians and fulfilling the story with the research, checking what people were wearing, how they were living, what they were doing. And all of that was translated then in the art, sketches, models, and step-by-step building up the cartoon movie. Uh, an experience which is educating people and bringing them back to the history and to historical topics, because I really think that it's important to also bring to newer generation the, the historical informations in the right way and in the way they would like to hear it and see it. So if you combine things, you can have a photogrammetry scans, you can have a cartoon style avatars, and you can bake it together.
Tomas Barnas: But you need to also do movements. Again, motion capture, recreating all the, the, the movements which are then brought to those avatars, uh, replacing or reconstructing their movements, how they were, mm, doing the work, how they cut it stone, and all of that. So that was very, very fun part of the project because you need to really do everything what people were doing back then. Doing all the assets, all the details, building up the scenes in order to recreate an atmosphere and scene where you feel good, you feel like you are in the past and that you are part of the story. Uh, you will see there, and you can see already there is like a little dot flying.
Tomas Barnas: As it was supposed to be for kids or for like the wider audience, we've decided to make a story like traveling in time because the target is not to reconstruct only Machu Picchu, but more of the heritage places. And we want to bring people through the travel agency into the past, so traveling in time. And we have our little robot which is guiding you, talking to you. And this robot we had very... Can we...
Tomas Barnas: Yeah.
Host: The stunning city was born around the year 1450 as a-
Tomas Barnas: We were lucky that the, the voice for the robot was Terry Crews, for those who know. So he was like funny Very, like, cool person matching to the robot. But, uh, as the experience is played all around the world, we had to play with the voiceovers. And here we come to the topics of AI. So making one, uh, voice with a real actor, so he was really recording it, is one thing.
Tomas Barnas: But if you want to keep the story and you want to keep the feelings across the entire experience and entire world, you need to play with the, with the voices. So- So then we used AI, which we trained based on his real voice, real he, uh, his real text and replacing it into different languages. Also Mandarin. Yeah. So also things like that could be part of the story, part of the project where you have to think broad.
Tomas Barnas: It's not only about the scanning. If you want to tell the story, you need to cover all parts of your projects or all parts of the senses people are looking at. And this is one of the last scenes or one of the scenes you can see. Uh, as I said, like, this is an experience for, uh, multiplayer, for many people coming into one place. It means you also see your friends.
Tomas Barnas: You see the avatars around you, which are walking there with you, and, uh, you can, like, communicate with them. You can show them something. You can interact with things. And, uh, this is kind of a view from multiple scenes from the projects, which all are based on the, on the scans and reconstructions. But as I mentioned, this experience was made for 100 plus people in one venue, and this, uh, requests or needs extremely robust background.
Tomas Barnas: So it's not, again, not only about visuals and how we use the scans here. We didn't scan only Machu Picchu. We are scanning every place where it goes to match the place, to put people there, to avoid any collisions with the walls. So, uh, setting up the venue looks like this. You are preparing the walls.
Tomas Barnas: We need, uh, markers which are helping with tracking. Uh, we have some onboarding headsets. And here, just like a sneak peek from various venues as we have opened it already on 10 places around the globe from Singapore, Washington, São Paulo, Vienna, so a few places in Europe, US, Asia. So this is, like, growing and growing, and I think that the VR industry is here for many years, but it still didn't reach its best years, and this is, like, slowly coming probably now, so you can see how it can look like as well. Yeah.
Tomas Barnas: And that's about me and Machu Picchu.
Host: Whoo! Tomas, thank you so much, and dear participants, thank you so much for your questions. Uh, 16 questions have been submitted yet. So let's bring them up over here on the screen, and let's tackle them one by one. Uh, there are plenty of them, so we will not have enough time to cover every single one of those questions.
Host: Please upvote the ones that you want to hear Tomas to answer. All right, so let's start with the first one. How many gigabytes was project and raw scan data of Machu Picchu that you just presented?
Tomas Barnas: Terabytes. Like, I don't know exactly, but we are always working with projects which are reaching terabytes. So yeah, one, two, three, it-- then you stop counting.
Host: And that naturally leads me to the next question. Uh, what was the processing time on this?
Tomas Barnas: So the entire project took us, uh, around a year to, to do it, like, from the-
Host: Mm-hmm
Tomas Barnas: uh, 3D scanning, processing, and, uh, reconstruction. But, uh, I would say like around three months, maybe, processing.
Host: Mm-hmm. Three months.
Tomas Barnas: Looking at Mario, which was, like, um, my colleague, which is processing that.
Host: Cool. Uh, so let's talk about money.
Tomas Barnas: Sure.
Host: Are you actually making money, uh, making money making 3D, uh, Gaussian splattings?
Tomas Barnas: Um, I had already this question somewhere, but, uh, the point is the clients are not coming to us to ask for Gaussian splats. They are asking us to create a reconstruction of the heritage of any places, to create a project in the best way possible. And whether we are using Gaussian splats or not, it's more or less our thing, and we do. So we do it almost in every project right now, and, uh, therefore, I would say like, yes, we are getting money for it, but we are not getting requests for it.
Host: Mm-hmm. There was a, there was a recent question, uh, that once again touched on the topic of money. How was it funded?
Tomas Barnas: Uh, so the, the entire project was made for a private client, which is providing these venues. So their strategy is to, uh, open as many venues as possible, and it's kind of a Netflix-like style, I would say. We just look at, uh, looking at another, uh, type of, uh, experience. So normally you have movie, you have cinemas, and now we are having like a VR cinemas.
Host: Awesome. Uh, again, there was a, there was a question where people can experience this. Where is the closest location from here?
Tomas Barnas: Vienna is the closest.
Host: From Vienna.
Tomas Barnas: Mm-hmm. Yeah.
Host: Cool. Uh, talking about the production process, what kind of permissions did you need to get for this project?
Tomas Barnas: So here it was easy for us because we actually got the images which were captured already before for another, uh, reasons, and, uh, we could work with those, so we didn't really have to, uh, arrange any permissions there. But normally, everybody who is flying with the drones knows what, what you need. You need to get the permissions from all authorities to get, uh, to be able to fly.
Host: Okay. Um, let's take one more. What, what's the best engine to learn in to start with a project like yours?
Tomas Barnas: Unreal Engine.
Host: Unreal Engine. Cool. And the last one, uh, you said that you had a customer. Uh, so, uh-
Tomas Barnas: We have people from, uh, Epic Games here, so.
Host: Oh, really?
Tomas Barnas: But it was true answer. Like, uh, we do like 99% of our projects like ending in, in Unreal Engine. But you can start with, uh, any kind of softwares. You need to go through like photogrammetry softwares, then you have some Gaussian splat softwares, then, then, I don't know, modeling, and then ending up, ending up in an engine. So there are many options.
Host: Awesome. Uh, let's look into the future. Um, are there an-- are there plans to scan any more fam-- uh, familiar sites and objects?
Tomas Barnas: Yes.
Host: Can you, can you, can you tell us more?
Tomas Barnas: No.
Host: Okay. So we have time for one more because you keep the answers real short. Um, are you hiring?
Tomas Barnas: Um, depends. Yes, always.
Host: Okay. So we're talking about one connection changing your life. Meet Tomas if you are interested in joining one of his projects. Let's give it up for Tomas Barnas.
Tomas Barnas: Thank you. Thank you so much.

Founder and CEO at Overhead4D and Co-Founder of 3DISE
An expert in reality capturing technologies and the Co-Founder of Overhead4D, a company that specializes in “TURNING THE REAL WORLD INTO DIGITAL EXPERIENCE” focused on blending innovative technologies with creativity. Im an expert member at CIPA Heritage Documentation and passionate about educating…