For Gladys, we were able to do a 360 video experience of her neighborhood, but also recreate tools in her childhood home via photogrammetry. They were 3D models of her mom's room and of the living room where Gladys could actually walk around the space and even pick up certain objects and interact with certain objects in the space.

Gladys was disoriented when she put on the VR headset for the first time and saw her childhood home. Then she became emotional.

At first she couldn't recognize the neighborhood that she was in. She kept asking out loud, 'Where am I, where am I?' When she realized that she was standing outside of her mom's home, of her childhood home, immediately she got very, very emotional. When she was inside, there is a lot of wows, a lot of oh my Gods, and more tears.

Audio clip – emotional reaction

I'll go to the point where she saw her mom sitting in the living room. Seeing her mom in the room where they used to have birthday parties, Christmases, quinceañeras, all of that I think brought back lots of memories for Gladys. To see her in that room was truly special.

For separated families like Gladys's, the VR experience can be incredible. But Alvaro is clear on one thing. The Family Reunions Project is not meant to glorify technology, nor is it meant to be a replacement for the real thing.

If I put on my optimistic goggles, I think it can help learn borders and reduce the distance to home. But despite all of that, VR and all the technology's never really going to replace a physical, real embrace. With VR and especially these virtual unions are made possible by this amazing technological progress, and it's so frustrating because there's no political progress that matches it. The fact that we have to use VR to get Gladys, for example, to be able to be in the same room as her mother, that feels a little cruel. There's no world in which that makes sense, where someone having a headset over their face is the best way that they can be with a loved one. It's not a world that anyone should aspire to live in.

That was Alvaro Morales, co founder of the Family Reunions Project, and that clip you heard of Gladys is from a VR experience he directed called Home with America.

I'm Claire Evans and this is You, a podcast about the intersection of technology, humanity, and identity brought to you by Octa.

Today we're looking at virtual reality, both its possibilities and its limitations. Like, does it really expand our world or does it just divorce us from our physical realities? Can it do both? To answer these questions, we talked to a professional whose work and life is completely immersed in VR.

Hi, I'm Timoni West, and I'm the director of XR research at Unity Labs.

Unity Labs is part of a larger company called Unity. I started our conversation by asking Timoni to tell me a bit more about Unity and what they do.

Unity is a game engine, which means it comes with a lot of complicated computing systems that might be too difficult for a developer to make on their own, like a physics engine for example, or a graphics engine. It allows you to add interactivity to whatever world it is you're making. It's been used to make games like Firewatch and Flappy Bird and Pokemon Go. So if there is a game on your computer or on your phone, there is something like a 60 to 80% chance that this game was made using Unity.

What about your job specifically?

My job is to figure out two things, basically. First, what do developers need today that we haven't built yet that's specific to augmented reality and mixed reality? Then what will Unity look like in the future when we're all wearing headsets and we're all surrounded by digital objects that can interact with real world objects?

How far in the future are we talking here?

Not very far for virtual reality. Right now, the way you interact with all computers that are common and in consumer grade is basically a sheet of glass. On the other side of the glass is the digital world. On this side of the glass is you, and no matter what, you can't go through that glass, right? Like you just can't. You can touch the glass if you have a touch screen, but you can't go any further than that. Of course if you have a mouse, you're not even touching the screen. You're actually even further away. So being able to step into virtual reality is you stepping into that digital space and being able to interact with it.

It's all fake. In the end, it's still 2D because those are actually little like 2D monitors that are stuck to your eyeballs that give the illusion of depth. When it comes to augmented reality though, sort of the underlying philosophy is that humans want to interact with digital objects in a more real way. We want it to be more real. This is an expression of human imagination. That to me is what the digital world is.

But the reality is, the way we interact with computers today is not satisfying. It takes years to learn. It's not very fun. It's not very personalized. It's not very ergonomic or good for us or healthy. So if we can start to pull out three dimensional objects into real world space and interact with them like that and make them sort of part of our own worlds, then we not only have the advantage of genuinely being able to bring our imaginations to life and kind of live in this sort of fantastic and real world combined, but it will also be easier for us to learn and teach each other, be able to see something that is someone else's imagination, and we'll all be healthier as a result too.

Let's rewind just for a second. Imagine you're talking to someone who's never encountered VR and AR. Can you describe what they are?

Sure. Virtual reality is allowing a user to put two tiny monitors very close to their eyes that allows you to trick your brain into thinking that you are seeing depth. By allowing you to put on those two monitors next to your eyeballs and by handing you two controllers that are tracked, we can actually see where your head is moving in the space and where your hands are, which allows you to interact with the computer or the tiny little monitors on your head in a way that feels much more realistic than just interacting with a normal computer, which is basically a giant piece of glass behind which lies the digital world.

Augmented reality. That's interesting. There's like two schools of thought. There's Milgram's continuum of ... Okay, okay.

I love it. I love it. It's already so complicated just explaining what it is.

Well, it's more like who do I want to piss off right now? The purest mixed reality people or the kind of conventional people? When most people think augmented reality, they think something like Iron Man, where you can see a hologram and you actually interact with a digital world. But on the classic academic reality continuum, made by Milgram in the 1970s, we've got the real world on one side and then on the other side you have complete virtual reality, which is you in the digital world. In the middle there you've got mixed reality, which is mixing digital objects that can interact with the real world.

Reality is just consciousness I guess, and virtual reality, I think it's just a different kind of consciousness. I don't know. I wouldn't think about it that much differently. It's just kind of a game.

I definitely like the real world better than virtual reality because it gives me a headache.

Yes. I actually work at a VR arcade. Then I played over 140 games, all to completion, so I spent at least a couple thousand hours in VR. It really makes you think at night. I've definitely been woken up a few times pretending I was still in a game I was playing that day, like trying to box someone or trying to shoot invisible guns or robots or whatever. It's kind of interesting if you play enough of it. It really does blur the lines, which is why I like it.

Okay. This is maybe a little bit of a leap, but a lot of our onscreen experiences now are on these proprietary platforms, social media platforms mostly, let's be honest, that have a really kind of monolithic design style. Have very little space for variability or for individual expression. I long for the days of skinning my Win apps and different weird styles or making my Myspace page look heinously hideous with lots of weird extra cursors and stuff. But you know, there's not a lot of space for expression or expressivity in a lot of our online experiences these days. It seems like mixed reality, augmented reality ... What a word salad of different ways of expressing it.

The other reality.

All the other realities do open up the possibility for individual expression.

Yes, and I actually would argue that they demand it. This is something I'm very excited about.

I was thinking about this earlier today and I feel like so much of human identity, personal identity comes from the way that we relate to the static realities of the world. Like the landscape of where we grew up, or where we have chosen to live. Things that we all in common that we can relate to and define ourselves against. But when we're able to sort of manipulate reality to our liking, what does that do to our sense of self and how does it change our possibility of defining ourselves in relation to the world? If that makes sense.

Yeah. I love that question. So much of our motivations are tied to our identity and who we think we are through shared unconscious or unspoken connections that I have noticed in trying out different virtual reality and augmented reality experiences. Things that I've never expressed to anyone or even that I could express, like how I fly in a dream that I've seen mimicked in virtual reality experiences. So I know that someone else shared that vision or thought of how you could propel a human up into flight.

I would love to think that, as spacial computing becomes more ubiquitous and we can actually share in these connected spaces, our very personal things that we think are unique to us that we would then find new subcultures. I suppose the internet did that to a certain degree. You know, that whole like boom of nostalgia in like '96 to like 2004. It was like, 'You like the smurfs? I like the smurfs. Oh my God. We were both kids in the 80s, crazy.' But I like the idea that there's a whole host of unconscious things that are just swimming underneath there. Things that we consider very private that we can then connect over because you see it in someone's house and you're like, Holy shit. You dreamed of a rainbow prism going through your living room too? I have one too.

Yeah, I suppose it opens up a whole new realm of possibility in terms of how we share our innermost selves with each other. Right? I invite you into my domestic space and sure I have a rainbow prism, but maybe I also have some 15 foot ficus that's got purple leaves and that's somehow expresses some aspect of my personality that I can now share with you in an unspoken way.

That's right.

That you're able to see.

Actually, so this is an interesting part question for you. We talk about theming the world and this is pretty ... Like, oh look, everyone's a storm trooper, I'm in Star Wars land, or everyone's got a wizard hat on, I'm in Harry Potter world. This sounds like fun, right? This sounds like fun. [crosstalk 00:14:04] I guess it depends who you are. For like five minutes at least. You can do that with anyone. It's fun to think that you could kind of create this other version of the world that reflects your tastes.

There's that part; that sounds great. Then I was talking or heard about someone who made a comment about how they would like to erase everyone's piercing and tattoos because they find them obnoxious. As someone who has piercings and tattoos naturally, I was not so much offended, but just like, am I okay with that? They're pretty permanent or at least very painful to get out. So the idea that someone would opt to not see things that I had expressly chosen to permanently put put on my body was just ... It's not possible now, so it's just sort of an interesting thought exercise.

That of course then begs the question, well, what's the difference between someone just pretend covering up my tattoo and me replacing everyone's clothing choices with Harry Potter robes? Where does that individual expression get to go?

Oh boy. I mean, that's a slippery slope, right?

Right?

Because someone could want to remove your tattoos. Somebody could want to make everybody in the world Caucasian. It can go to a dark place really quick. Like all technology. All technology, there's always a sinister possibility.

That means that every major platform has to have these modding laws in place that have to be implemented on, again, not a per platform or a per device basis, but on a per individual basis, which is crazy.

If I could create my own world, I think transportation would be greatly, greatly improved because the way that we get around is so slow and it's so hard to visit different places that I think that that would be the first thing that I would make easier for people. There would either be teleportation or everybody can fly, which I think would be a lot more fun.

Well, if I could create my own world, I would probably make this big forest full of monsters, maybe.

Nice vacations spot, like a beach and sunshine and the ocean and stuff like that. Escape and join a world where there's a nice sunny beach and warm and actually feel like that with heat lamps or something like that so your body feels like actually there more. That would be probably my world.

I think VR, like many technologies, plays into our desire for transcendence. Some people might define that as a spiritual thing, and I have no doubt in my mind that there are going to be all kinds of wacky and amazing spiritual VR cults in the next 20 years. But for some people, that's more of a dream state, physiological, psychedelic thing. Really anything that allows us to transcend the banalities of living in a meat body in a slowly deteriorating world. I think this is all going to be extremely seductive to people and that's why VR, or rather mixed reality, is going to be a major force in our futures. I mean if there's a choice between a tedious reality and one in which there are no boundaries but the limitations of your imagination, what are most people going to choose?

Back to our conversation with Timoni West of Unity Labs.

What impact do you think that mixed reality will have on sort of fashion and design or forms of self expression that are embodied, like tattoos and piercings as well?

I would love to think that in the future, we're all going to dress up like giant columns of queen like skulls and all this really, we could just go nuts. You can make such cool stuff. We could look so fucking cool. But the fact is, we could today and the reality is we don't. Part of it is consumers. It's expensive or you don't want to have to think about it or you're not inclined to think about it or you don't care about it. So, you don't feel like what you look like on the outside reflects your internal state.

But part of it is very, very tribal, right? If you're the one who's sticking out of a group of people who are all in the same industry, same age, same social class, you, you have to go out of your way to do that. I suppose if we break down the other barriers and it's cheap to be fabulous ... Which it will be; it will be much cheaper in mixed reality. Then maybe we'll start to see people expressing themselves in more interesting ways. So maybe it'll all kind of come to a head over time.

Like the Internet has allowed people to really embrace these subcultures that we didn't have a name for. I think part of that is just people being able to more easily find their people, or being more easily able to say, like on TV, look, here is a subculture that identify with. So maybe we'll just see an explosion of that.

But I guess it opens another question about you know how computing and mixed reality technologies will affect art or other forms of creative expression. When there was like a distinction between material creative expression and virtual creative expression.

I think we're at a really interesting inflection point because we have all these like two dimensional digital worlds that we watch in movies. But then how are we going to top that or what are we going to do next when that digital art is next to or aligned with or equivalent to all of this physical art?

It seems to me like they would have to be a huge move away from the figurative as being a marker of quality, I suppose. I mean, people get burned out on watching 3D generated water effects. Because a real waterfall looks like a waterfall and at a certain point you kind of hit this point where it's not important. It's not impressive anymore.

It's 50 feet. It's 100 food. It's 100 stories.

Right. But something that is explicitly looks nothing like anything on this earth or doesn't look like a waterfall, that's for sure. Maybe that's the direction. Maybe the more immersive and realistic our technologies are, the less we have to be beholden to making things look real and feel real, whatever that means.

Yeah. That's interesting. So moving away from the physical and then art really becomes about an expression of what's in your head without any material limitations whatsoever.

Yes.

Okay. That's interesting.

Yeah. I mean, there's going to have to be an entirely new lexicon of creative expression that develops in these spaces that we can't even begin to think about what's going to be perceived as valuable or perceived as labor in mixed reality space. That's something that we have to leave it open to new generations of artists to consider and perhaps to make sure that we leave the technologies open enough that artists can adapt and explore freely and sort of bend and break the technologies to their will.

Today, digital art is not valued for much. There are people who will spend 300 hours making a piece of art that's used in a game and seen for two seconds, or a movie. They can't sell that. They don't sell it. The assets just live there and they're owned by Disney or Blizzard or whoever it is that owns them. So I guess there's an opportunity here. When digital objects feel like real objects, when they have that level of permanence, for them to take on more value basically.

That's interesting because yes, but then also didn't people spend lots of money in Second Life buying stuff and clothes? And don't people spend money buying clothes for their avatars in games and Fortnite and all that? There is a level of which people will spend money or will value something if it has that permanence or if it's part of a world that they engage with regularly.

It's part of the identity.

Yeah. It's part of their identity.

Right. That's it. The people who spend a bazillion dollars in Second Life are heavily invested. Like, Second Life is their second life. It's their first life. [crosstalk 00:21:50] Yeah.

And Fortnite, Will Mason had a great post on Facebook about how Fortnite is effectively one of now the largest fashion companies in the world. It is not a game company or it's not making money as a game company. It's making money as a fashion company.

So there's something that happens when people immerse themselves in a technology. It doesn't even have to be an immersive technology. A game is extensively a 2D thing when you're interacting with it, but we still project ourselves into it in this way that almost transcends the technology. It's emotional. It's probably like psychosomatic, spiritual for some people.

It's a world.

It's a world.

It's a world that you go to. And I think humans are very good at just feeling like we're in that world.

So in virtual reality, the weird part about having a digital body right now is it's not fully tracked in any case, so you just look down and you'll have hands. If the IK is all fucked up, they're at weird angles. It's not even your hands, it's just controllers that are tracked and we're trying to extrapolate out. We can get up to about the elbow correctly before it starts going haywire.

Oh really?

Yeah, that's about it. So even if you were to wear a digital skin, if you suit yourself, it tends to kind of just be uncomfortable at today's level. There's no augmented reality technology that will allow you to clothe yourself today, but I can't imagine that once that body tracking technology is better and has a higher refresh rate, that looking down at yourself and seeing things would be a lot more fulfilling than it probably ever would be in virtual reality. Which is an interesting thing, because in virtual reality, you're not really seeing yourself and it's very hard to trick your brain. So you'd have to have some way of getting all of you into virtual reality and then re-skinning from there, and computers aren't that fast yet.

Yet.

Yeah.

That's always been the thing that's my number one alienation about VR, is just the sense that I go from being a fully flesh and blood human being to this sort dot. A dot floating in approximately the same height as my head around in the world.

It's funny. It's one of those things where in the early days of the internet, it seemed like that this is the way it is now. We're all going to be seven foot tall kangaroos and toasters.

Totally.

When I was young and on the early internet, I could never find that thing that was the thing that I felt was a pure reflection of me. It's very difficult to find one thing that allows you to express who you are, but perhaps in a mixed reality environment we can be much more fluid about how we present ourselves. We can change from day to day.

It's also really easy to be ... I'm thinking two faced because it's kind of a literal version. I don't mean that in the negative sense, but just I could literally have a version of myself that I showed to you and a version of myself that I showed to you. If all of my tattoos were augmented reality tattoos, when I'm around my CEO, I could just get rid of them. Right? He would never need to know. That's so interesting. It's a little scary.

It's scary because you may not get to ... I don't know. It just makes intimacy a different proposition. A more complex proposition. I'm sure there'll be all kinds of novel forms of new expression. It does make me believe it will be more difficult to get to know people truly at their essence. But all of this presupposes that we're all on the same level playing ground with these technologies. That there's mass adoption. We all have the headset and it's all the same headset and the UI is compatible with everybody else's experience. Which is not likely.

Well that's, again, where we need to have that individual aside from all of these different platforms. If we're doing it right. If we really want this to go to the next level.

New technologies offer people the possibility to start again. I think that's what's inherently tempting about them. When the internet came along, there was a great deal of enthusiasm about the possibility of remaking society by connecting to each other as individuals in this new uncharted space. Such are the utopian aspirations that come along with all new things. People think that they can do it all over again and do it better this time. Creating a system that's more equitable or more efficient or more beautiful. But once adoption hits a certain point, the new world just becomes part of the old world, and that's when we lose interest in technologies and start to feel alienated by them. Because there are rarely technological solutions to human problems. You still have to do the work. And sure, it's a lot more exciting to think that we can build something new than it is to fix structural inequality within our own society. Which is why I think people are so fascinated by things like blockchain and virtual reality, because they offer the possibility of a new world again. But I think like all technologies, regardless of their affordances, once we're all in, it's going to be the same.

I guess we can talk a little bit about potential dangers and darkness. What do you perceive as being sort of the potential dark side of augmented mixed reality?

My biggest fear tends not so much to be the dark side as it is just the banal side, where we just keep doing the same stupid stuff we're doing and it's all platform based and no one can really do anything they actually want to do with the computer unless they've trained up themselves.

But on a very dark side, people could feel empowered to bring the worst things that they want to have happen to life. We touched on this a little bit. Whitewashing or whatever race washing everyone around you. There could be just crude and violent and terrible things in the sky everywhere you look. There are ways to, if hackers chose to go this direction, where you could really just make everything hell on earth. It wouldn't be legal. It would have to be hacked. But if people can control what you see in your reality, they could want to fuck with you.

What about sort of the hazards of kind of fracturing reality, I guess? If mixed reality is so engrossing and so potentially beautiful and exciting, et cetera-

We hate the real world. We don't want to be there anymore.

Does it make us hate the real world? Yeah.

One thing I'm very excited about is public art in virtual reality and augmented reality and spatial computing. I recently saw a bridge, I think, in Vietnam that is two giant hands like holding hands or holding out hands like this. It's just this giant sculpture of a bridge. That's going to be so fucking cool when we can have these like towering works of digital art everywhere.

But there's so many weird physical limitations to virtual reality today. In your head you're like, 'Oh yeah, I'm going to spend all my time jacked into the matrix.' Maybe if it really was the matrix that would make sense. But the reality is today, when I put on a VR headset, it has a very specific smell. It smells like rubber and plastic-

And sweat.

And foam, and sweat. That's right. I can't really interact with a lot of things I want to. Many experiences are underfunded and we're rushed to deadline and aren't as interesting or satisfying as perhaps even the maker wanted them to be. So it's going to take a lot to create a world that would be genuinely satisfying to the point that people would want to spend all their time in a plastic scuba mask instead. It could happen, but we're not there yet. In the future, I suppose.

There is also, I guess on the flip side of the darkness, this possibility to use mixed reality to help us understand the world that we live in.

Absolutely.

That's often the thing that I wish I had when I think about mixed reality technologies. It's not so much like having like a little virtual windup duck running around on my desk, but the ability to immediately identify any tree, for example.

Absolutely, yes.

I have to hope that, yeah, these new forms of reality, augmented and mixed, et cetera will allow us new vectors by which to connect with one another. New strange intimacies, and not alienate us into being sniffling babies alone, watching a video until we pass out.

Here's something interesting about spatial computing then. Its strength is kind of the opposite of the strength of everything else digital that came before, which is that something that as far away is now here, or something that it does not exist and is now in front of you.

Whereas I'm only going to see your outfit if you're standing in front of me, if it's tied to your avatar. Maybe I'll see a video or maybe we could do like a 3D model of you or something like that. There might be a way to remote in or have a video call or something. But it's much more tied to people and locations and spaces in a way that is literally the opposite for Twitter. Twitter, doesn't matter where that person is. I will see their tweet when I go to twitter.com. But I will not see some amazing location based art you made unless I go there myself or I send it to me and I see a small version of it or a differently localized version of it, in which case it might be beside the point.

Yeah. I mean, perhaps ironically, this technology that's meant to kind of change the nature of reality and suck us out of the real world and into this new fantastic world is actually going to bring us closer to the physical world because we will have to actually go places to see things and to see each other.

This is You, and this is me, Claire Evans. Thank you to our guests Timoni West and Alvaro Morales for joining us. We'll be back next time to talk about digital assistants, the weird little robots we are letting into our lives. Don't forget to subscribe, share, and leave us a review wherever you listen to podcasts.