Reality Lab Lectures: Jessica Brillhart - Radical Experimentation: (full talk title in description)
Hi. Everyone so, welcome. To the. Lecture. Excited. Here. Today. The. Company in the anthem collection and lots of exciting. Tools. And experiences. Before. That, we. At Google and, in everything, created one of the first time, or. Maybe one. Of the first day the art films ever. I'm. Gonna probably jump where a camera team in. Google the jump camera for the ones who cannot see it's. A. We. Are cameras, out there so. I, guess. They started, their career to just told me that when she started her film career by. Making popcorn in. The. We. Were supposed to keep that between us. That. Wasn't supposed to Marty the interim, yeah and I'm, really good at it very, good. Which. Is at least of my, top. Innovators. Under. The, age 35, because of the popcorn thing yeah. Yeah thank you for thank. You thank, you for that's. A that's a heck of an intro now I'm just hungry for popcorn and, we're, just over at the lounge what do you call it was it's called the, Faculty. Club and they have popper in there that you should not eat, and. I, started going off on a tangent about how I was, the best popcorn maker and anyway. They. Do need me they. Needed me oh gosh. Well. It's really great being here thank you for having, me Steve and I actually worked at Google for, quite some time I actually, got my start in the immersive space here in Seattle, because. I was actually reached out to by a group of engineers led by Steve to. Check. Out a rig that they had been building and. They weren't sure what to do with it and I actually wasn't, sure what to do with it either but, so. Seattle's, been quite quite the home where lots of lots has happened here and so, it's really great to be back I. Am. NOT an engineer by trade although I almost became one I come. From the traditional filmmaking world. I went to NYU, as. A. Filmmaker, study film and, then I ran across the street to, the Courant Institute of mathematical, sciences because I couldn't stay away from nerd. Dumb at the time you, know ITP wasn't a big thing yet so like it was really difficult to to meld both sides. Of that coin. So. I would go there and I think it was when I started learning Perl that I was like oh hell nah and then I left. And then I I started thinking about what, I ended, up actually doing was thinking more about the way that computers and technology affect, society. And how the internet was actually affecting. Culture and I was more interested in kind, of behavioral, and user behavior and how that was shifting, and. I ended up going to, work for Apple which was more tech and then I started working in a small production, company which was editing, and then I worked at Google which, was somehow in the middle of those two things as. Their I guess their first filmmaker that worked there cuz and they didn't know what that meant and I didn't know what that meant but we tried and. So worked on a bunch of marketing. And advertising campaigns, for a lot of the new stuff they were working on including glass which was fun because we're all wearing it now. Yeah. So but. Also thinking about the future and how like augmented, having, a contextual, layer augments, the world is actually quite important. And, thinking, about how that would actually relate to the way that we use technology now. And. Then. When VR happened, and when jump happened, I was tasked as a creative, that, was interested in tech and interested in the way users evolved, with tech to, think about what we, could do with it it's the question of why which. We don't ask that question, enough, I don't think right we kind of just make stuff and we say yeah it's cool and in our silos we think it's cool cuz we like what we, make and we like seeing our friends make good stuff but, outside in that big old world there, are a lot of people who don't understand what this stuff is they don't know why it matters and I think it's up to us whether. Or not you're creative, or more tech oriented to test. That to actually think, deeply. About. How this stuff affects culture, how this stuff can be adopted on a mass scale because. If you can't answer that then. There's no why, there's. No reason to be doing this it's fun you could do it but, don't expect it to ever work, beyond. The scope of a university, or a. Tech company to. Me Frank. So. My. Abstract, title is always just it's kind of like a wordy thing but the the point of it really is how. I really. Do believe that creative, iteration, within, the development of new technology, is not only important. But it's crucial if you don't have that you don't know the why you don't answer the why ever you just make something that's cool to you but you don't understand, how. A user may actually use it why it might be important to a user in, her everyday life, so, it's important, to know that so you can actually think about its eventual adoption, into, the world in. A natural way not, a way that's forced.
So. I'm gonna go through three different technologies, that I helped developed in this way and. Bring out through iterative creative, process the, first is jump which is a 360, degree, stereoscopic. Ecosystem, for live-action, VR, content that was created. And developed, at Google, the second is Bo's AR it's. An audio-only augmented. Reality technology. Developed, by Bo's which. Allows users to experience spatial, and immersive audio and the, third would be something called inception. Which is a convolutional. Network. This, is a little bit more on the fringe level. As. You, how many of you know deep dream. So. Deep dream came from from that, inception. Was, basically what they tried to do was the enhanced thing you know like in all the TV shows we're like enhance and suddenly the little tiny blurry thing becomes a very clear crisp, killer, you're like oh that guy. So. Google is like we can do that so they tried it and then one of the lead engineers on it actually reversed it I think for kicks I don't really know and as he reversed it the system began to actually dream on top of the image so, what it did was out a pixel scale actually start to imagine what that pixel might represent what. You end up getting are very interesting, images, which, represent, absolutely nothing except for maybe halves of dogs and steampunk aesthetics. And bananas, and so, on we'll, see some of that shortly. Warning. You ahead of time so. But. A lot of it had to do with how. They actually brought deep dream into the world and how I was able to sort of work with them to figure out you know what are different things we could put together to, help bring deep dream into a. Different, medium and is, that interesting or not. So. I want to also look at how. These. Three impact. Assessment. Of that of the tech and resulting output like how do these how, do these technologies, actually affect particular, things the, first is you know how do how does this technology affect current pipelines, the second is an assessment of the potential cultural, impact these technologies, might have and the third is assessing the potential ways of garnering interest, at scale, not. Just for us to geek out about but. How this stuff actually relates, to the people who exist out there in the world, and. You. Know why should the public embrace this emergent technology, with any amount of enthusiasm and. Can we see and predict the potential embrace that could occur. So. Starting with Google jump. You. Know being, kind, of I guess the cinematic, eye for, the company for five years I've seen a lot and I've worked a lot of different engineers and a lot of tech some. Were, more well baked than others. And as I was doing this you know one of the more unfortunate parts, of the way things were running at the time was that the creatives worked over here and the engineers worked over here and that's just how it worked and yet a marketing manager, producer in the middle they sort of decided what got what and who talked to whom and that was it and one, day I was sitting at my desk I actually member was finishing there was a project about Google. Was getting renovated and I was working on a film about that and. As I was sitting there I got an email from, someone. From the from the cardboard team which. I had heard about sort. Of in. Circles. And. This. Person asked me you know we're. Developing, a stereoscopic. VR, rig you know having thought about creating immersive content and I. I, read, this email and I thought it was so weird because engineers. And don't reach out to creatives. Like that's not how it works so for a second the system kind of broke me down and I was like I can't answer this I'm really in trouble the, answer they actually sent this to me by mistake, this person's crazy so, I just like ignored it and then two days later I get another email saying hey look if you don't want to do it that's fine but at least let us know because we'd really like to work with you oh, okay. Sure, and I finally answered and, just blindly, left like, I don't, even remember what I told my manager I was like I get there's some engineers, in Seattle, that need to see me and need. To go see them he's like I don't know what this is and like me neither so I go. And. Just been kind of winged it and. You. Know I visited. Visited, Mountain View first to see some demos and then. I visited, here. And visited Seattle in the offices, and I, was met with with. This. And. Half of me was like oh my gosh what have I done and, the other half of me was like that's pretty and, I. Actually. Think I wrote that five at one point I had to label them cuz it was really, tough, to know um, so. I looked at the engineers and I said well what do you want me to do with this and they said well we were hoping you would tell us. So. It's. Thus began a very beautiful wonderful.
Friendship. This. Was you know they showed me a few things is actually I think the there was a flowering, tree outside the University of Washington, that they showed me but this was the stitch that did it this, is a group of the group of engineers that, were, working on the camera and I've. Been trying to film engineers, my whole career. Practically. And you, know you there, can, be nervous and they can be like they're kind of like have to read the things that you told me to say and but, this is the first time where I actually was in the midst of these of these. Great folks, and I totally, got it I got why what, they were doing was important, to them they're just hanging out having a good time throwing, things around and I, knew immediately when I saw this that that had help that, was the selling point it was like that's it whatever, this is I, don't know what it is but whatever this is I'm, here. For you let's, let's figure this out. So. This is you know as you. Begin these things you. Know there should be like someday there'll be a hash tag called like my first stitch or something but. This. Was the first, couple. Of stitches I actually. Recorded. Here in Seattle, and. We worked side by side both, the engineering team and myself work, together yes that guy indeed has nunchucks, and he is using them, and. We, practically, I mean we just put it through its paces and, you know there was artifacts, and so on at the time but and. I think Rob got really upset because I almost the ball must hit the camera so he was kind of freaking out and took, it took our toy away but um. But. We learned a lot like we made all the mistakes like we just worked together we had no idea what we were doing but we just decided together we would walk around and just try to figure out what to do with it. This led to a, very long I, wouldn't say a long trip it was actually shorter than it should have been but my. Producer, Nick and I basically, went, to Japan. Puerto. Rico. California. In case the camera broke and it did, and. Iceland. Just. To throw, it at, different, things to see what would happen again we didn't really know you. Know, the camera. Liked. Certain things more than or the rig like certain things more than others. We. Filled at Arecibo which. Was really great so under the massive, dish. At. Arecibo and and on top of it too I. Broke. One of the rigs there. Because. It's curved and then when you move the when you move the actual platform which we got to it kind of toppled, the thing over it's. The thing, but. It loved Iceland it's, a very emo rig it loved Iceland. Because. It was cold and because there's no other people around, disrupt the signal that was connecting all the cameras together these are unsynched, GoPros so. And. So if you press a button, you. Don't really know if it's gonna work so it was kind, of like that now, even. Given that you might be thinking we got to go around the world you got to film all sorts of stuff that was such a romantic wonderful. Trip I get that a lot oh hell, nah. This. Is basically, all the time like. I mean this is SD, cards for each of the cameras, we have backups, for each of these we had hard drives for each of that backups, of those hard drives each, of those individual, GoPros had to actually be charged, and. We completely, forgot the fact that you know different countries have different power. Connectors. So then we had to like think I think at one point Nick, was like sawing one, of the cable, it was bad so, we. It. Wasn't, romantic it was messy and. There. Was always a chance for error and somehow, miraculously, it worked I mean, that's the thing at the end of this.
It's. Great engineering, I know who. Who let those engineers, I don't even know who it was so. For. A creator it was it was I mean it was like anxiety city because you know we're trying to hit this Google i/o deadline, I. Could, not see a damn. Thing I couldn't see any footage, I mean I had no way no way of knowing we had to develop like a I mean, I I worked with Rob where the engineers to develop something that would actually quickly stitch some of the camera feeds together even though they weren't in sync so I could at least know if the cameras were working because we don't know, and. It so it was like shipping, off drives being like alright we. Only have one shot at this goodbye. And. It, was hard and it was exhausting, and it was so frustrating, but, it was the, most exhilarating experience. Of my career, of my life I think it's the same for the engineers that worked on this too because we, didn't know and then to have something that we didn't know turn out to be something is kind, of remarkable. So. You think that that would be it but you. Know it the hard part was. Just beginning. Because. I'm a filmmaker, I was a filmmaker I don't know what I am now but I was a filmmaker and so a lot of the ways that I was doing things always I was thinking about content creation generally. Was. Going to actually prevent me from doing the right thing because. It's a completely, different medium vr is just it's, not film and. You. Know a lot of the things that I was facing weren't things that were being discussed yet and so. When I tried to go ahead and Google it there was no result and if there was a result was usually by someone who never made a thing in their lives and they, just had opinions which. Still exists so. Just. To go, through the. Basics, of filmmaking, you know as, you know the, frame is everything, to, a filmmaker that's, what we make that's what we craft. This. Is how you tell your story, but. Your frame doesn't, mean a whole lot when. You can do when. You're audience member can do something like that right it's. A who. Cares what your opinion is I decided to look there so now what. So. The, way stitches, are being delivered, at. The time and still are honestly are. Like this which makes it seem like VR is a frame, but bigger. And. I. Realize pretty quickly that. You, know my, process, for, editing. In a, visual medium in this kind of medium. Had. To change dramatically. Because this, wasn't, working I was editing, in this, like kind of frame but bigger mode, and I would look at it I would watch it back and I realized that it was just bad and, everyone. Knew it was bad and it wasn't working at all. So, my engineers when, I was like really freaking out which was pretty often and I. They. Were like we're gonna eat lunch like. Okay. Wait. Lunch, so. We go out to, this Thai restaurant, I'm sitting there and I'm. Not eating cuz I'm stressed and I look up and actually, if you have him too it's like those speakers up there so. I look up at the speaker and suddenly, it all made sense. Which. Was the way I was seeing this. Was. Not the. Right way of seeing it but I should have been seeing was something that was more like this. Which. Is more to say it's a world of potential frames that's what every, thing. That's. What every stitch really represents, it's an experiential. World where. They're all there. All sorts of infinite possible frames that could exist. And, it's. A world that doesn't necessarily stay, static either it ebbs and flows it changes over time. And, add to that you, know an immersive, world begs for, an audience to be part of that world and there, was agency, there even. In something as simple as turning one's head that's. A decision that's a choice, and. So. That was, a behavior, that I ultimately didn't, really have control over I could predict it but I couldn't completely, control it myself. So. This in turn made traditional editing wrong a wrong, approach to editing in VR, because. When we think about editing in VR we think about it as frame to frame. But. Since we're dealing with worlds now the. Process, ends up being something that's more like world to world and, these. Are worlds that extend from one another and my job as a creator is to essentially pull these.
Users These audiences these, visitors through. The universe I created. So. One way you can think about this actually is. You. Know thinking about points, of attention, and placing, bets in particular areas where someone might be engaged. This, is something that I've been calling matching on attention which is similar to matching. On action, in editing. So. By, understanding where, attention is placed we can essentially and more comfortably, take a visitor for something she's paying attention, to in one, world and then match that to, something else that she be paying attention to in the next so we can actually get. A get a better and, pretty gonna. Get. Can't, even talk we can get to a place where we can actually be a little bit more sure that the person is going to be seeing what we would hope she would see or experiencing, what she hopes what. We hope she would be experiencing, an. Example, of this is, something I did with the Montreal Canadian, ice hockey team. Because. You, like you do and this, is a scene from a. VR, experience like I did called go Hobbs go and. This. Is an example of what, like, I'm matching on attention. Technique. Might look like me. No. Sir Malvo leukorrhea make, potato swear. De. Souza. The papyrus, there were. Contacted. Reflect. On the amount, of Edessa gotta keep on, songs, recently. I wanted, to separate medieval, music. So. I, could. Have just got of the rink right okay I know you're looking at the thing that's on fire. It's a thing that's being passed around and so I know you're gonna probably be engaged in that area so I'll rotate the second world so that the rink shows up next. But. Then you would never see that guy and he's. The best part of the whole thing. Right. And and, it's basic, it's, kind of basic, I call it like the creepy baby market research test which is like if you have a baby and text. People, are gonna generally look at the baby not the text but, if the baby is looking at the text weight the what the person will happily do or a user happily do is look at the baby and then look at what the baby's looking at and then eventually get to the text so what you end up getting is the dude looking.
At You then the dude turns his attention towards the rink and you get the rink anyway. Just, to be sure that I wasn't completely like. Being. All, weird in my own head I actually, worked with a team called Retton ad in Montreal who did a heatmap test just to see if this would happen and turns, out it. Did I. Don't. Know what that person was doing totally. Like enamoured. By something over there um I. Mean. This is actually brought a lot of questions up around like is it interesting to do viewpoint aware editing like is it is it interesting to have like general, areas of interest beyond be, triggers, for continuing. And edit in a particular, way and the answer is maybe I think, what's interesting is, it's, fun to actually predict it without those capabilities, just to know that those capabilities could, and perhaps should exist, but. It's a fun idea, thinking, about how you can build a kind. Of responsive, edit to a VR piece based on attention. But. I also started thinking about rebellion, that. Was actually inspired by the, two guys that created mist I, went to talk to them in Spokane, when, one time a couple times and they. They. Were telling me Robin. Was telling me that you know he'd go into a VR experience, and. Look kind, of a pay attention to the direction that he felt people wanted him to look in and then. He. Would do the complete opposite and. He, was like you'd be shocked there's like nothing there most. Times and actually happened me too, I would go in and be like okay I'm gonna try that and you just turn nothing, which seems like such a waste. Cuz it's you know 360 content, and all. So. I thought well okay well what what can I do as a creator to sort of explore what could be possible in that. This. Is a scene from a VR experience I did called residence, Kennedy. Young, girl here is playing violin. And. If that's what you pay attention to in that scene great, I mean that's fine. That's she's, there and that's sort of the point, but, what happens if you turn around. Now. You, see her parents from the doorway those are actual parents but um, I'd, argue, if. You want to talk about film this is far more cinematic, than, the other one cuz you still hear Kennedy she's, still playing or violin but, then you can see the parents kind of slightly grimace and then like you know yeah yeah.
But. Think about how we can use the space in its entirety. Is. Something we should think about far more often the contextual, areas, of being able to, be. Able to take a point, in a scene and be able to rear up resent it in many different ways and, in various permeations. So. By leveraging where what and how someone might be engaging with these worlds, you can help comfortably, navigate her through the experience, you've created and the, universe suddenly, has cohesion. Even. If they're disparate things. Um. And in, a way it's not too far removed from what we've been thinking about in film it's just not called the same thing alright it's, not the same thing technically. So something, like you, know you're in a world where, there's no real point, of interest or place to engage where. Everything, is kind of happening all around you and there's nothing to particularly pay attention to I mean, that's like a wide shot it's like establishing like. Establishing, shot where it's really just about setting the scene for something or just getting a sense of space. Something. Like I seen on a tram in Japan when. There may be two. To, four possible. Points of interest it could be seen as a medium shot just. A way of engaging with you. Know couple of, characters. Or a couple of elements that could be in dialogue with one another it's, the same kind of idea. And. It seemed like a glacier climber climbing, a glacier. Who's the only person making a sound who's the only human in the space can, be seen as a close-up we pay attention to that person probably the most for, the longest so. Really this is about you know how long we engage and the kind of attention that we give to that world, and. It's important to note that not all worlds, are the same you, know different worlds come with different energies as we've seen it's not all one note and if you know there's some VR experiences, or Eric's penny immersive experiences, where this may, be what, happens just, kind of a OneNote thing. And. That's okay but it's kind of boring and who wants that. But. We also do when our visitors feeling uncomfortable, either so. You know thinking about physical, movement thinking about fatigue is something that we sometimes forget to, consider. So. We also don't want someone swinging, their heads around pointing, things that stuff expecting, things to happen unless. That's the point like anxiety the VR experience, maybe that one but not not. Not this one so. What we really want is good experience. Will flow so it's like a dance it's music and what. That kind of dance is and what that type of music is is really what's up to you that's that that's part of the creative process right, so. This. All kind of came out of me through, this idea, of a speaker in the ceiling and Thai food I guess that I didn't eat but. The it's. Interesting to note that like a lot of this went to we had really great conversations, with Adobe and they actually brought in a lot of this thought into their, line of to, premiere actually, so when they our first, view our tool kit came through a lot of it came through conversations, that we had with them about you know things that as a creator I actually cared, about when it came to creating. Work. And I think Adobe Research is working on a anneli. For. VR but maybe not right not right now maybe not but. It's it's it's fun to see how the stuff can permeate I. Made, a couple of, three. Kind, of major, VR. Experiences, out of these these thought processes, world tour was the first job experience. That we, made. Residents. Being the second and the third being go heads go. But. Then I started really thinking about cultural impact so we talked about pipeline changes and how like the ways in which we create within with these new emerging tech how that can kind of shift the ways that current. Pipelines, work and function but, we also think about how this actually affects culture, and people. So. I thought a lot about cultural representation, after. This as well and just for a bit of context this is the golden record it. Was attached to how, many of you know about the golden record a lot. Of you so, for those who don't this. Is a record. That was developed by a small team led by Carl Sagan and. Andrew, Ian also and.
It Was a it basically. Contains. Human. Experiences. Photos, audio, sound effects stuff of that sort this, record. Was attached to voyagers, 1, & 2 which. Was with, which, had a mission to head towards the outer planets and into interstellar, space sort, of a message in a bottle to, the aliens. So, during the 40th anniversary of the golden record which was a few years ago I thought how cool would it be to reimagine the contents, in, VR for, people if this is meant to tell aliens what it's like to be on earth you, know we're aliens on our own planet maybe we need that too you, know can, we take this content and actually reimagine, it in a way that allows for other people to connect with each other. So. One of the pieces I made was something. Called, Navajo Nation. Which. Was inspired by a song in the record called the Yabba jay or the, Navajo night chant as it's, called. On the record and this, image of Monument Valley now, I'm very, quickly realized that there are a few problems here the, first is at the Navajo community I was talking to was generally, unaware that the sacred song was, on the record they, had no idea, the. A bitch a has only performed, at certain times of the year and in a certain way so having it floating around in space for all time for the aliens to discover seemed. A little weird to, them, and, the second is that this image is backwards. You. Know and like the response was oh well the aliens, won't care. I'm. Like I'm pretty sure the aliens we care a little bit you know they'd start to make assumptions, here and the Navajo really care about directions. Cardinal directions are massive, for them as part of their as, part of their spirituality, so. They care at the very least and we should at least care because they care so, even, with the best intentions we can royally, mess things up. So, I asked myself okay, now that I'm here in Monument Valley having, discovered that these things could be actually, really bad I I need to figure out how I could potentially help. Can, I help and. So, the more that I explored Bluff Utah which is where I went to do this this this piece I. Observed. How much access, the. Community gave to. Aliens, you. Know if it, can't. Come down to like how well do you know how long did they know me and how what kind of access did they give me as a result of that and this. Kind of was something I noticed throughout the, time I was with them so I thought that's actually kind of interesting cuz.
Usually When we make these experience, we're like go get it all of it free go and we don't really think about access, being, kind of a tiered, thing. So. What if that was the experience like what if you could and, could not do an integral part of a narrative like what if what if culture could be encoded in the, agency, that, the audience had in the experience, what if that was represented, like. Representative. Of. Who. They were and, what they who they were in what their culture represented um so, when we think about interactivity. Most. Of us think like this so two simple. Options right on or off open. That push. This button pre, binary, stuff this. Is Zork. Hand-drawn. By by David leveling I think from. 1976. Um. We're lame comparatively. If we think door 1 into r2 is cool and Zork was doing this in 76, then you, know maybe we can think a little bit better about what we can do here so thinking, along those lines you know interactivity. And the decisions we make are more. Important, than they appear, collectively. Interactivity. Has meaning its own narrative arc and each decision is marked by culture, experience. Upbringing. Religion. Race gender a button, push is never just a button push it isn't I have. Been talking to the unfortunately. Now next Google spotlight stories about, how they had an interactive, tool kit that they use for all of their animated. Pieces, and I, had brought them this and said you know I think that there's a way we can leverage what, what I feel we need do with this and what your you've been working on they said yeah let's try it with live action and see what what happens and so, we. Looked. At what we could do and I sort, of made these four kind of micro, VR. 360. Films, that centered around Grandma, the rodeo land, and powwow, which. Were the four kind of major kind. Of touch points and, then, I thought about how accessible. They were to. Someone who was new to the culture, um, the rodeo for example is a moneymaker for the community so come on in give us your money but. Grandma given, her uniqueness and her spiritual nature her importance, she's much harder to get to and. So. What I did was, I created a main. Kind of hub which was a 360. Shot of Monument Valley with. Each of these vignettes, in each, of the cardinal directions that, corresponded with, the importance, of those vignettes so. The most important one in the North. The. Rodeo which was in the South. Trying. To remember what the other one was powwow in the West and land.
In The in the east and. The. Hogan's and how far away the Hogan's were from that center point really determined, how easy it was for that person to get there and all, of those by. The way vignettes ran, like, I'm like a like four trains on a track so, as soon as you started the experience, the the stuff would go and you could go in and out of those experiences, to a point but they would not stop and wait for you there, was no like okay we'll wait and take it back like nope life moves on this is how it is. So. You. Know I feel like earning. Your experiences, is still something that's kind of interesting right and I think it's actually pretty important, and. It, should be more than, a button push that just happens whenever I think that there's something really interesting about the narrative arc that can exist through interactivity, and how interactive components. Can actually be part. Of the, way that we understand, the experience that we're in. So. How do how, does like the medium of live-action 360, VR be interesting to folks in mass says. Asks, everybody, and. We're more specifically. How can they see that how can someone see themselves in this like how can they feel considered. So. The second piece I was able to make for the Voyager project was a piece. Called Beethoven's. And. Music was a huge part of the golden record right it was it was you know trying to translate. You. Know and other concepts, well that we're trying to be translated for like, you. Know how do we eat how do we drink how do we fall in love what those things represent, how do we how do we show them to to another life. Form. So. For, me the question was you know given, the music. That existed on this record you know what, if we could explore what hearing, meant to all of us like thinking about bringing a perceptual ability, and then putting that against something like music Beethoven, particularly. Since Beethoven, was going deaf as he was writing it. So. Given, Beethoven's creation of music in spite of his deafness I felt it could be important, to unpack, this idea so. That I could build an experience from the ground up with, both the act of hearing, and deafness. In mind. So, on one level of Beethoven's fifth is a great performance OB difference myth by the Philharmonia, orchestra, in London on, another is Oliver, Sacks notion, of movement parsing how, deafness can reshape one's own perceptual, abilities and make something like sign language easier, to crack another.
Layer Looks at how the Voyager probes themselves, were able to hear on their journey, and. Finally, Helen, Keller which is an interesting story because, I was at MIT makes, me sound like I'm so scholarly but I was at MIT at, the time and this guy comes running after me after I do a lecture and said, screams, Helen Keller at, me really loudly, which. Usually you run away from that too you're like okay this. Nice guy. But. Um he was like no no she was she was deaf and blind and, she actually heard Beethoven, for the first she has like a whole letter that she wrote there's. A website called letters of note where this was actually recorded, which she had actually written in two, radio stations saying, that, she had heard the difference between brass and, strings. And, the, timpani, and the in even different vocal, intonation, like she understood, it because she put her hand, to. The radio, and could feel it so, haptics there's. Interesting notion that haptics could actually be helpful, in something like this. So. Here's, a snippet of what. The experience was like. The editing of it was also very fun too because by i again, i beholden, to the 360 vr format i couldn't just like move the rig around right so a lot of it was trying, to figure out a way of. Enticing. A, person, to feel like they should step back so as soon as I said Pekka would go like this we, kind of moved back that way so it felt a bit more like there was some energy or some force being pushed against you so it didn't feel so weird to be moving around the space so explored a lot with that as well, and. So we premiered does it the Sheffield doc fest a. Couple. Years ago and, what. Was amazing was, that. There. Are a lot of people who were deaf that were showing up, and. They, felt very much. This is the first VR experience that they've done. That, was made for them. They, weren't just the afterthought, they were kind of the point like they felt actually considered, which is great. And those. Who could hear just fine actually, came back and, unprovoked. They took off their headphones. To. Experience it again because they thought they missed something they, were like oh my gosh they seem like they held off fine I wanna do this and um. You, know it's interesting to, think about how we can connect people through perceptual. Like, you know someone, who has a perception disability and those who are, you, know healthy, in that regard I guess having. Them actually connect through experiences, like this not, overtly hitting people over head like this is deafness the experience, right like. We're not trying for, that here it's like how can create something that's beautiful on all on all levels and all permeations. And. We owe that to ourselves to. Be better about creating experiences that, feed, more into that into, that vibe. Um. And, also, thinking about people who are gonna be at noisy bars and not have beefy, computers, and they're gonna wave their cellphones around and be like VR cuz they exist and it happens I've been one of those people. So. Um you know and creating VR is extremely frustrating right or any immersive context is frustrating, and it takes a while it takes it, takes a lot of attention and time. So. I gave, myself, a. Task, and I said okay you have a day. You. Have to use stuff that's available online. And you. You can't like don't overthink it just do it, so. I created this piece about. The Weather Channel. And. I'll, just show you what it is. I. Mean. That's basically it. It's. For. Girls. Like that. So. It's, it's it's four, panels, of The Weather Channel taking, at different points during the 90s and for, some reason when I did this recent research learns of this research it, took I should put my glasses my, research it turns out that I.
For. Some reason people. In Omaha Nebraska recorded. The Weather Channel all, the time so. I had like you know 89 90 to 96, 99 you, can see layout of the subtleties, of changes, in the Doppler Doppler radar, and I actually really love I should I, kind, of want to go back for a second huh know whether. You can always turn to it's. Like waiting for me waiting. For me neither I don't why the horses there but whether it's fun now. If you, are you, know if, you look down so I called it conditions at Omaha so if you look down you see a postcard from Omaha, Nebraska. And. If. You look up. You. See Kenny G. Is. He oh my. Gosh she's home oh. Thank. You thank. You mine, was the band live so you want so. The so. It's an animated gif just like from gif you just dragged it on there it's just him playing, forever. And, some people would. Actually engage, with it and and love it like around them and so on it's um but we'll just lay down and look up and watch Kenny G perform, the whole time and that was it and I was fine too. The. Response, I just like released it into the world and the response was not what I was expecting. You. Know I was still Google at the time so Google filmmaker makes VR video mashup of 90s Weather Channel footage, and Kenny G and I can't think of a reason not to watch this. The. Weather Channel - Kenny G is the least necessary of VR experience, and we love it, I. Think. Kenny, G himself, see, all native Kenny G I spy, someone sexy, playing. Smooth jazz in, VR, for. Context. In case you wanted to know Kenny, G likes, making sax, puns which is really just taking sex, and making it sax and then saying, things about himself, and the world so this actually took time this wasn't just like oh yeah VR he wrote that it's, just like, retiring. Not. Really that, I'm. Not saying this had anything to do with me. But. About a week after that couple min I mean like two weeks like in two weeks Aki weather came out with their Weather Channel ish, app. For. Fergie. R and. The. Result was very different. AccuWeather, is VR app is like dystopic, dystopian. Design fiction. Yeah. Which, makes you think that, you. Know we live in and I'm I am of I love high fidelity stuff, I still, make that stuff I think it's great. But. Sometimes you don't need that much to, get people to be interested and what, you're doing and to be interested in the world you don't need a bunch of data thrown at you is actually a lot of responsibility, and less, is more it still, works. You know it's still actually kind of important, um I'm. Not saying you're getting accurate weather from conditions. At Omaha but you could you.
Could You, know. You could. And. You could. Have, a play maybe, all it needs is really you know Kenny G and a you, know 90s shag rug for you to like sit on or something I don't know but it's it's it's really you, know the, idea that something. Like this could, be enjoyed anywhere, and. There's really no harm in that. So. All of that in. The VR world came from this thing and, this. Came from iteration, on an emerging tech a piece of emerging tech and just, throwing yourself at it doing, iteration after iteration seeing. What didn't work what did work and then working off of that and, you get something like conditions, at Omaha you get something like world tour get, something like residents, go. HAP's go all of them very different and. Even something with the Navajo or something in space like. Lots. Of possibility, there and all sorts of use cases that come from that. Moving. On you, with me still that's, a lot. Bose AR so, when. I left Google I started. Thinking about what I wanted to do with myself in my life and I thought about kind. Of how important it was to be reactive, to all possible technologies, that existed not just one and not necessarily from one from one company but being able, to see and sense the entire ecosystem, of all these different technologies that will come into play. So. You know shockingly, found a name that worked it's. Not VR AI it's Bray it's French. Because. Knew French. And, what's, great is it does have all the letters in it but in French it means true, or, real depending. On how you how. You use, it. Believing, of course that technology. Is is a tool, not. The goal, right. And. So. It was kind of a remarkable thing when one, of the first folks to come running at me as soon as I was like how my own company was Bose so, Bose was developing, technology. Audio, driven AR technology. That. Generally, when companies come up to me and say I've done this thing and if they are I'm usually like great and then kind of wait to see what happens and but. They actually had a great, demo. Of it at South by Southwest one, year and it, was like literally just someone, saying. That the restaurant is over here like. Usually your hope you have your phone up and your compass saying the whole time and now, I heard someone say, over. Here and I turned and it was like oh my gosh audio like that really does make a difference huge difference and I've always had spatial audio and all my pieces but this.
Was Something else it was something, more. Profound than that. So. Looking at get at current pipelines I was like okay great you know what I kind of want to create an. Experience that like let's take a multitrack, recording, of an old musician. Or older musician because. They did that you know like what if we took all the different stems all. The different instruments singers. And so on and kind of spatialize, them and let someone actually move through that music move through the audio move through the track but, if I could do that for one thing I could, do that for all things I could do it but podcasts, can do that with field recordings I can do with documentaries, they do it narratives I can do with everything and so there's this interesting thought, of like oh wait so I can't just do a one-off thing because everyone's been doing bespoke, one-off projects, I can, actually create a platform from this this is a platform, so. I was like okay how do we do this what do we do and there, was nothing. Nothing. Out there so. I have you know it's like you come to that point where in all of you know this where you're like okay we really want to make this thing what's, out there and then you're met again with nothing and you realize that you have to make it yourself so. We, ended up creating something called Traverse, and. Traverse is a platform, for spatial, audio experiences. That makes listening, visible. You, can move through the stuff that you hear. It. Was actually built for. The. Home not. For festivals, not for bespoke conferences, it's actually built to be used at home so. From the ground up and, I have it with me so you can demo it if you want I mean it basically just uses an. IPhone, I'll, come back up I promise. Goodbye. It. Basically uses these like you guys some of you have them like. So the AR technology that's built into those really cool frames that you have is actually gonna be built into a headphones and those, is just the first to do it a lot of people gonna do it. So, thinking about like okay finally, we have a kind of immersive tech that doesn't look like it's going to eat my it's, like actually something that I may have to also use for other things like listening to music regularly, or communicating. With my parents occasionally. So. And, then also just using your phone as well so it's like an iPhone. Okay. So people have devices like that that seems simple, enough. So. It. Made sense from from a pipeline stance, that we had to kind of create that pipeline ourselves, it didn't exist it should exist so we had to think about how we would create it but, then again thinking, now is the tech side being like okay what's the cultural impact of this so, it, wasn't enough to just take you know just getting. Some bespoke someone, to do something it was like who do people listen to or no at least so. We ended up working with Elvis Presley it's. People. Apparently. And not, apparently in 1969. It, was the first time Elvis Presley was recorded separately from his band he was on a film, I think, and he so. The recording, is Elvis separately, in all the different stems as guitarists his bassist drummer the backup singers all of them are recorded. Separately, so, this, album was called from Elvis and Memphis so we took suspicious, minds, and also. Power of my love and created these, traverse experiences, from them. So. It actually looks so you know although it's an audio experience. And you're moving through the audio field. The. Visual. On the device on your phone is actually representative of where you are in that space so it's a map basically, so if you think of what an album cover is to music currently.
We. Think that the evolution of that is is a map where, are you in the space each of the colors kind of representing, generally, where where various, instrumentation. And and sound sources might be but. If you hold press, and hold on the screen another, contextual, layer shows up so, it's not what comes up by default it's like the default design right like what's there what's not there how does that come up, if. You stand on Elvis and hit sing along the lyrics pop up no. One press that button because they thought it meant that we could hear them singing and we're, like how would we do that and. You. Know again you can move around to Ed or any of the other session, players and it feels as if you're moving towards her away from Elvis. What's. Great is that also some of the session music, arguably for me better to listen to so I'll actually go up to ed who's the harmonica, player on, power. Of my love and he's just going at it he's just like his, breath is there and it's just like he's just killing, it and you, can just hear Elvis from the vantage point of Ed which, is pretty pretty, awesome but, you could do this with any again if with anything really and the idea was to create it knowing that from a cultural perspective people. Are listening. To music in a particular way audio in a particular, way and making sure that we don't forget that just, because this new tech is there, and available. On. Another side, of that we did something called the arm of insight which. Was taking publicly, available assets. From the Mars insight. Lander so, images. And audio actually, was I think it, was the first instrumentation. On Mars actually, collect. The sounds of Mars or it was able to read the sound s'mores so. We have that in there as well horse, of a slightly different color it sort of hardened harkens to like old games, that I used to play as a kid, so. You know if I could have a text-based thing in the beginning to give you a sense of who you, are which is the arm of insight. Because the insight Lander can't, move so we had to make you the arm of it, so. You move forward actually, to start the experience, when, you do. Sounds. Appear, sound sources appear a surface of Mars appear you can deploy and, activate, various. Instrumentation. On Mars which again is modeled after actually, what is there on what the lander actually, does so it's you know very educational. And all sorts of ways and, then halfway through the experience a dust storm happens, which, is cope look, at there which was conveniently, around. The same time that a opportunity. Died. So. That was a fun conversation really, and then a dust drone happens they were like ah NASA, it was like wait a minute um maybe. Not so. What. We did was we we actually futz with the visual. So you actually do have to rely more on what you're hearing because, one of the things we didn't realize were that we're really bad listeners, we.
Actually Have gotten, gotten, kind of sidetracked by the whole visual stuff and stuff on our phones that, we have to teach people how to, not. Necessarily, we. Don't want to rely on the fact that they will change their behavior but start to teach them how to behave a little bit differently so it's like trying to graduate, get them to a particular, place, I've. Been talking to this guy for a little bit - I get lucky within this field. Because everyone's like I love immersive and they don't know what it is. Yo-yo, ma, wanted. To know what, the stuff was so, I brought everything I could and he this is using a daydream headset and I think he thought it was positional, or just, assumed he could kneel so he did that and I'd tell him he, was doing it wrong which, is kind of scary. So. We worked with him on a Tribeca Film Festival. Traverse, experience, called into the light. Which. Is his performance of cello suite number two and D minor that. We mapped to a building in New York so as you ascend to the roof you actually move through the movements of Bach, and, what we did here was we evaluated, a space basically. Where a Tribeca Film Festival the. Immersive, experiences. Happen we, got a sense of kind of the trajectory, like how people are gonna be moving where. Are they moving to, and from what's, around them. You. Know what are what are some places where we could put certain elements or markers or, yoyo and so on. So. The markers themselves kind of look like this they're, beautiful, they are essentially. QR codes but they look nice and. From each of these markers you. Basically, get access, to each of the movements so this is movement one in the small little booth that they gave us here's movement three that. Would take you up the stairs movement. Four. And. Five, and. Basically. We had to do a bit of a coordinated. Dance. Of sorts because we, had to have a guide. Take. Them through and basically. You. Know guide them basically just guide them through the, experience. Because. If we left people to their own devices they probably would have activated one thing and then not. Known what to do so understanding, as. Part of the user experience of this we had to have someone there be, there for. Support was actually quite important. And. There's still a map you still knew where yoyo was at all times but. We also integrated. Another. Which. Was when the markers were. Activated. The music would happen but then we had a our visuals, around. The person but they could either do or not I mean they could just see where her yo-yo was. Or. They, could walk, through this pretty awesome, piece. Of artwork but. I think actually was built and tilt brush too. And it's spatialized as well so. Can. I hear it. I. Think. I. Think I space telephone so the, the. To, Brad stuff this year he's, the. Movement. Of, instruments, or so, movement is like the section, of the of the suite so, your. Movement doesn't like, doesn't release, any of. So. Those those are just drawings that were it, inspired. They were inspired by the movement, that was crated by another artist oh I see. No. It starts I mean it rides. Along when he plays, but it's not like it's not responsive, to the actual audio, you know I mean give it more time maybe. You know that would be cool it. Ends up on the roof, which. You can see here the. Idea of going into the light meaning you're going through kind of a dark kind of crowded. Space and then into a space, of peace and serenity so the idea of like emotionally, what the exterior, of someone, kind. Of adds to the experience which. Is important for a our experience is I believe. So. Interest, at scale like how this is actually scale well everyone, like smiles when, they listen. To this stuff they get it it's audio they, use headphones and, the phone and they get that stuff to people. Were taking Instagram. Shots. Of their phones which was interesting, it's, just like selfies of the experience, a, lot. Of folks. Really picked up on it too saying you know it feels like this is, something. That we've been waiting for for a bit it feels like maybe this could be the killer app. The. Idea that, you. Know it. Was something that was actually quite powerful as an immersive experience. Even, without like, extensive. Like high quality visual, components, was actually cool people. Saying, some. Of them saying. We. Won we. Won the future. Of experience, at South by which was crazy I'm. Given what we were and. The. Idea that the next frontier for immersive storytelling maybe in your headphones I'm not showing you this to brag by the way I mean seem like it I'm saying this because it's.
A Technology that people weren't paying attention to at all and then. Suddenly everyone was talking about it because they're like right this is actually quite immersive it actually works, for me and it will work for my mom and it would work for my kids and it would work for and it made, sense and actually answered a question, for. Many people so something. Also consider is you know mass adoption for, for emerging tech is hard but, if you think about how people do use tech currently, and then again guide them to behaving, and thinking about the immersive space in a way that could be interesting, are those, other your audiences those are the audiences they're actually gonna buy headsets, it's. Not you know right now those aren't your audiences because they don't even know what what. This stuff is what that means so, giving them a sense of the possibility, a taste, of it I think is very important, if, you want anyone to care. So. Last, but not least inception. And. This is really a story of. Two. Teams, there is a VR team which is computer vision and an, AI team also computer vision and they. VR. Team was. A team I was working with here in Seattle and you, know we were filming, and experimenting. With jump these, are some rocks from. Iceland that are great they're, great rocks, rocks. Are great in VR shocking. So. The. IDI team, who. I knew some of them a few. That were there. They. Were working on, a system again we they had reversed it and they had come up the system had become deep dream and. Some artists were actually using it in pretty interesting, ways Kyle McDonald, Dennis to, video. Content and I. Saw that was pretty astounded. By how good it was, so. These are teams sat, next to each other in group at Google Seattle, literally next to each other right. Next to each other Steve right next to each other. And. One day I was working there and I, get I, get, another email they give me the best emails I got this email that. Just had this in it and and. It was the subject was isn't this cool question, mark. Yeah. You, know great. Easily. As deep dream I made it it's cool and I was like yeah okay so so. I see this, and. So. I'm gonna talk about I want to talk about current pipelines. Yeah. You could, get this from from the system that's that's, only fine. But. If you this was okay, this is the great, email one and then, Mike. Tyco who was working on the AI side was. Like oh okay yeah I see what he's doing let me do this instead, so, you basically change parameters, right like so when you're like okay this is all deep dreama's and then you realize you can actually go from puppy slugs to, beautiful, pagoda, mountains, just by changing some of the parameters I think that's actually kind, of important from, a pipeline perspective, a lot of creators right, now are little worried because they're like well if a machine can do that what's my job it's, like well that's your job but, what, what it actually outputs is your job that's, the artistry that's the point. So. When we saw this we're like okay not as bad kind. Of good and, so what ended up happening for, the first time ever was these two teams working, together to actually bring you can tell us jump because of that but, it's a it's, the they, understood, that there was some kind of neat way of leveraging. The stuff that we were doing to. Help explain what they were doing and so there's really nice friendship, that started to emerge. But. I couldn't just stop there so I kind of kept bothering them about you know we've shot all this really great stuff aren't using jump, all. This fodder that's never gonna see the light of day why, don't we just throw it into the system and see if it works which as a creator asking an engineer to do that I'm like it's easy and the introduced like oh my god you're crazy like, it's not easy like there's all sorts of stuff that I have to consider, but. We went back and forth on it I worked with this guy, Doug. Fritz who also worked in Seattle, first bell and. Again. Seattle thank you so we we. Worked on this thing called deep dream VR experiments, which. Was the first time deep dream has ever been integrated. Into the. VR space it's. The first time and we, didn't expect anything we just sort of did it there was no like this is a story or this is because, of X or this is the great conclusion we came to there was no great conclusion we, just wanted to see what would happen and. Here's one of the first. Tests. We did which. Is with the waterfall shot here's. A bunch of eyes. It's. Kind of crazy, I. Don't, know if those are foxes. This. Is a some. Of its kind of bird-like I was let it play because I've been talking for a while and snakes.
Some. Snakes. Or. Worms, snakes. On Arecibo. There's. One that we've gotta wait for this not. This one it's the next one. Some. People are like it's like acid, and then the people who took, acid so I could snot like us. And. All the eyes that. Appear it's pretty great um. Anyway. Um. If I were to attribute this to anything that has existed in the past I guess Eadweard. Muybridge did this with with, again, with another new piece of technology at the time which, was a camera, and doing fast photography. Rapid. Photography, to capture everyday. Pieces of life so like a woman, jumping over the stool which, happens, and a, guy, in his underwear throwing, a ball and a, man tipping his hat but, what was great was the the actual the horse, running is is a famous, example because, at the time no one knew that a horse could run, like that actually. Painters and artists had no idea that all four, whole legs. Of a horse actually ever left the ground at once for, us it's like of course but back, then they had no way. To perceptually. Understand that and so with this new technology they finally realized that yes in fact all four. Legs. Of a horse leaf ground. And. So. Similarly and, just by instinct, you know we, just took what we had and threw it out to see what we would you, know what kind of horsey revelation, would be revealed to. Us I think the biggest one honestly was the parameter shifts like the idea that you can actually change the, output. And, the intent based upon the parameters that you choose so. Cultural impact. Thinking. About you know. Who. Decides, on these parameters right like what, data. Is using, being used to train these systems. Whose. Data is it who owns that data what kind of data those questions right like culturally speaking like what is its representing, exactly and how do we have conversations, around that. I think for if anything we didn't figure out we, didn't figure this out but we thought about it at length like. What is this what. Does this actually mean. I. Also. Said my stock to various other teams working in, machine learning because I was really fascinated this include bugging demos office. Back, when deep mine he, hated me, back. When I because, they were using deep reinforcement, learning to train their systems on. Atari. Games and probably a lot of you know about this but it was using. Games like breakout this is an example of what it did when it really, learned breakout, like. What no. Just. Crazy stuff and so. I saw this and said you know it'd be interesting if, you. Know we could think about old, films, of you're like you, know Jerry, mouse and. Gene. Kelly and, their. Dance sequence together thinking, about you know is there a way that we could teach a system how to dance if we can learn you, know how to play games like Atari maybe it could meet our games it could learn learn. This I'm. Sorry I'm like can we train this on films. Example. Of what it is. And. They responded, oh yeah. It's. Way too complex, and. I was like well why they're like well cuz we can't this is back then and now you can do it seems but it's like you know then they're like we can't isolate these, two figures, there's like no way it's too messy and, I'm like okay well if but, if but.
You Can do this with a target yes, what if ayat are sized. The. Data and, they're. Like okay yeah. That might actually work so. I. Did. So. You gotta get it you see it, it's. Great and they're like Oh awesome that's not in our. It's. Like it's not in our like marketing wheelhouse, so if we can't do it so I mean it was I think they actually they were also starting to work on alphago. As well so they're like we have priorities, now I'm like okay well, this is fun you know though, but, stuff like with Poe's net and other stuff that I know you're working on which is really great to see I mean you could do this a lot more efficiently than that. Another. Thing I was working on was I started getting me going on like 3d faith this is all stuff that I saw earlier today which makes me feel like I'm so, old or something but it's like there's this this this tool kit that came out from. Like creating 3d facial reconstruction from a single image and. Then. You know taking, a flight image and then reconstructing, a face so, again another Jean but. I took this scene we, are the music makers and. We are. The dreamers of dreams. So. I took that scene and I it. Was not the best pipeline, or workflow that I could have come up with but I was you know doing it myself so I basically. Did. Like an image. I. Took, expert, as an image sequence and then put. It through the that, system, and then exported, that brought it into Photoshop. Actually, and it, was amazing how it actually reconstructed, the other side of his face so, suddenly the thing he was saying to for assault was something he could be saying to you say. Idea that you could take old films and actually reconstruct them and create immersive experiences. Not, by recreating, t
2019-06-27 04:04
