Marty O'Donnell, Highwire Games - 2018 DigiPen Audio Symposium | DigiPen Institute of Technology
Please. Welcome, Marty, O'Donnell. Thank. You Bryan thank you everybody, invite her here you've all had lunch it's, not time to have a nap it's time to listen to this talk, I'm. Gonna split my time with, my. Trusty. Audio. Design. Dude named, Ian Shores who, was, a student here at DigiPen when I taught a one. Semester of a class and. Ian. Was a rock, star he came, to work for me and now he's full-time at high wire games, also. He's. Found, other people from DigiPen to work first like Nick the programmer, and. So. How many we have now from digit before three. Three. Like modern did you push you yeah. And there's some older people too oh that's, true I. Know, so. Did you pen we are extremely, happy this is here and we are glad to poach your students. So. Here's. My laser pointer this. Is my sunset, Brian had a sunset, but, this island is not. Whatever. His Island is Mercer. This is we'd, be Island and this is not a lake this is Puget, Sound so that's the difference there are sunset, views just wanted to get that clear. Yeah. So high-wire, games is the name of the studio I'm gonna go through really quick what this is highwire was founded, it's an independent studio it is founded by industry. Veterans, and we average 20 years experience. Luckily. Now that we have the, DigiPen. Guys there our average has gone way down so we feel younger which is great. We've. Worked on a. Lot. Of sort, of what I would call bleeding, edge technology games. Like weave we. Were at, Xbox, one launch title. Which meant that there was a lot of stuff happening that we were hoping would happen we, weren't sure it was gonna happen, even. Just bringing a first-person shooter to a console was it took some doing and the, guy who really, worked hard on the controls of that was a guy named Jamie Gries Murrah he's our creative director and lead designer he. And I've been friends forever, and it's. Great to work with him the, other guy the technical, director Jared novel founded, naphthyl. Founded, airtight studios, he's he's. Got a great. Bunch. Of people who were engineers, and he's always looking for more engineers we are also are always looking for more engineers. So. We, as. A team have done. All these games or. Worked on these games halo destiny, half-life, second. Son. It's. Just it's it's an amazing team fits all we also have guys from there halo, destiny, half life infamous, these are all huge triple-a. Titles, now. We had the small independent studio, we're doing. Small. Independent things, which is a really, really big change for us and we love it so we're trying to bring sort of Triple A production, techniques to, an, independent. Studio that has very little money which, is fun. So. Obviously. We. Ended, up being, a PlayStation, 4 PS. VR exclusive. Title that's what Gollum is but, we started working on VR, right, at the beginning right when the first. Oculus. Kickstarter. Systems were going out we had guys working, with with that one of our programmers worked with all the early stuff he was an early Kickstarter. Contributor. For oculus, okay. So. We've been working we've been wanting. To know what is it like to actually make a real game in VR and, there, haven't been a lot of games out especially four years ago there were no real games out there in VR and now there are some but we had it's, funny it says. Well. It doesn't say here but at some point we said early mindshare. It's. It's not that early anymore but, we're still okay because we haven't shipped yet but not, a whole lot of compelling. VR games have shipped yet so we still have a chance of being hopefully. In that first swath, of games that are early, games so. The name of the game that we're making is Gollum and it. Is a it's. A game, about an, endless City it's. A small, cast of, character, as there's you play as a as. A young kid, who has an older sister named, sky and. Your. Your, father is alone he's lost his wife you're.
In This sort of desert, type city. And eventually. Your, older, sister teaches, you how to control. Golems, which are either could be the size of a doll this. Big and you run on the floor run around the floor like that or. Ten-foot. Tall stone, giant. Golems. That are made out of stone and clay and you have giant swords, and you hit other golems so, that's of spoiler alert as some of the other golems are bad just so you know. But. What the fun thing bought this game in VR especially, is is that you everything. Is one-to-one like when a golem swings a sword at you you. Don't just push, a button on your controller you actually have to use your Move controller and hold, your sword up here, if you, hold it here it will hit your head if you, block the side it will hit your shoulder you have to block like real swordplay and I, have thrown my shoulder out several times actually. Playing through the game because. It gets pretty fast. Here's. Just a little. Video of some. Of our stuff. And, want, you to watch this this is early content, beta, content. And. Crank the stop somebody. This, is all working progress from about two years ago. The. Nice thing about doing music. Prequel. Music is that then you have music you could use for trailers, so that was music from echos of the first dreamer which is some of the prequel music I did for a goal and, some of this music of course will also be in the game you. Notice when you saw, that crystal in front of you that's when you're a doll you're holding a little crystal when. You're a golem you hold a sword or several, other kind of weapons. The. Hard thing about doing. A talk on VR, is that, I can't, show you or, have you hear what, it feels like to, be VR on a 2d, flat screen standing. Here these. Speakers are not going to let you know what VR actually, is. Because. VR, is going, to be all. Headphones. Completely. Immersed H RTF and it. Totally. Works just like Brian was saying it completely. Works everything. Is perfectly, spatialized, you know exactly where everything is at. Least that's what I thought would happen so. We're gonna talk a little bit about what's. It like to make audio. For VR how successful, it actually is and some. Of the tricks you have to do in order to make it cool, even. If the technology is not doing everything you hoped it would do so, the first thing I want to talk to you about is is just, well, first, thing I want to talk about is I have a, someday.
I'm Gonna write a book called the rule of threes because I feel like just about everything, can be put. Into groups, of three every rules a set of rules I have I have three rules when, it comes to like I'm. Even Brian talked about it today with piano. And. Forte, and then met, so I mean, I would never use metal piano in mezzo Forte, those are two shmoo she in the middle it's like that soft medium, loud that's the three and. Four. Velocity. It's like you. Know fast, medium, slow, and, for. Hardnesses. Of surfaces, it's soft medium, hard, and. Once, you have enough things like that the rules of three it's so, interesting how those threes can fit together and give you just cover just about everything. So. When it comes to like rules. Our developing, games I have, three. Rules so, here's. My three rules for game audio and this goes for every game I've ever made number one first, do no annoying, now. The reason this is important, is because the. Most annoying, thing, in a game almost, always is, an audio, thing. It's. Really sad but. I think you can be so annoyed by audio. And as. A creator of audio if you're the one who actually made the annoying thing you just feel horrible. So first, thing always, when you're listening to whatever you're doing don't do something annoying, number. Two sound, makes it real music, makes you feel this, seems self-evident. But, it's. Just something to keep in mind the. Reason, things really come to life and they feel like they're real is because you've worked really hard at making the sound work and we've heard Nick's talk today about how, he does sound and. Makes those cars come to life you think about those cars would be just dead without that really, important sound now. Sometimes. I think when it comes to sound design we work ourselves out of a job the more you stop, noticing the sound the more the sound is just taken for granted like, of course that's the way it's supposed to sound that, is really, hard to get to the point where people, just it just disappears. Music. Makes you feel because I'm. Sorry all, academic. Composers, music, makes you feel, music. Is received. By the limbic, system in your brain you're, hardwired, it's like sex, drugs and music, all go to the same place that's, a great place to be by the way for the brain. So. Just, remember that music will make you feel and the, third rule is. Everything. That looks like it makes sound make, sound. Okay. How. Many people are our veterans sound, design people, in game business. How. Many people have built, made, and shipped a game where everything you wanted to have make sound made sound. Right. You never get this this, is a actually. These three things are. Aspirational. I I'm, not sure I've ever shipped a game where all three of these things ever came true and I never play a game where I feel like I think all three of these things I just nailed. Everything. That looks like it makes sound make sound that is really, hard to do and it's almost always at the end of the game the artist threw some hey. I made an animated shader that we're shipping tomorrow, and I'm like how does that make sound I don't even have a way to hook up to that so. It's in the game. Silent. It's doing some cool wavy light, thing and it's. Been effect when the artist showed it to me he says yeah this cool thing that goes, Jun's. Room. I said really are we shipping you with the game so you can make that sound. So. I've modified, my rules for game audio to. Include VR. Sound. Makes, it really real. Music. Still makes you feel so. That's it that's the modified, thing and. The. Reason for that is that. You. Put those headphones on, and you're actually in this environment where you can look anyplace you want and in our game you can actually move any place you want smoothly.
You Actually just move. And. The. Sound you hear we know where your eyes are and. We, know where your ears are and, one. Of the things that's really interesting, is that you make little moves of the head and you can actually like, Brian was saying earlier that you can that's, one of the ways you can. Localize. Spatialize, sound you can figure out it's over there you might not know exactly where it is but you move, your head a little bit and we know where your ears are and, it shows that you can tell it is there, so. Now. How do you do cinematics, in in VR we're gonna go through this a little bit. We. Worked with a company called mocap. Now in Seattle to do cinematics. None. Of us have done cinematics, for VR before, there, was no book that I could go read that says here's how you produce cinematics. In VR so. We knew we wanted to do mocap, we. Also knew that some of our cinematics. The. Player character would be in the scene sitting, there and. Would. Be one of the characters in the scene and look around so you're as a character, in the scene you're, essentially, 360, degrees you could look anyplace you want while other characters, in the scene are, doing, the cinematic, script. What's. Also cool is that, of course there's no book that's what I'm saying well we had to make it up so we made it up pretty much trial, and error like. Well I think this will work what do you think I don't know let's try it so. Trial. And error everything. We did. The. Third thing was what's. Interesting, is that. If. You look at a camera in any movie when you look at the camera you've broken the fourth wall now. In VR, there. Is no fourth wall really because, you're in the scene so. The characters, actually are going to look at you and that. Actually, is amazing. Because when a character looks at you and VR. You're. Like oh gosh they they see me I'm actually here, I'm in the scene I'm actually twine, the little the kid brother, sister, whatever whatever, you want of sky.
And The, sky is my older sister and my father is sort of this big mean gruff guy and, they're periodically, because people look at me so. In order to do that on the set we had Jamie, would. Actually, take. A position in, this on the set and be the person that, will eventually be the player, and, the. Actors did, their scene and every once in a while I would say hey, this is a good time for you to just glance back and look. At twine instead, of seeing a camera there or some bunch of ping pong balls that represent, the player they, were actually seeing Jamie and they could look into his eyes and then, in the game itself, we have eye, tracking. So. If you're in the game and, you see somebody look at you when they're talking to you you, can move your head and, they. Their eyes track, your eyes as you're moving around which. I didn't know is that going to be creepy. It. Actually isn't creepy it feels kind of cool feels like there's somebody actually there talking to you so. One. Of the first things we had to do was. Try. To remember what the next thing is here oh yeah, yeah of course. This, this you've seen stuff like this before here's. So. So. Real-time facial, capture. And. Just. To see okay this is gonna work this is how we need to do it but we what I wanted to do was instead of just performance. Capture which is getting everything all at the same time we. Wanted to like and, I've. Trust me on halo and destiny we've done, mul, cap we've done perf calf we've done what, we call face over, where, you've already finished everything and then you just fake it and have somebody redo. Their face while. They're listening to their voice. This. Time we did it a little bit differently, we. First. Of all we went to mocap now and, drew. Taped. All the, scenes. Color. Coded for each scene we were gonna do some. Things had. Sets. That we needed to put in there and of course with, mocap all you need is something that takes up a volume you, don't build real sets, this. Over here is the room where the face over or the facial, capture plus voice capture, happens at the same time so. What we decided to do was. Put. The actors, in their suits build the sets get, them off book rehearse, this, get. The best rehearsal get, the best take we could on video, and then. We went in and, chose. The one we like the best had, them redo the voice and face capture, based on, the. Best rehearsed. Thing. That we captured on video, then. We used that voiceover. And facial, capture we knew that would be the face and this. Was now the the. Feeling of what the actors needed to do was in their bodies so we went back out and. Played. Back in the studio the, voice recording. And. Watched. The, video. Up on a big screen if they needed reference, but they redid. Their bodies. Action. Based on listening, to their own voice that they had just done so, that we had really good clean voice. Capture, and, facial capture and, then. Body capture, so, it was a little bit tedious. To do these three steps but that's the way we did it. Oh. Yeah. This is kind of fun this is you'll, see this scene coming up here, this. Is the father this, is twine, sitting, here, I, know this is Skye this is my sister. Well. In. The game and this. Is the wagon so I, don't know if you could see this in the darkness but there's inner tubes here so. The camera is actually. We're getting camera, motion. Here, based. On the fact that this wagon, is sitting on inner tubes so. When the player when, the with Skye walks on this it actually moves a little bit when we're. Pretending, like the horse is drawing this where we had somebody actually move. The wagon, just randomly, a little bit, and. If, you look at this shot, here, there's, Skye this last little position of the father and Skye. You'll. See the the Pro Tools session where, this movie shows up there's. Jamie he's putting his head about, in the position, where. The players head will be so. When Skye needs to look at you at twine. Which, is the name of your character, she's. Got eyes to look at which is cool, so. He the next thing after we did this we got it roughed up, now. I'm having to do with the normal thing right I have to sound design Foley and music, and score and do all that stuff just with a normal, movie. In Pro Tools so. Here's that. Next. Time we'll, stay home and, take care of twine, in a few, years, when you're older, maybe 17. The summer. I'm. The, best scavenger, in the village, without. Me you, come home. This, is you right here that's your empty head. So. What. Was interesting about that is like okay sure that works I hear horses I hear cagin. I hear them talking, what's. Missing of course is a, whole bunch of stuff we don't have any outdoor ambience, we actually are moving this wagon through a real environment, so. Ian. Put in all sorts of great ambient. Sounds that you actually move through while you're in the wagon.
And. Of, course every sound that's been fully tuned here has to be emitted from where it's happening. So, the horse sounds have to happen where the horse hat is and his host happened to have endowed where his hooves are and. There's. A little flag flapping and, there's wagon, creaking which, is happening from several emitters, her. Footsteps, are happening from where her feet are, of. Course their mouth their. Head is what emitting their dialogue. And then. I had well here's a good sound of just wagon wheel rolling, well, there's four wagon wheels so, I you. Had I, didn't. Think you'd have to do this but you really do there's in, the game there's a wagon wheel there there, there and there because. As you sit there you. Move your head around and that stuff better. Be where it is otherwise, it just stops being real and as soon as it's real it just disappears, and you think yeah I'm in a wagon, so. A lot of work just to make you in a wagon. This. Is fun he's. Now the giant, Gollum he's. Playing back there for a pre-recorded. Voice. No. He's at. Least three times taller than she is so we had to do this as a separate take. And. This is our animator Steve. He's. Just placeholder. Tell. We're making this up as we go along you're. A little taller than sky for the scene. There's. Jamie, and. Because this guy's so called Jamie has to hit his head way down on the floor to be get the right perspective. So. Then we have to we actually did a video reference, of where the player would probably, be when the Golem walks, into the scene and. He, hits it with a big piece of vacuum, cleaner. So. This is the perspective you're gonna be in the game. Gholem. Stands up. Can. You believe it this is a spoiler alert some of the golems are bad already told you this. Yeah. So. You can see how sound needs to make this real alright, so here is the Foley session, and some, of the music for some of that scene right. Here this is where the character, your twine character will be once again no head because this is where you're looking and, you could look 360. Degrees any place you want there's. Your sister. You. Should see your face right now. I. Forgot. What it's like to see your first column. So. Remember every sound you hear has to be, low. Come. From an emitter in this scene someplace. What. Happens next I don't know one. Of the other cool things about this is we actually so, in some of the early cutscenes of, cinematics, you, sit in a position and you can look, any place you want to 360, degrees which, means that every every single sound has to be cut up placed. In an emitter and triggered, at the right time. There. Is a there's. Some scenes later on where you actually can move, wherever. You want to move during a cutscene, which is, very interesting I won't talk about that because that actually is a spoiler. Now. One thing it's so funny listening to Brian talk, about the error of. Spatial. Izing audio. Doing. As many movies and games as I've done there's a certain amount of cheating you can do you just sort of like laugh that, no one's gonna notice that right I can cheat that, and. So, there's a scene where you're, in bed because you've been hurt and your. Father is talking to you and he's right there looking right at you, so. There's footstep, sounds for him there, was a little candle, sound over here. You. Can look anyplace you want but you can't move.
He. Reaches. Down picks up a crutch and, then. Gives you this crutch so, for, me that was Foley meaning, I was just I needed to make sure I had footstep, sounds for him his. Dialogue, came out of his head and the. Foley sounds, like his movement, sounds the son of him sitting in his chair, we. Just had all those sounds coming out of sort of the middle of his body so he, reaches down picks up this crutch and it's all coming out here and then he reaches over and does this now you're sitting here, okay. So, I swear, this arc of error. Is exactly, what you showed because. As, I watched the Foley sound all the Foley comes from here he's moving, and he, places, the crutch by reaching out and. The. Sound, of the crutch hitting the bed and the in this little chest. Still. Sounded like it was coming from here and when. I I didn't, know that until I put everything into the engine and put the, headphones, on and put, the, hmd. On had mono, display. And everything. Was working great father, moved. Crutch. Was there but I heard the crutch here it was just about exactly, that arc that you were saying I I noticed, that sound, so I had to take the. Crutch sound out of. The. Middle, of his body fully emitter. And stick, it on its own emitter, so the crutch has its own emitter so it goes like this taemi, never healed the same I'm. So, sorry. Here. For. When you're stronger. Or. I can, always. Carry, you around on, my shoulders right. So. The, amazing thing was I didn't, I thought for sure you can cheat that that you, saw that video right it didn't look like there was that much distance, between where. His body was and where the crutch ended up but, inside. The environment it. Was just enough of an angle where it stopped, being real and I. Thought who's, ever gonna notice that I couldn't live with myself I had to I had to cut it and redo it so. What's. Next I think this is the opening cutscene and, I'm I don't want to go into IANS thing. So. What's this see better. Falling, off a clip or something. Wait. Did. You fall off the cliff. Twine. Here's. The thing well HRT F really work and convinced me well, I'm sitting here. She. Walks behind me she's talking I hear footsteps, now. I could, actually twist, all the way around and try to see her but that's nobody. Wants to do this they do the natural, thing. And. Then she sits very close. Was. This nice to have music. I. Miss. Mother too but. You. Can't keep running off like this. She. Wouldn't have wanted that. So. That's. Not the final mix because the final these are just the. Foley and the chunks, that I know need, to be represented, what's. Really fun about this is her voice is far away because she's far away so. She comes, through space she gets louder as she gets closer her. Footsteps, are far away until, they get closer. She's. Right next to you and we were sort of wondering what we don't even know what the distance is of like she's too close she's invading my personal space but, we I think we guessed pretty good it feels pretty good pretty nice. So. We, did there's no book on this right we don't know if somebody's like right there you might go I don't want to play this game it's too much but this is actually pretty cool but. Once you do all of these things and of course then we do, it environment, sounds, and they're all around you and the. Environment like, the birds is over here so when you move this way he's over here this, way, all. That stuff works really really well because. You're always moving your head and looking where you want to look and sure, enough I, had Brian come out to my house when I was working on this particular one because I'm like Brian I don't think HRT F is working and. So here's two people, have been in the game business law time and. He's listening to and he goes. And.
Yeah I'm not sure. So. If both Brian and I weren't sure this. Shows, you that there's a lot of there's some distance to travel before this is like the, matrix okay, we, finally when we finally did get HRT, F working and not just a simulated, 7.1. Which is what I had which, was apparently what was going on at the time you. Get some really great panning, in seven one but you don't get real HRT, F sure. Enough she sounded like she was walking behind me and, I, think he is gonna show you this other thing which. Is a crystal, that you hold and it's. Something that is very cool and localizes, really, nicely and. Let's. See, oh yeah. Here's. Just sort, of a fast movie, of us setting. Up the sets tearing. Them down setting, up new ones tearing those down, there's. The wagon. And. Here comes the bedroom with, the stairs. So. That's all you need to have you, don't have to be fancy with your stuff it just has to be correct. For the environment, you're building, so. Ian. Is going to talk a little bit more now about. What. It's like to work for me. What. It's like to do actual, create more audio in VR and some of the things he learns and I think he's probably learned, you'll. See one of the very first thing he's learned from working with me. I. Didn't. Practice with his I practiced with mines, so. This is like, Murray said I'm an audio designer at high wire games I, do sound effects and implementation. And this. Is my little portion of the talk three rules I thought I knew about VR, audio and how I broke them it's taken with the rule of three it wasn't planned to just happen like that and, then, I also have a bit at the end three new rules if I make four three four VR. Audio, okay. So rule, number one you're. Not supposed to use effects, after you. After, you render in by normal because that's gonna mess, the imaging. Right, and. Okay. That's actually I should before, I say that I actually, think all of these rules are good rules for you to know, the. Point is not that they're bad rules and you shouldn't follow them but that. You should be okay with breaking rules and sometimes there's good reasons to break the rules so. It's. True you definitely don't want to put like a ping-pong delay afterwards. Because then you're not gonna be able to tell, where anything's coming from but, H, RTF has, this tendency to make things quieter, and we, had, a problem for a while trying.
To Get things loud enough and I tried. A bunch of different things they weren't working and I remember I was talking with Brian Schmidt who's like the star of today's show for some reason not, for we, know why but, he, gave me this advice he's like well just try it just try a limiter so I did and, it worked and things were louder didn't, mess up the spacially spatialization. So, but a bing about a boom that, one's done all, right broken rule number two never. Bacon URI verb and again this, one is true. You. Want to really work with dry samples, and use the environment reverb, and the attenuation so, you don't mess up your your. Distance, cues. But, that doesn't mean you should never use it I have two. Examples, for this first, one is your big old sword or, anything that's, naturally, reverberant. This. One's kind of a gotcha but if you bang, your sword really hard on something it's. Gonna reverberate, and so we bacon, just a mono reverb into that and put that on the. Sword so, like, say, you hit. That. The, hole him right there. He's, one of the smaller golems, they're scarier, ones, so. You hit him and he screams out in pain and, you. Hit him really hard with your sword so it's wringing out a little you swing through you. Still hear him screaming, screaming from there and wherever you put your sword you can hear it ringing around it's really cool effect. Example. Number two is not that but that is a cool screen shot. Example. Number two there's things that are never gonna be I'm, sorry camera people I'm gonna be moving around a little bit it's just my thing. There's, things. That are never gonna be close to you like. Bats, in a cave or birds. Off in the distance. And I have some really nice reverbs, in my DAW and I. Don't have as nice reverbs, and wise. It. Just sounds better so, you do it right you want the game to sound better and. Then along, those lines there's other things that, like.
You May want them to be really close but. They just don't sound good in in. The environment, reverb so, an example of that is like water. Drips or rocks, falling. Like. You might want to have a little dripping water they can go right up next to but I couldn't get it to sound good through the environment reverb, and, so, I just made the decision put, it through a nice reverb that makes it sound far away and put, it in a place where you can never get that close to it. So, never bacon reverb, ex no thank you. We broke that rule too and then, broken rule number three insert. Sound here spatialize, is best this is kind of a more in-depth one which is why I say insert sound here I actually have three examples of, insert, sound here so the first one is the. Human voice again. It's, true the human voice actually specializes really well in. That scene, that you saw earlier. You. Put it on skies, with skies that character you put it on skies head, she, has her voice she sounds far away she goes behind you it works the, going behind you part is really cool, the, acting is cool the story is cool but as far as like the the, sound quality of the voice dialogue. Is king you want to make it really interesting and you want to like you know focus on the dialogue but, it didn't really pop, out as being really cool it. Was just right, so. We did a couple things to add interest, to it, one. Is you, really accentuate. The. Environment, reverb er we did just. A little bit and. So. Dialogue is never one of those things that you bake, in reverb 2 because, VR. Is really good at being intimate. Sky. At, a, time, she's far away and, you get the Sesame. Street thing, where she's yelling and there's more reverb and she, gets close and in these nice and intimate moments you want to be able to suck away all the reverb and like. Have really nice dry signal and hear the contrast, so. If you if. You're a little more you. Little more emphasized. Reverb you can hear that effect more. And. We also do other things add interest to it that aren't necessarily, realistic, like. When. She's yelling over that, cliff she saw in the thing she goes twine. It. Wasn't in there because its environment. Stuff but she. We. Just put it through one of those slapback, delays. Twine. It's. Kind of subtle and, it's not realistic at all but it sounds really cool. So that's. Part, a of broken, rule three broken, mono. Sound specializes best again. True. I don't, know any spatial Iser that, works. With. Multi-channel. Sound other than dropping, all channels but the first and then specializing, that from the position, but. There are instances, we. Use multi-channel. Sound in VR spatialized, in the world. One. Of those is let's. See kind of a. Typical. Use case of like 2d blend or spread. Where. You have something. That's far away you want it to sound point-source and when you get close you want to be a lot more wide we, have these insect, clouds that you can actually walk inside so, when you walk inside of them it goes full, hard stereo, and, you can hear them all around you and you hear movement, it's pretty cool, there. Are some issues. Well. There's like one little issue with that with stereo, in particular, and that if, you have like kind of a virtual emitter sort of thing and they're right here and you turn 90 degrees this way then, you hear like a really, close to mono image. But, that wasn't really problem. In this case cuz you you know you walk in you're like whoa this is cool and you turn around you walk away. So, you're never you're never like oh walk in and turn 90 degrees and sit my head there for a second, and listen. If. You did though it wouldn't sound right ah let's. See I, had, a second, example but I did make a mistake my, presenters, notes are over there. Oh. Yeah. So. One, of my favorite parts in the game I, forgot. About it one, of my favorite parts in the game is you go underwater you, take your golem underwater and. That. Has no, H well it does, have some major teeth but for the cool parts it doesn't have any H RTF. There's. Two parts to that that don't have HRT if they're really cool there's the underwater ambience, that's just around, you and then there's also the transition, effects that cover up the crossfade. So. The underwater, ambience. Just. Kind of didn't make sense to use HRT F in the first place, because.
HRT. F is really good at externalizing. Sounds, and if you think about when you go underwater in a swimming pool everything is like really up, close and you feel really claustrophobic, so, HRT F wasn't gonna be good at that. And also like speed of sound and water is something like four times faster than an air so. The. Itd would probably be messed up but, again, we're trying to forget the talk or forget, the tech and hear. What it sounds like so I just found like a. Really. Cool stereo, ambience, it sounds like you're underwater you know, like some sort of like slowed down hydrophone, typical, thing it sounds really cool a, crossfade, to that no H R K F whatever sounds. Really cool and then the transition effects so. You have, like submerged, transition. And emerge transition. You. Like the emergent, particularly it's true for both of them but I'll talk about the emerge. You. Come out and you want to hear like some nice stereo, effects of splashing, on either side of you and. I. Designed. A cool sound for that and. Bryan, earlier, mentioned, that, high. Frequencies, get sucked out a lot of the time in HRT F. And. So those like nice crisp water, drops all. Around you they didn't really play very well. It. Was HRT F it sounded spatial didn't sound like really wrong it just wasn't that cool. So. Just. Did nice stereo recordings. And, it's. The kind of effect where you know you go underwater and it's. Like maybe a two-second, effect so, you can move your and you'll hear some sort of movement but it's a it's again like the bugs you're never gonna have time to really go like, be like this and go oh that sounds mono it sounds wrong sorry. If I'm going off mic. Alright. So those are my two examples, mono sound, not. Always the, best use and then, this third one I don't. Know what kind of time I have left, cool. So. It's. Rule of three but, I actually want to talk about more, than just real-world sound so I'm going to roll. Some other stuff into real-world sound. So. A real-world sound suppose, you supposedly, spatialize, is better, than a lot of other sounds, I just hit a button and. That. Is true. That is true for the most part like the human, voice we already talked about that and also, running. Water is a big one it's like the survival, oriented, sort of stuff I've. Had to turn down the attenuation, radii. Of running. Water a lot because it really just grabs your attention, before. Other. Sounds. But. That doesn't, mean. Synthesized. Sounds don't spatialize, well so. There's, this dream ball and, you. Saw a little bit of it earlier, it's when. Usually, when you see it it's in this like dream void world. And. Yeah. Making, you work. So. It's. It's. Facial eyes is really well and I'm, gonna try. And guess why it specializes really well I think there's two reasons one, of them is yeah it's a really broad band sound there's. A lot of like Chris, penis and it, varies. A P but. Also, non. Sound reason and I think there's a lot of non sound reasons that thing that sounds, can spatialize, really well is.
You're Just really focused, on it because in the world there's not much sonically, going on there's not much visually, going on and. There's not much to do except for really focus, on this dream, ball and pay attention to it so you you're. Focusing, on it you expect it to have a sound coming from where it is and it does and it's kind of like the dialogue. Coming out of the center speaker effect, at the, movies where you know you have a character here and a character here and like senator, speaker right here and both of their voices are coming out of that but, that's. How they mix it in movies right, but, you hear it coming out of their heads so I think, it's because you're focusing on it and then. Also. The broadband helps a lot, this, sound though this is the this, is the dream stone and, it does it also has a synthesized, sound but it's not really, that broadband, it's kind of this warm choir, pad ish sound, and if. You heard it on on its own sorry I don't have any audio examples it's hard to do audio examples, for VR, but. If you heard it on its own you'd, think it doesn't spatialize, very well. But. It actually specializes, really. Well and I think that. Is entirely, because of focus. I'm, calling it focused not like the Y's focused parameter or anything it's just like your attention. So. There's. A couple things that we. Do in the game to encourage. You to really focus on this dream stone one of them is yeah like the dream void it's kind of a boring environment. It's. Not a boring environment, but compared to this thing that you're holding it's like, not as interesting. It. Has a cone attenuation. Which. If you don't know what a cone attenuation, is it has like a, direction. That the sound goes out and it changes, usually gets quieter, outside, of that direction. And. That's. Not really realistic, at all but, it's. Well. I mean, this is like a magical, crystal spoon, thing so it's not really realistic but. It. It's. Not realistic but it's fun like. You can manipulate it, one, to one you know what's going on and you kind of just want to play with it and so it's encouraging you to focus your attention and I think that really helps a lot that's, kind of yeah. So, I was going to talk a little bit about this before but I was gonna wait till we got to this this, is one of the most important things it's actually an object in the game that you're holding you, see your hand you see your arm and, it. Looks it's basically, the same as X size as a, PlayStation. Move controller. With. A little thing at the top that's the crystal and everytime I'm checking to see how the spatialization, is, what's which I think is interesting because once, again Brian Schmidt who talks, about everything before we do. If. You're. Looking, at it and you're also the, motion that's happening, is in, your control completely, it's, an object you're holding in your hand and you're looking at it in the game and so. When, you move it you know, that it you've moved it over here so if the HRT F and the spatialization, works, really, well because, your mind is in total sync with your arm so, I moved it over here and it's now it's over here I rotate. My wrist just a little bit and the cone attention continuation. Works so. Every, tiny move you make with this above, behind around.
It Just, works so I'm always testing it with my eyes closed in virtual, reality to. Make sure that the sound actually is doing what I expect it to do and, we worked quite a while on it and I think it's actually pretty cool it's probably one of the coolest things we have other than on the sword when. You become the big golem with a sword there's these little metal rings that, have physics. And me at, first just like oh, the. Designer is like we don't need to make those make sounds and I'm like oh my gosh they have to make sound the first thing people are gonna do is see that thing and they're gonna shake that and if, you see the Rings move and you don't hear them going, think. You're gonna be really depressed. Thankfully. They all work so go. Ahead. Oh, no I actually, had one more but, this is one of the non real world zones so it's. Just another example of focus. So. We have the, some loot in our game you can pick up you're kind of like Hoover, it up in a vacuum mode. Luigi, style and. You. What you can do is like the game is really focusing, or encouraging. You to focus on it you have this really, severe tunnel vision you can only see through a little thing and, one. Of the only things you do in that mode is click loot and everything else gets low past and. This, sound, if you if, you, cancel. It and it falls to the ground there's, this little physics sound that happens and it's nothing special it's not broadband, it's not. A long looping sound where you can like reposition. Your ears and see where it is it's, kind of quiet so it's you actually told me to turn it up the other day and. And. It's, it's. But. When it hits the ground just this little dink it sounds like it's coming from exactly where it's coming from and that's not like something. We tried, to do who's happy accident, but it works really well and I was trying to think about why it works really really well I think it's. Non-audio. Reasons. Dare. I say. Alright, so the, three rules. That. I thought I knew they are all broken. And. We, have time right for three new rules I think so. Rule number one everything. Is a sandbox. So, like Marty said the. First time they get the sword they see the Rings they want to play with it right and there's, like golems, often in the distance that you can fight in this whole city that you can explore but the first thing you want to do is hit the rock and, hear the sound and hit the tree next to you and. Even. In a cutscene when you like traditionally. Don't really have that much agency, you, have control over the camera so, like if you put, interesting, little things in the world like the, window behind you and you move oh there's a window behind me and we're. Like that candle. Like. Candles barely, make any sound but we have like a little fire sound with this really extreme. Attenuation. Where like, the sound, really changes, a lot as you go and it's not realistic. But. It's fun. And so. Since. Everything is a sandbox, and VR. Is just fun to play around and just, the. More you can place those little subtle things to make it to. Emphasize. That fun I think. That works really well. Next. One everything. Has a velocity I, have. Spent so much, time tweaking. Velocity. Curves for. For everything, because like, it's, not like you have the egg press. A and you do the quick attack and it like can hit a couple things and like, three different speeds or something right like. I have a sword in my hand, and, there's. Like a tree in front of me and I can like lately Tapan and why, would you ever want to do that I don't know maybe because it's a sandbox and it's fun you're just playing around and you can also swing it really hard and like throw your controller across, the room and break your stuff, and, a. Normal game play like it's usually somewhere, in between but you can do any of that stuff and, it's.
Not Just the sword it's like you know you hit the tree and like. Just like when. The sword you, hit it really hard and you get more of a ring out from it the tree hit it really hard and like it's, not just a funk from the from. The wood you're hearing some of the trees rustle, mate or the leaves rustle maybe for a little longer. So everything, the. Most important, sounds the ones that you hear and most have, like velocity. Layers just like a sampler. And. Then. The, some of the less, important sounds they're, just like kind of volume but basically, everything, especially physics, stuff it has. Really. Finely tuned our TPC so I just spent a lot of time on that and it, helps, a lot and once, you get it right like Murray says you, don't notice it. And. Then the last one this. Was. My, favorite I really like doing environments, out on this project. It's. Really, hard takes, a lot of little tweaks the. The. Most, let's, say the hardest part is portals. Entrances. Between. Two, different environments, and. Wise. Now has the. The, portals feature, which. I'm super, excited to try out but, golem. Is on an earlier version and it doesn't have that so we, had to come up with our own solution. So. When you walk into a. New. Area imagine. You're like in the portal and. You. Know you trigger a crossfade, it. Doesn't. Work because like a lot of times at. Least me when I'm playing I like go to this new cool area and I'm like at the threshold I'm like whoa this is cool where did I come from, and if you do it on a on a timer. Just. A crossfade, then you like you're like hearing the stuff change around you and you're like still half in the house but it sounds like you're outside and it just like completely breaks it so. What we ended up doing. I think. It worked pretty well it's a little finicky sometimes, but. Works, works. Well most of the time is we. Have, like. Environment, a and environment, B and. Most. Of the sounds in the environment are 3d emitters but just, in case I messed up at some time and you. Know there's no 3d emitters around we, have this little room tone around you that's attached, to you so, you have environment, room to Ma and environment room tone B and they're, on, a state, in, wise so like we we, hit state B and it does this crossfade, and at the same time there's an entrance emitter or maybe, a couple entrance emitters and, it's inversely. Tied. To that state so, if you're in room a it's playing room, B ambience, but spatialized, from the doorway or the. Portal and as, soon as you hit the trigger they, both cross fade and. And. Now. You're hearing in your head room, B and from, the door you're hearing room a so. That works pretty well most the time. Another, environment. Sound is difficult, I have to look at my presenters and that's again oh. Yeah. So. This is a pretty cool thing. The. The. Room tone I was talking about earlier the, non-3d well it is kind of 3d this. Is kind of a popular thing to do in VR audio. Where. You have like usually, like a quad, a set of pods that. Have. The. Location. Of the. Listener but they have the world's rotation. So. What. That means is like if I move. Around, they're, gonna move with me what, if I rotate, like, they're. Here I rotate. They stay there and. I. Think so, so we have those they're. Not quad though I think, most, people when they start looking into it into. Environment. Sound for VR audio look into ambisonics. And. The ambisonics is really useful but we didn't find it useful for our room tones, and. That's, just because there's not, a lot of content for it and it's, not super malleable I could. Use something like the ambisonics toolkit, to, make my own ambiences, in.
Ambisonics And that is cool. But, it takes. A bit of time you know and I got a lot of ambiences to do so just, keep going. So. What did I look at next I looked at like 5.1. And 7.1. Ambience. Libraries, and it was kind of a similar story with those. Not. A lot of content, and there's also kind of expensive, so. I looked into quad and, quad. Worked. Pretty well. At. This point I'm thinking like okay I'm making my own content, for it what can I do that's a good, balance between. Specializes. Well and is. Not a big hog on my time. So. Quad, it's only four channels I can like probably cut up the stereo file and, I. Don't know use the last half as the back two speakers, or something right so I tried that. Sounds. Pretty cool actually, I think I tried duplicating. I tried, a couple different ways. And. It, sounds pretty good but, it has a little bit of a stereo problem where wouldn't you go like this you get a get a mono sound field, with. Quad. It's, just more. Mono it's more Center heavy. And. It, wasn't that bad but I thought we could do better. So. We ended up going with the the, solution we won with in the end is like a trilateral, speaker. Arrangement. Sorry. So. You have three. Speakers equidistant. On a circle, all. With 120. Degrees, apart so, you have one here one here and one directly, behind you and they, have world rotation, and. It's. Pretty simple, to make the sounds, I. Like. If it's a 30-second ambience I take 30 seconds, of left, and right channel is a stereo ambience and then like 30 seconds of one of the, those. Channels from, not, that mark green. N and. I. Was, a little worried about the correlation at first like is this gonna sound like, it's all the same thing, and, yeah. There's like correlation. Obviously, between left and right the back. Didn't. Bother me so. It's good and, then. As you. Can probably tell when you turn around you can never get to a point where, things, are like really trying to collapse inside they're all pretty much same. Ankles around you. So. Yeah. That's three. New rules for VR audio and if I may one, more rule, don't. Forget to break all the rules all right, so that's my, part hand it back off to Marty, thank. You. So. I'm just going to close it up we've been having a lot of really good fun. And when, Gollum comes out you should get it. It. Should have been out by now but it's not because making a. VR. Game is actually really hard we're working with Playstation. And stuff and, we're working with unreal, stuff, and we're working with wise stuff, and we're working with physics, stuff and like, the. Only way you can tell how it all works is once you get the entire thing built, and you put on all, the, accoutrements. And get to actually see in, here what it's doing and it's. Amazing, how much it changes, we're starting to anticipate I was going to end up being as we get to that point but it actually takes a lot of work to get to the point. So. This, is Gollum get it when it comes out but if you really want to and now of course this is I, would. Never actually, push anything, but. Because. Of the first dreamer is available, now so you can get it when if you feel like it and want to hear some music you can go right online and buy it all. Right thank, you very much. We've. Got time for some questions looks, like mr., Philip, Clawson has a question are you just stretching, he, does okay. So. I have a question about. Reference. So you, mentioned a little bit about, moving, and testing. The sound with your eyes closed Marty when you're removing that thing that's really cool. Because. With. VR. Like. Maybe just letters do you ever close your eyes for, reference. Checking your spatialization and then do you have another standardized. Thing. In your game to say ok this, is what I want my 3d reference to like. This is what I want 3d is down like do you take, the headphones off and tell Ian to clap or, yeah. You know it's funny too in order to make this all work those, of you know how these things work you actually have to have the headset on then, you put the headphones, on it's actually VRS, kind of clunky right now but it's really compelling.
So. It's actually weird to put all this on and then close your eyes but. That's what audio guys do so. Every. Once in a while like. By. Closing my eyes if, it's and, I'll go to Ian and I'll say, yeah. Those guys need I need I'm not getting enough occlusion, obstruction, on them I hear, them and they seem like they're all around me and I want to hear them like when they're over there I want to know their their verdict. Verticality, is really tough but if you see some goal and go up the stairs and then he goes behind a wall you want all that to work with occlusion, and and. It. You feel like he's above you because you saw him go up there so. But. Closing, your eyes and trying to imagine what's going on it's a really good test one, of the other cool things is some of these golems actually throw these Spears at you and of course they sure, when, they go and hit. Behind you and you can whack them out of the air as are coming at you and, those actually spatialize pretty well we we've got them nice tight mono, loops that that go past you and and you. Can tell where those spears are so, you can it's, possible to actually keep. Your eyes closed and, dodge. The spirit, which is it's a game into itself yeah, there's actually. There's. There's a couple levels that I do a lot of the environment sound and I can, walk through now, with. My eyes closed I can get to the end of the levels like the catacombs, mhm I. Don't. I don't think it's anything I did I think it's. Yeah. With. All those 3d sounds, you can sometimes, just close your eyes. So. You found there's any issues, with music. Being. In a VR space, but it's not diegetic, in the world it's, just. Part. Of the emotion. That you're yeah. So I think this, is why I say music still makes you feel so music makes you feel it's. Not part of your reality and, neither is narration. Right, so. I've made the aesthetic, choice and we'll see whether people like it or not but to me music, should just be right, here it's. Like where music is in movies and TV and other games, music. If music starts moving, like. If the you know I have this nice you know beautiful stereo image. And. This you know strings are over here and then, I'm actually in the game and I look there and the strings are still there I'm gonna go okay where the string players they're in this world so. And if I have a narration, and, I. Put somehow have that also or my if the narration moves as I move my head or the narration, stays, in a physical spot that, that doesn't work either so I, don't. Think anybody's, going to question when they are, looking. Around and, they hear the birds you know stay pinned. To the environment, and they hear footsteps behind her and they hear dialogue, but, the music just is there I don't think anybody's gonna go gee why isn't the music why isn't there Orchestra in this environment I think that nobody. Will question that, so. That's that's, the decision I made I think it works we'll. See. Any. More questions for Marty Orion there, we go drew. Yep. So, did. You try different binaural, engines, and which one did you settle on. Whatever. The binaural, engine. Is that comes with wise or, is it Sony as, Sony's yeah since, we're using. PlayStation. It's, really limited it's the, sony proprietary. Tools or I think you can use ro headphones, well. Yeah we're using the Sony proprietary, tools which. By the way like have no controls, it's like on. Exercise. Anybody, here knows how to turn the, volume up on the Sony on, the hmd like. I'm not kidding you plug the headphones in, you, turn it up all the way and I'm like. I. Want to go so loud that it hurts my ears so I turn it down a little bit like, you know I like running at seven and a half or eight but. I have, to go all the way to tend to get it loud, enough in the headphones, so. Does, anybody know how to do. You I. Said. A lot of that depends on the impedance of the headphones well I, have Sony headphones and, then I also use the earbuds Sony gives me. So. They ship headphones, with. Well. That's something I think that they should look at. I. Wonder. If it's like. Giving. Dynamic, range to or. Some. Range. To the convolution so the all the HRT, FS you. Know have these curves and how, it sums and all that so they're they're, making sure it doesn't distort, but.
That's Not their decision but, now it like Brian, yeah cuz everybody knows this back of the xbox days like I don't you tell me what that ceiling is yeah I'm. At the same issue when I was doing you, know 360 video, audio. And. On. A title, and yeah it just like, I I, had to just pull it down and I didn't know where it was gonna clip but we were on. Gear. VR at the time so yeah it was just I. Don't know what it was a ceiling, I couldn't see it yes. See where the ceiling is and I want to say okay let me go as close to the edge as possible and, and if. I if, we get into distortion, territory, like. I'll take the blame but don't, suppress. It for the lowest common denominator, and make. It so we can make. Things pop this is why we had to put some compression, limiting at the very, end of the chain just so we get something. That feels louder right, I see an aura band binaural, meter coming, soon I hope so. Of. Course you could also run the headphone jack into an amplifier and turn it up which, is something we're doing but most, people won't Ian when you made the decision to go with the three, sources. On the MC rule of three it worked apparently. Where. Did you make that change did, wise do that or did you do that in the engine yeah here I'll come out here so you can see me a little better sorry thanks. So. That's in wise using. User-defined. Positioning, you. Can just set. You. Know on a two deep well actually in a 3d space. It's. In the positioning, tab you do 3d. Sound but, and so doing game to find you do user-defined, and then, you uncheck, follow. Listener, orientation. It's. Pretty. What. Defines the three points. So. The. Origin. Point in that volume, in wise is, the, listener, and then you, have these, things called paths, where. You can like if you have a bird flying you can say okay it's gonna start here and at five seconds it's gonna be here that's gonna alert between them, and. I. Just don't, set a path I just set a point and. And the other thing about this particular thing is that we're still relying mostly, on emitters. In the environment, so, as you walk past things, things, are really doing. Stuff but what I wanted to make sure was that you, could never get to a spot in the game where. Suddenly sound. Just sort of disappeared, which happens, a lot sometimes just like you want to have some bass sound, that. Just covers everything but because it's VR it. Can't be as static as I used you know got away with in other games, this, was I think a really good solution we know everything's, covered with something interesting, but. Not super, interesting, most, of the interest comes from the point sounds yeah, that's one thing you told me was make, it so that when I turned my head that. I can hear that I'm turning my head yeah but otherwise. Yeah. Like so even in the opening thing, that you see you're in this void, and, it's, basically, a, stereo. Thing, but it's. Spatialized, enough so that when you turn your head and the void. You. Hear things around you but there's never anything that you're gonna pinpoint, and look at it and just decide, it's there you're, just in this void which, then allows the dream ball that is truly. The. Star, of that scene like. You all you want to do is follow where the dream ball is and it comes around you and you hear it in this space but the void itself is just. Interesting enough that it changes as you turn your head but not specific. Enough that you can localize. Anything. Good, any, more questions, for Marty or Ian. Well. Another, hand for our guess.
2018-09-13 12:36
