Amazing experiences and how to build them – from the people building showcases and demos -

All. Right everybody. Welcome. To the if Deaf live session. At Bill 2018. We've, got a great show for you today we've. Got some special guest, Kilian, McCoy and Jaime. Rodriguez. Musical. Guest Pete, Brown. I'm. Your, host Jeff. Burt oft and now, the host of our show Nikolaj. Hey. Hey everybody welcome, run, of applause for our other host Jeff, we're. Coasts yeah. We. Haven't agreed on that man. So many people here I'm so excited to see everybody here it's an exciting day it's great to be at Bill 2018. We're gonna have so much fun you have no idea man, I flew in all the way from Redmond, and my, arms are tired, oh. So. I wanted, to try that joke anyway, so what, we try to do here is we have some amazing guests, we have Killian McCoy who's coming all the way from our, team to talks about Windows machine learning we, have Jaime, Rodriguez, who's hiding somewhere trying to fix his demo he's, gonna talk to us about mixed reality and we have musical, guests Pete Brown it's, gonna talk to us about MIDI and I guess we. Have some amazing demos we want to inspire you we want to show you what's possible on the windows platform together with the cloud and Microsoft, 365, and we're, hoping to do that with all this amazing hardware that, we have here not. Before we get into it I have some jokes you want to hear some jokes I think. I think we've got a minute you guys hear. Some jokes okay. It's. My stand-up. I guess, I don't, know whatever it's called why. Do you JavaScript. Developers, wear glasses. Anybody. Because. They don't see sharp. Anybody. Uh I. Got, it one more I got a better one I got a better one okay so developers. Are indeed banned these, days right there's one company poaching, developers from other company so, I thought I used this to my advantage and I go to my boss's office and I say hey so I have three companies they're after me I think it's time to talk about the race I think I need a race my. Boss thinks about it and said okay. How about 5% I like, 5% is good thank you I'm, gonna leave his office I'm, happy, with what the answer my boss stops me at hey by the way who's, after you like. Well we have the, cable, company the, gas company and the electric company. Okay. I'm. Gonna stick to my developer, job from now on, okay. So before we get started before, I get our first guest on stage to. Use this is when I do all my development, I kind want to showcase what I've done today we, had some great announcements this morning so I thought I show, you some of the stuff I've been working on if you guys don't mind it's gonna take about 3-4 minutes is that okay I think. What you should do is do it in front of us right okay, your screen add my already, prepared it it's right here who. Saw our videos. That we put out last week. Awesome. If you haven't seen our videos go to AKA that msf. You'll, have a lot of fun with all the videos well who. Saw this. Video where we use a Russian, nesting doll and was. Beyond, creepy I didn't sleep for a night after, that video no. Okay, you, should watch that video I'm. Gonna do a vid I'm gonna show you what that was all about do you want to Scott. Hanselman this for me I think I can hold on. Thank. You it's a verb already okay. I'm, gonna stand down here cuz you know people need to see the screen so I'm, gonna use this guy and let's pretend this guy is you. Dopey application, a saml you do the application and this is gonna be a demo about Sam, you dopey zamel host you guys saw that demo this morning that, announcement no, you, don't know this, allows you to host a utility, out zamel, inside, of a win32, application, so now you can take all the goodness of fluent inside, of anyone, thirty-two applications, such as WPF, and wind forms and. What I thought I'd do is I'll, try to host this inside of a WPF, application, and see what happens so I have my WPF application, put, that guy in there. Like. That and now, have a WPF, application, that has usually P component, in and. I thought what if I go a bit a little, level down I take a WinForms.

Application. Put. That in there and then. I have a WinForms, app, right. There has a WPF, content. Content that has Ulupi content, can. He do it let, me show you this. There. It is so I'm gonna have a virtual, machine for this so here is the Ulupi content, right here it's just the border with, a stack panel and a toggle switch right, here. Is my WPF. View, this is a WPF, control library and I'm using this new thing called sam'l host and all, I'm doing is here in the sam'l content I'm just pasting the. UI the sam'l, that I showed you from my utility application, that's all it is just the same you do appease a mol code that's in there. Now. I didn't stop there as I mentioned I also put this inside of a WinForms application so, here's, my wind forms you wind, forms library this. Basically, just has a border, and inside. Of that border I'm just initializing, the WPF. Component. That has the utility components, of that when runs so. What. We're gonna see now I'm starting all three applications at the same time we're gonna see a utility, application, which has its own content we're. Gonna see a WPF, application, that's hosting the UWP content, and we're gonna see our WinForms, application that. Has a WPF, content, that also has a utility, confidence, out of it ooh. Now. You probably wouldn't want to do this you can go to WPF, or WinForms directly from UTEP you wanna do that but I thought I'd try it out now. I, see. I see you have one more doll. We. Should try it one more doll so. I thought you know this is a this. Is a good scenario but there's actually other scenarios, where maybe. Other developers. Want to use take advantage of this I thought what. If we try, vb6. So I have a vb6. Container, here right here so I wanna see if he and get bb6 to work this way - yeah. Okay let's try it out so, I'm gonna close all this stuff down. Oops, here's, my vb6. Project. And. I opened this up guess. Remember this who, remembers this oh wait. We have a lot of vb6 developers, this is amazing. Nicola. Remembers it from when he was six years old apparently I started. This a week ago for the first time so thank you so here's our vb6 application. And all we have right now we're, just loading the WinForms library inside of this vb6 that's all it is we don't have to do anything magical we, just have to reference the the wind forms a dll, that we created through our wind, forms control library I'm, going to run this and you, can see this is actually a, vb6. Application. Thank. You as a wind. Forms applicant. Content WPF. Content that has Ulupi content, and it just works that's, it what'd, you guys think. Good. Stuff right okay that's.

All I have for you guys thank you for coming now. I'm just kidding actually gonna get to the real people that do a bunch of this will fun stuff so, for our first guess let's go ahead and get them on the screen we. Have Killian, McCoy and. Octopus machine learning. Hey. Killian hey, I'm. Gonna have to hold this we, don't have more than two microphones. So. I think, you brought a clip with you right yeah should we watch the clip we, need audio, okay, let's watch the clip. Pppp. Boo, boo boo boo boo boo boo boo be, pretending. I'm doing the audio. Right. There this. Okay. Once. You explain what we just watched. So. I made an app where people were making crazy faces and, it uses Windows machine learning and a publicly, available onyx, model okay, so can we play with it right now okay. Let's. Go ahead and get that started. Okay. So I'm gonna play with this once you tell me what we're doing great, so, this is the initialization, screen, so, as Nicola, makes different faces into the camera the apps gonna pick it up and the emojis gonna switch. Okay. So Nicola when you're ready go ahead and press start. Okay. Anger, I got to go ahead and imitate angered you'll see there's a countdown in the upper left hand corner and then there's a pretty screen on the right hand side so you can see that everything. Is being captured in real-time and updated and the UI. I'm. So good be neutral that's amazing, awesome, ok what's next so. You'll notice here that. Nicholas. Best face where each emotion, was captured, along with its score and it's posted and a really cute gift I think, you did scary, for, everyone. Is. That an option where, scary I don't know fear yeah. I'm very good at being scary, yeah I'm afraid, ok. So how does this work. So. Everything. Is what show, us how this works yeah. So. I'm gonna go ahead should, I show you how it works in a smaller form with this let's, go through what is this like how did you make this happen how does this work so using one ml I incorporated, an onyx model that is constantly, evaluating, every single video frame that comes into the app and assigns, a score to it and this is happening in real time okay. So what, is the model and what is onyx and how does that work so a model is the result of running a machine learning algorithm, on a robust set of data so essentially, you, give the model and input and the developer, receives an output that is typically, a list of probabilities, and then, these probabilities, are corresponding.

To The type of input. You give it based on the data that the model is trained on right and onyx. Onyx. Is, the open neural network exchange which is the industry standard for machine, learning models it was developed by Facebook, and Microsoft and, so it is the, type of data file that you would use in machine learning so. How do I make onyx models or can I find, onyx models just laying around somewhere yes, so in this app I actually used that one that was available on the onyx repo on github however. You can create your own onyx models through CNC K or you can convert them from other file formats all those tools are available online right so, this, is the azure. Gallery. Which has a lot of these models right so, you can find onyx models ready to use plug and play and win ml and you can also find some at the, repo. On github and there's more but this is the one I use for my application and there's. A lot of cool wins and including, Yolo so you should go and check that out yellow, yeah it's. Called Yolo yes okay and you. Mentioned you also have a sample that we can show us okay. I can show you how easy it is to make something with one mo. So. The first thing this is a pretty blank application but I went ahead and did two things in advance and the, app manifest I made sure that the webcam capability, was enabled the other thing I did is I'm including, the windows community, toolkit to access a camera preview control, that'll make it really simple to get camera frames to feed into the model so. The next thing I'm going to do is I'm going to actually drag in the control for the camera preview and I'm. Going to assign, it a name. And. This, control is coming with the next, version the windows community toolkit yes. That. Is correct, so. After, we have our camera, preview in the sample we're, gonna actually do, the most important part of this app which is including the Onyx file so, as you saw it was avail I downloaded, it from github so we're avoiding that step and all waiting for it and I'm gonna add it to this assets, file a. Motion. For plus yes. That, is what it is called. So. Once I've added this onyx file to the assets folder I actually have, to change the, build, action, to content, to, make sure that it runs and doesn't, throw an error and then. After I do that you'll notice well. Actually you couldn't because it was off the screen but, SES. File was automatically, generated and this gives you a interface. To use the model input output as well as the really cool evaluate. Async function so you could have totally written this yourself but it was made for you so I'm, gonna go ahead and use that can we see it yeah. It's. Dutch baths can be kind of tricky yeah. There. We go we can get us we can do this okay here we go okay so this is just auto-generated. Code that it, just happens as soon as you drag that on yes awesome how, do we use it right so. You go back into this file it's pretty blank the first thing you do with any Windows machine learning, application, is you load the model so, I'm gonna go ahead and include that code here. So. This isn't declaring. The model and then we're gonna actually load the file that, we added to the project and after, we load that the next step is to actually get the input um, so different models require different kind of inputs for this particular motion model it wants video frames and so. We're establishing, that the camera that we included in the sample is we're going to assign those frames to the model input and we're. Assigning it to an event handler which we will now declare, and fill, out. This. Control just gives you frames yes, yes. It handles the initialization and the cleanup so it's really handy so, here. Is the second step of any machine learning usage in one ml it's, binding, the input model so. It's taking the video frames that the camera preview helper, the control is giving us and so. We can actually put in this input and then get an evaluate. It and assign it to an output variable. And. We just call it valid a sync right, so, I'm gonna put a break button here before we run this code okay, I. Want. To stand up for a second there okay, oh man, okay. So while this is running why, should I use because you have the same motion model also on cognitive. Services so why would I use this versus cognitive, services great, so when ml has some advantages, because, we're, evaluating. A video stream which literally takes a frame in every second we want to make sure that the evaluations, are as quick as possible and we can update the UI as quick as possible that, is only. Able. To happen because we're, doing everything locally, on the machine and we don't have to worry about the round-trip time the low latency is a really, big factor for live, feed evaluation. Yeah, okay. This is ready so. We should expect now to get a frame. And then, we gotta hit a breakpoint.

Okay. So. You'll notice here in the output variable we actually have a bunch of numbers so, I mentioned before that you. Expect an output to give you a list of probabilities, you would typically expect that to be between. The value of 0 and 1 the. Reason that this isn't that value, is because it needs to be run through a softmax function and. So. That. Is yeah, look, sham save run, no class for Shan everybody. So. Much better so sometimes. You have to massage the inputs and outputs and machine learning models in this case for, the emoji app you saw before we actually cropped the video frame to only feed in the face into, so the evaluation didn't. Have to deal with extra noise and in this case we we, would typically run the output through a softmax function so, we would get the probabilities, you'd expect and I assume input. Outputs already defined on the azure, onyx. Website, right or github right you can see exactly the shape of the model and what it expects and what it returns and, all of this is documented on their github repo if you want to use this particular model well. Thank you so much for showing us all this is amazing thank you okay. Ron applause everybody. Okay. I think we can move on to our next. Guest, we, have actually. Yeah, I forgot, to mention ahead of time but we. Actually have a word, from our sponsor. Oh I forgot about that yes you talked about first just. So y'all know high quality. Productions. Like this take a lot of money I mean when you're paying a guy like Shen, to, be the one who brings the chairs up you know it's gonna cost you so I didn't realize we actually got paid for this one. Of us gets paid for this and, at. Any rate we, told them we'd give them a little scout out for this so, I'm. Gonna go ahead and take a minute and talk about our sponsor here, so. Let's. Go ahead close this out. What. We have is. The. Company's, called goat insights. Ok, and it's a really cool company to talk about because it actually they're, really using, Microsoft, technologies. Here now, insights, are something that we provide, in the Microsoft, graph and you'll, see that goat insights is is powered. By. The data that comes out of the Microsoft, graph. So. Let's not, we're. Searching for there so, what you get with go to insights is two things first. Of all you get this little array of images, of goats in the bottom because, their goal, is to provide, you. Points, of relaxation, within, your workday, so. That you can be even more productive and to get you to relax they, show you pictures, and videos of goats wait aren't you a good farmer aren't, these your goats I. Are. You are you are you accusing me of fabricating. This whole idea of a sponsor, just to show pictures of yes actually I am ignore. The guy behind the curtain they're, just really cute goats I mean you know who doesn't feel relaxed, when you see this now. The other important, part is the top part of this and. I'm gonna go ahead and refresh because it was when. We changed. Resolutions, it didn't seem to go through but. The top part is where we see our insights. Okay. So insights. From Microsoft. Graph you get sets, of insights, that, look at your data the user's data and you get things like the, Mail's that are sent you, get things like.

Hold. In here. Gotta get this screen loading here you get things like the calendar, and then, you get wrap-up of data that goes with that so. Yes. All. Right. Now. There's. Something wrong with our screen here all right there we go so at the top what I'm showing you is an insight panel and then also this schedule, that comes from, graph so, I'm able to pull what's on there and then, go ahead and display, it in my apps so I don't have to go back and forth between outlook, now. One, of the other things, that I, am. Able. To do here is I'm, able to do things like send notifications. Okay. So I started to send a certain amount of text, messages and then the notification, comes in excuse me emails, the notification, comes in and tells. Me that I, have notifications. Or. Sorry that I've sent a certain amount of mail and then, I go ahead and. Give. Me a link to go look at the pictures, of the goats you know I'm working really hard I need to slow down a little bit now. Also on our calendar. We'll. Come in here you'll see I have, a budget meeting that just appeared on my calendar, now. This is a three, hour budget. Meeting which means it's going to be long and grueling and I'm gonna need a little bit of a break afterwards, so. What happens, is goat insights, will then grab, the next, open spot on my calendar, and schedule. A half-hour goat, yoga session for me that. Way I have a chance to relax now. This, data comes from the Microsoft, graph here. Cieaura, by fire egg and for. The graph you can actually go to the graph Explorer, and what, it'll do is I'm logged out in this case so, the graph Explorer is going to show me what you are Li call to get the data and then. What data I get back, okay. So this data is in a JSON object so, it's really easy for me to consume inside my application, they, also give you all these sample, queries so. I'm going to go ahead and look for this one the people I work with and, this. Is going to be important, for me because I'm gonna go back to my AB app. And, add a new feature here. Have this file called graph helpers, which is making the calls for me. So. I'm gonna copy one, that I already have here. And. Then. I'm gonna update the name. To. Get people, data. Now since the responses. Are all formatted, very similar, I'm able to use this same helper, and change, the URL, and then. I'm gonna go back to my markup, and uncomment. Some, of the markup that I have in there. And. Save. That and then. When I go back and refresh my demo page. Right. You'll. See I have this little bonus that appears in the goat yoga section alright this bonus is driven by that API call where. I'm calling to get the people that I work with the. First person on that list is, the person, I work most, closely with and since, don't yoga is a team sport it. Knows that I'm gonna want to invite my co-workers, to come to go yoga with me so, I'm able to pull all of that data from the Microsoft, graph, okay. Now I'm a PWA, guy so, I really. Can see the power in being able to take something that comes from the graph where we get all that intelligence and combine. It with features, that I can get on the client so, we worked with them to build the goat yoga PWA. Now. The PWA, has, the same features, that it has inside. Of the. The. Browser thank, you. But. We also have some additional features, here, a. White. Screen a white screen. Wait. For this to load. Feet. Away stands for progressive web app regressive, web app and we have access to the windows API I, mean try, it one more time and so, what I'm able to do is instead of getting notifications. I can get, rich. Notifications. Okay. Instead. Of just, having pictures. Sent to me I can actually change the desktop image, of my. My, PC, or the lockscreen image, so. That I can have those goat pictures, come right to me. Okay. We. Can see that happen now, one of the other things that I'm able to do here is.

Well. I'm in the app here, I get. A, new. Bonus that's. Available to me inside the PWA, that's not available in the browser and this, bonus actually, goes into mini, player mode all, right so mini player mode can then play a video, on screen. During. That three-hour meeting, so. You can imagine if it's a budget meeting for three hours everyone, in that room is probably feeling a little stressed right so. This is a way for me to bring some of that relaxation, to everyone, and nothing's, more relaxing, than a looping, video of baby goats, eating corn all. Right so that's what we have here now. To show you how easy it is to call those api's. I've. Got a sample, of them here now your, windows, api is in your progressive web app are just, JavaScript. All. Right so what I have here is a couple calls where I'm testing, to see if I'm in the app because, if I'm in the app I get this object called Windows and, then. I call for a view management. Change. Into. Compact, overlay, all. Right so that's the small window that floats above everything else and then, whenever I'm done I click on it and I go back and I go back to the application mode, default so. It's just JavaScript, that I layer on top of my app if, I have access to it I'll, call it if I don't I just play a video in the screen of the app so I'm able to combine the intelligence. Of the, cloud with, the power of the client, which. Really makes. Me a Microsoft. Developer, using both ends of it now, you may be asking this might work for goat insights but why do I as a developer want to use the graph well, if you're using if, you're building a native app or you're building a web app you. Get a lot, of benefits, that richer. More powerful, applications, have from the graph so first of all you get access to the, users calendars. And their email data and the things that we know about them is Microsoft, but. You know you also get access to their files their file storage and the ability to write your data's content, into. The graph so. You're able to write to the time line and things of that sort that gives. You richness, gives, you pick up where you left off gives, you authentication. For free that, you don't get and. Unless. You want to put all of those pieces together, yourself, so, it makes your apps more powerful, I think, that's amazing but I have a boolean question, okay, what if I don't like goats what if I'm deadly afraid of goats like what, do we do then well. You know what I think our next, guest may, be able to help with that okay. I guess, let's get our next guest on stage thank. You Jeff you are welcome thank you go to in tanks, let's. Welcome Jaime Rodriguez on stage. Okay. So. I here gonna help me with my goat fear. Okay. I can. Try let. Me just make sure things are connected because I was failing to connect while I was there okay, um we're, just gonna close that window myself oh. Yeah. Third Wi-Fi okay I. Can. Help see I'm helping. All, right so here's how I can try um have you ever heard of empathy, therapy gluten oh well. It's, you know the power of virtual, reality okay. People actually use it to for example your friend spiders. You can create, an application and, put spiders on the floor and get familiar with some fine control doesn't sound fun so I thought for today since I was gonna do virtual reality and mixed reality I will build, a really quick app that, shows you how easy it is to create the mixed reality application, okay, um so let me just go here, don't.

Hold It for you yeah, sorry, he. Can do it for ya okay, you're. Good at holding things yep. Um, so, here I have. An empty opportunity, application support for building mixer makes reality applications, we're doing with unity what. I want to have is just unity with a you know empty obscene and. I pre created a couple prefabs so why don't we for example take, into a scene, and. Just add an environment. And. Then we'll add a Goethe let's. Add Lola. Right here the one says goat in the front I'm, her name is Lola and we'll just drop it into the environment we're. Done where'd you find these PFF this came from the Uni Asda store so these are free assets you can just download them you didn't have to do and I didn't have to any work it. For creating a mixed reality application, the one thing I have to do is set it up to be the camera at zero zero zero because. You're gonna wear this in a minute and, we set these up for seated experience, instead of a room scale because as you can see it worked tight and we're. Very tight on space ok so. Let me just try that now normally, what I would do is just so you know how to build it is here I would go to filed bill. Settings, and, then universal, click. Layer settings make, sure that at the bottom right here in. Extra settings, it says mix reality or virtual reality supported and there's, the K support it is windows mixed reality here you could use oculus, or you know, steam. Or other platforms, if you wanted to with unity but in this case we only have mixed, reality and then I click build and this, takes a couple minutes but I have one already pre-baked so if you're dangerous of time so. Let me close already be working for an hour. Here, um and I'll just start it okay. You. Guys can see hopefully. You can see what we're seeing right here on the screen and, I, just need to make sure these controller, is on you. Want to wear it, not. Sure yet I, haven't. Decided. Come. On you got to do it. Fine. So right here all you got to do is hold in right here the controller okay this is your hand, and. Put this on. Stand. Right here if you want or, wherever. Yep. All, right now press the thumb the, thumb stick just go ahead and press the thumb stick not the tom pad yep. And. All. Right, now you have a goat right there and it's a little off because you're moving a little up but up here you can actually be in control and, you can actually go in there and start for example petting it if. You do the grape control, you. Can actually pet it a little bit clothing a little closer and pet it and then if you're scared you can just press the d-pad and that will do the horns and you know about the universal, horns signs, we they praise. The deeper, yep. And. See, anytime, if you're afraid of goats or bulls or anything that has horns anytime. You do the horns in front of them in real life they're just gonna fall asleep that. Goes on any goat so you can go to the zoo tomorrow and, you, know walk in front of all the bulls and do that in front of them. So. That's it that's how easy it is to create a virtual reality, application, you saw dragging, and dropping a few things exporting, and compiling the visual studio and. That's it you're done. That's. Pretty easy for empathy therapy, but there's a lot more power that you could do so. For example imagine that, you. Were a train, operator and you need to retrained so. We could use that exact, same thing virtual, reality for you to learn how to drive the train in. This case crea Jeff wants to draw the crane did you say do, you said crane operator right yep so, it's always been my dream to be a crane operator if, software, development, doesn't work that's totally, my backup place all, right so so. Why don't we do the same this one we're not gonna build from scratch I can. You can hold that one or yeah go, ahead and wear that one. And. You look like a crazy for, this one you're gonna need both controllers. So. Go ahead and press the windows, button so. That you can exit that application. And. Then to your right there's, one application. That our, crane operator. We are take, the Left controller let me turn it on for you. Now.

Normally, What we would do for this is he's going to be one of the students and what, I would let's try it real quick but then, I would be the instructor and I have 30 students in the room so what I do is I wear the hololens and with the hololens i can, see exactly what he can see what he's doing as you can see he's moving around look, forward and move the crane so. Let me go ahead and run this and you're gonna see here through these PC hopefully I can connect because I was having was part, of this yeah. You could be they struck out, but. So. Here's our beacon let me just make sure we can connect Nicola for you because. Here's the instructor I can see him and how he's moving all the crane and everything else so I'm and then I can control the variables for him to actually. Okay. It's. Trying to connect but it's because we plug two networks he's having issues so um so. Let's imagine that I was an instructor and you know I was running these tests the first thing I would do is I'm from this PC or from my whole lens I would just stop and start a test you, can do the steps that I would tell you just by pressing your left, hump that's our backup for no network so. In this case I just started test show them underneath that there's a crate, Jeff. Looked down and I think it was a crate and so, his test big it's been you know to move this crate from the red circle, on to the green circle and. For me right here I can see everything that he's moving and I can add variables, so. For example if I wanted to add fog onto his scene, go. Ahead and add fog that's right now he cannot see very well, so. You can see fine I'm okay, go ahead and turn it off, right. Now, if you can see fine there's other factors that we could do so for example we could throw in some rain and. Today's forecast it's a little bit different than usual. Praise the left thumb. Because, it's you know Cloudy with a Chance of Meatballs so. We're not just doing rain that's easy for him to work with the wiper for. Him we're actually giving him a little, bit of clam food, now, you're see he's loosing here because he's doing the Pollard's and he's like all the sticks are coming down which was a point of him not, that he needed to boy doing a test and. The last one we can do just for fun is we've, added some birds Jeff and so why don't you spread the tungsten one more time, right. Yeah. These birds are little hungry so be careful. Yep. So that's, how you know if you were a train operator you could use this type of application to actually train, for real moving the crates and practicing the right and the right things without, having all the birds and everything else attack you um, so I'm gonna hit and exit there Jeff. Because. The more interesting thing is the code for this right. Join. The club. So. All right I can hold this you so. The main thing with that they're more the reason I wanted to show you that demo is because that we've actually released, in some of the samples for four lands for print for Windows mix reality we created many different sample, apps or. Reference applications, and that is actually what we call our CROs devise their application, and that one right here has.

Unity Networking, so you can go back here into the scripts and see there's. Unity networking there's um there's a whole a toolkit you don't got you didn't get to see today all the total toolkit UI because I couldn't project the whole lens but. There's whole lens and you're connecting, to the two applications, and you can see how here, which is one scene we've. Written the application, the runs in the virtual reality like you saw in the, desktop right here if this was connected you would see what she's moving around without seeing the city you would just see the relevant parts and, then obviously in the hololens so for anybody who has a devices, or who, doesn't have them but can just run it on the emulator you, can go download the source code and actually, just go play with it on your own and learn how to write this kind of application, so. This is one application that runs, vote, on that PC on that application. Is mixed reality device exactly, all right when you do it be application, runs across all of them here's. The URL for it so github.com we, name our samples, crane operator, there's, one sample there today but in the next month for the build tour for. The windows. Live inside a tour we're gonna put all our other samples, out there in the next couple of weeks because that's we're gonna use for the tour so, for those of you that are coming to the tour you should come and do our labs and. Check out the source code ahead of time so you can actually be familiar with once you get to the session, so. That's what is, so much for us thank you good time to show us this we, appreciate it, okay. And we'll have all the URLs, at the end as well right so I think I'm getting a bit tired so I think it's time to get some music going here, so, when we go ahead and invite our musical, guest on stage. There. You go so, we're gonna have Pete, Brown which comes all the way from the east coast to, talk to us about MIDI and some awesome music that he's making hey Pete. Pete. Music. You. Think that's. An awesome photo of my unfinished, basement office thank you for using that one all, right I'm gonna pop this out. So. While pizza setting up I'm gonna talk I'm, gonna explain a bit what he's about to do so, let's go ahead and switch over to my, machine here. You. Might switch into one there, we go man. Your go-to still on here it's precious. Okay. So, Pete. Is going to use two things he's gonna use the, new eye tracking, capability. I guess capability, that's in the new April, 2018, update and also, he's gonna use MIDI to, talk to you all this magic, that's about to happen here so let me explain a bit about I, guess and how that works as part of the new April 2018, update there's, now ability, for developers, to take advantage of I gaze hardware, such as a Toby Hardware that, you can actually get X&Y, points where the user is actually looking so you can make games or you can make accessible, applications, for everybody and as, part of this we also are, going.

To Ship a. Library. As part of the Windows commute toolkit they'll they'll enable you to add, this support, into your zamel applications, really easily so I'm, gonna launch go ahead the windows commute token sample app this is a preview of what's gonna come out later this month we, have this new interaction library and so it's gonna allow you to simply, just use your eyes to, control the whole application so for example if, you look at the sample for this you, can have just, an attached property, or just one line of code to enable just, a button to be clickable with your eyes so the user could be just using their eye tracking look, over to the button, looking, for 400 milliseconds, and the button will click we, can set how many times you wanted to click or how long they need to be looking at it so we're adding this capability so you don't have to worry about the underlying API you, can just get the support to your applications, so, Pete's gonna use this to, build an example. Of how to play music while, he's doing his a DJ, stuff at the club so there's tons of accessibility, in, business, reasons. To implement, this but instead Pete's going to do. Music. Okay, cool, thank you can we switch over to this one. So. How many of you have been over to the, Tobii eye gaze. Booth. That's, that. Way I think right cool. So Dwayne, from MSR. Over there put together this really cool application, that lets you play the piano with your eyes and. So what I've got there is I've got that application, here and then I brought a bunch of MIDI synthesizers. Up here that I'm going to be controlling. With this if you're not familiar with MIDI it's a standard that's been around since about 1981. It's. Still alive today and it's something that back, in the first release of Windows 10 we added. A new API in for, uwp and desktop applications to, support MIDI, and then about a year ago we added in support for Bluetooth MIDI, which is a new, variation, on that standard so let, me first try this out and see if I can somewhat. Play the piano with my eyes here so do, you have audio, on this. Yep. We do good there. We go. So, I'm staring. At different dots. Can I get applause for that no okay all. Comes. Now, I'm. Terrible, at playing piano with my hands I'm even worse at playing piano with my eyes so, what I decided to do here is add in a second, page to this application, that gives us a slightly more realistic, scenario rather, than playing the piano what, I want to do is I want to trigger a part in. All the gear that I have here and the reason is that a lot of times when you're up and you're doing some sort of performance your hands are busy and you're out of ways, to trigger different things as in, that performance so, let me first start off here by showing you what this button does when, I press this button first, of all let me start this when I press this button it's, going to increase, the volume immediately. Of a second synthesizer, I have up here so for example. Right. So, if I if, I turn that down again you can see. Okay. And if there are a one-man band and your hands are busy with all instruments, you can use your eyes to also you use other instruments, right right so that I just want you to hear what that does can, you hold this for me. So. Let's let's have a slightly better performance here, I'm. Gonna start this. Now. The speaker's aren't half-bad right, so. It's really good right you're getting into it you want to do a bit more than performance, right, you're. Getting ready for a transition. And. Then you can easily bring in that additional, part. Okay. That's amazing a little good and you can easily bring in that additional part just by using eye gaze on this stuff here so cool. Sample application, I had a lot of fun with it thanks again to Duane who I think I saw in the back of the room here somewhere for, putting together the initial application if, you guys want to play with more I guess stuff they have a whole booth way over there you can sit down you can mess around with all the sensors, they have there the, toolkits gonna come out and if this one's gonna have all the support in there you can start putting in your applications, I want to thank all the speakers and everybody all the guests that came on stage on around applause for everybody.

Thank. You all your wonderful people coming to our session we, had a lot of fun I hope you had a lot of fun we, have stickers here, you're. Gonna let Pete, play us out but I hope you enjoy the rest of bill all. Right and for those links there's a. Akms, slash. If deaf live. So. Visit that thank you everyone.

2018-05-10 15:01

Other news