Hello Show | Making the metaverse a reality — enabling technologies, tools, and platforms

thank you Jeff hello everyone uh welcome to this panel on making the metaverse a reality uh first a quick introduction video wearing a virtual reality headset the visitor is going to experience an immersive visit of the cathedral Notre Dame de Parish [Music] until now to take full advantage of this type of experience visitors would have to bring a laptop in a backpack connect it to the headset to process the large quantity of data necessary [Applause] orange offers today the possibility of experiencing immersive visit without needing a laptop how by leveraging its 5G Network capability and expertise in cloud and extended reality the graphic comments are passed directly to the cloud via servers installed on the network instead of being processed on the device's graphical cuts and it worked the headset streams content from the cloud through a very high output and low latency 5G Network providing a more comfortable experience for the user and a more productive technology platform for creators and for venue producers [Music] the orange 5G powerful Network combined with Cloud XR opens up new possibilities for businesses and consumers in a variety of areas foreign a great example of the power of crowd makes reality a lively 5G and for a more in-depth discussion of the tools and enablers of the meters please welcome our panelist first crazy Nikola who's the co-founder and CEO of Cortex server hi crazy hey everyone happy to be here foreign business development for XR and product manager of XR at Nvidia hey Greg nice to be here so um thanks to both of you for joining the discussion um so to maybe set the stage why don't you uh talk a little bit about your company and or products you manage starting with maybe Greg yeah sure so I'm with XR at Nvidia and the way to think of XR Nvidia is where a horizontal supporting many verticals so you think of manufacturing vertical um m e vertical the architecture and construction vertical and our XR team supports all those with a variety of of sdks and of course our gpus and such and that's uh in that group on the product management and business development person yeah so I'm a co-founder and CEO of cork XR this is a company which specializes in Cloud streaming technology uh we have a few products already in the XR field about Cal streaming some of them built using Nvidia cloud XR so very excited to be on this panel and talk about this type of technology in use cases these days our focus is on delivering end-to-end Enterprise Solutions uh for XR and pretty much creating an ecosystem of applications for Enterprise XR awesome so my first question is that one thing that we saw in the introduction video is about the power of mixed reality in the clouds um why do you think this is a a good path for the future adoption of AR and VR bringing it to the crowd Crosby go ahead now I'll follow you if you wanna awesome awesome sounds great well listen I think um when we talk about the metaverse the first example that we see is gaming and there have been already amazing XR gaming titles uh but what we believe at quarkxr is that the the real transformation is going to take place uh at how people work and I think XR is currently transforming a lot of Industries uh in areas like manufacturing Automotive architecture a lot of those Technologies where Nvidia actually has many expertise but I think uh when we talk about those work related use cases and when we talk about that the current XR devices that we have like you showed in the video uh their mobile devices and I think it would be very unpractical for a lot of 3D it engineering professionals to have to do their job in on on mobile platforms so it practically would be impossible so I think what the cloud offers here is the capability to have very complex Enterprise applications which are running in the cloud thanks to gpus and then on the client side you have the usability and the portability of those Standalone headsets which are going to continue to get Slimmer and Slimmer so these are the two ways we are going to walk towards more more massive adoption of XR yeah and I have to agree with Crosby you know so so we've got mobile devices that have certainly caused a let's say an explosion on the VR scene so far we're still waiting for the AR scene to take off but we have tens of millions of users now in Mobile VR headsets and and that compute platform is is a really nice mobile compute platform it'll get Slimmer and Slimmer and we'll probably see compute you know stay about the same maybe increase the shade but you can't fit you know even that that Notre Dame model that you showed which is one way of thinking you know the the Notre Dame is a is a new place in the metaverse that you can go to experience something new it's in the cloud you couldn't fit that data set on a mobile headset and so you certainly couldn't do the graphics rendering the high quality Graphics rendering on a mobile headset GPU so getting to the cloud for graphics is is and data set size is absolutely necessary for the new workloads we see for the new places in metaverse while whether they're a location like like the Notre Dame Cathedral or they're a digital twin in a factory those data sets are going to be quite large they're going to be one and wanting photo realism and then also what the cloud offers is once I have access to that compute layer I can start thinking about doing artificial intelligence assistance in my environment so whether that's analyzing video from my AR device and and having that AI assistant look at the video and and give me some context back down to my VR panel or whether it's voice activation where I don't need to use a GUI or controllers in a scene where I'm examining a car and I can just say change color change position should change camera change lighting and the VR system sees that that's an AI assistant where I'll probably need another GPU in the cloud Doing Network so getting to the cloud is is really mandatory for really enabling the metaverse use cases coming up right so uh thanks yeah that's good you covered use cases outside of purely gaming and entertainment uh that is very interesting and I know uh that is obviously uh the where the market is is headed and uh and you both work with industry uh customers and companies uh could you highlight one use that is a kind of off the beaten path that you've seen recently of a mixed reality maybe you regret first yeah I'm off the beaten path so so training is I'll say the beaten path right so training in VR in in AR is is huge that's we've seen a lot of academic studies now that training is is really accentuated with it with virtual reality with metaverse type of things so we'll see those that's beaten path though you know if you think of architecture and uh and automotive design where you get the beautiful digital twin that's kind of beaten path I mean a lot of people are using VR actively we're using streaming VR with a variety of proof of Concepts and and actual uh people working and streaming VR to do those design use cases um off the beaten path and and not too far off I think medical is going to be a really a big deal we see a lot of training in medical now but um we see multiple companies trying to figure out how to bring in digital twins to surgical Suites as the surgeons are doing work and how to how to visualize those how to register them so off the beaten path just slightly just because it's not common day is I'd say medical is going to be a really interesting use case coming up if I can add a couple of these sentences yeah so I really agree I think digital twins are really key because in many different Industries as Greg pointed out they enable so many different use cases like training so when you have the digital twin of either a building a product uh or a medical device in in XR then you can do very sophisticated simulations training planning but also one use case that we've come across a lot at quarkxr is people were using those Technologies now for remote work and remote collaboration and actually a big struggle when doing remote collaboration on mobile XR devices is okay I want to work on let's say an automotive project in XR I'm unable to open those large Cod models or those large Bim models or what have you so streaming is a very good use case where you render this sophisticated models in the cloud and you stream the output to to XR devices and then the other aspect is a lot of those platforms right now that people use for remote Corporation because of the same rendering limitations they can't fit reliably more than 15-20 people into the same environment while having sophisticated realistic avatars at the same time so this is a very good example of how to use the cloud to basically put more people in the same environment and this is I think something we're going to touch upon when we talk about the metaverse in general and Virtual Worlds and I I think I'll mention one off the beaten path thing I was thinking through as I listened to Crosby is you know so it kind of intersects uh media and entertainment and medical and I've been hearing from a group of software vendors thinking about mental health applications so you can imagine going to a metaverse world that's really beautiful and relaxing and a place to detox while you know our society is getting busier and busier our work is is kind of invading our home life so I need a break I need a mental health uh just half an hour or 15 minutes in in a metaverse that has AI assistance that kind of can hear my voice and the stress in it and suggest scenes or or meditations or music or what have you and and so I think a lot of creativity is being focused on that area and that's a little bit off the beaten path right we think of Industry we think of digital twins with car models and buildings and factories and but I think the mental health group is also looking at this and gonna going to use these techniques to a really large benefit for many people that's a very interesting idea and actually it's good that you mentioned a lot of content related to this mental health idea and my next question is about when we saw the shift from web to mobile we saw an immediate wave of creators and service providers building for mobile um what are the barriers right now to this shift happening for AR and VR content of everybody shifting to create for mixed reality uh at once or in a way what current barriers are there right now to this yeah it's interesting so we've shown you know over the life of cloud XR which we introduced in in 2019 at Mobile World Congress La we've shown it you know streaming VR was possible we didn't know for sure that it was with the motion to Photon latency demands but late latch and and 5G networks and such will will allow that to happen now and then so the the piece we're seeing that is the most difficult part is scaling so orchestration so I put on my headset and I want a lot launch a remote instance you know how do I get to that machine how do I provision that machine how do I get my content on the right machine closest to me in the cloud how do I get on the 5G Mac um what services what uh what slas are we going to give people to heighten that experience in a really fine tune that 5G Network for for XR so those are the scaling scaling and optimization are the challenges now we can do it we've proven that we have a variety of isvs software vendors that that are working on scaling like Crosby's company and I think scaling and optimization are now the path and once those are done people can just put their headset on click on an experience in their headset provision a cloud instance whether you know it needs to be super low latency on the mech or longer latency in Regional data centers will be a decision by the the software talking to the system and this experience will launch and off you go so I think that orchestration and scaling are the challenges right now and that's a lot what we hear from from businesses um basically the conversation is never about what can we use XR for I think every industry has found certain use cases that we mentioned like in manufacturing and Automotive in architecture Etc as Greg pointed out it's all about scale I remember back in 2019 when we were talking to Deutsche Bahn and some other super large organizations the question was like how can we use this on an organizational level so how can I put all of my employees into XR so this is the scale that that we need to think about and also when we talk about complexity I think it's good to think about the the cloud as a development platform right now if you need to develop for Mobile versus desktop XR mobile is obviously more challenging you have to put a lot more optimization into it to get beautiful content you know to to make it look as if you have a desktop GPU you need to do a lot of trickery in your in your XR content so this is one Challenge and then the other challenge is there are multiple platforms right now out there uh probably we're going to talk about them but you need to release a specific version of your app for each and a different platform if you're using a cloud-based solution in contrast you have a easier path to development and you have what basically one version that is supporting all those headsets so scaling is the name of the game I understand um and there's a couple of things that you guys touched up on that are interesting is uh first on the device side so you mentioned an explosion of devices um and fragmentation of all the different platforms that they run in terms of Hardware uh where do you see that the current hardware specs of all those devices uh right now and and what do you see as the path forward for a better seamless experience in terms of of the headsets available right now I think the openxr standards are a nice a nice start right and that and that will you know one of our biggest challenges with Cloud XR is is writing uh client libraries for multiple devices and and engaging the the device manufacturers to build those client libraries and we see you know different strategies some want to have a very exclusive platform and some want to have a more open and general platform so as openxr comes out also on the on the server side as we start seeing applications written in Linux and and openxr drivers sitting next to those applications and you can start thinking of XR environments being kubernetes managed scalable type of environments um not needing necessarily a Windows system and and that and then the common you know Unity plugins in openxr for clients those will start May making it where we can build general systems where the the device creators can just take those libraries wrap them into their their device um the isvs can just launch their software in a kubernetes container and off we go so I think I think we have a lot of optimization and orchestration coming over the next year to two years in that space for and open xr's really kind of introducing that platform or that capability yeah we're also banking on openxr very much to first of all make our lives easier at cortex R but also we see it as a pad to eventually supporting all of this through webcoins right so uh today if we look at a lot of the Enterprise world and a lot of the internet is in general where is the web-based uh whether you access it through through a mobile client on on or on your desktop a lot of the applications that we use today and the content is web-based and I think we need to get to that point in XR where I'm able to click on a link probably walking with some credentials and and then a server is provisioned for me and I have access to a certain Content Library be that for work or for uh you know entertainment but on the devices side uh we uh we want the devices to get Slimmer and slimmer uh as as the current iteration of xor consumer devices have shown us we can significantly shrink the size through through new lens Technologies like pancake laser lenses and whatnot obviously miniaturization puts stress on how much compute we can fit into those devices so obviously that's another great use case for for for this the cloud setup that we're discussing today eventually and we are quite far from this future but eventually we want XR devices to something similar to my glasses so I think offloading as much competition as we can from those devices is the key right uh so um and one last last uh question uh regarding internal probability so uh for uh the headsets we've seen open in XR is a great standard um and now for the the create creative side uh there's many creation platforms uh I know Nvidia you have Omniverse as a product that that tries to to breach the gap between some of them um but those many 3D creation platforms are currently still quite not interoperable yet uh is there more work being done on this side of the standardization yeah I think you know if you if you look into our Omniverse we think for 3D environments the USD uh format will be the HTML for for 3D metaverse environments right so so we're we're working with a number of groups on USD we're building connectors with other groups so um if you think of the USD scene graph Omniverse nucleus as a place to collaborate and build these new worlds be it a digital twin a car model or uh or a great escape place for mental health um that USD underpinning is our attempt to start building a platform that many many people can use and and really natively plug their development software in and and create these scenes and have other people come in you know create one and Revit and have another person come into Maya and alter it create an Unreal Engine and have somebody in substance come in and do material rules for you so that collaboration platform I think is key to you know building a digital twin or building a metaverse world and then being able to update that and keep it fresh and keep it new so so I think that collaboration platform is key we know VR that collaboration is a superpower of VR right for automotive designers we see people working in Korea and Frankfurt and Irvine California at the same time around the same car model that collaboration is is a key component to where we're seeing VR grow and uh in the metaverse collaboration the creation of metaverses the collaborative creation is going to be I think it's super power not just for VR but for the collaboration to build these these sites and such so USD is is our kind of um our platform we think it's going to be the the significant underpinning for the metaverse and and that's why we're working with Pixar and groups like that to to Really build it out to service that industry yeah it's a really awesome yeah really really awesome that Nvidia is so open uh we uh that's one of the things we love about Nvidia as a partner is the openness right now um some big organizations when they talk about the metaverse somebody has a particular idea of what there's it's going to look like for their platform or for their organization and instead what we want to see and I think a lot of the developers in the exor space would agree with me it should be like the internet the internet today is very open I pretty much can create an HTML page uh put it on the internet on some server and you can access it through whatever device that you that you have because we have those common Open Standards and I definitely want to see the XR industry going into that direction and not load the gardens and what not so this is something that's very important for us all right yeah thank you both um so we have a bit of time maybe we can take a question from the audience uh let me check all right so we have an interesting one about um the types of security and identity management uh tools that are present in your products can you talk a little bit on those two two points security and identity for the meters and mixed reality yeah uh well this is actually a yeah I'll I'll take it first if you don't mind um for us this is this is on the top of our list for um for for our roadmap for next year and uh obviously uh as as Greg pointed out at the top this type of approach streaming from extra content from the cloud in real time it was not clear if this is going to work with motion to photo latency and whatnot and the the first question that companies in the space like Nvidia why cork XR had to answer is is this possible how do we make it possible and now that it is possible we're going to see a lot of those very important crucial features to to business applications get added and on our side we are working with several organizations that they have a user management Enterprise user management software to make sure that we are integrated with them and you are able to provision users and manage users and have identity in the same way you have on mobile so it's still very early on but we we definitely have identified this as as a problem that needs to be solved and we want to have a solution that works both for you know I'm just a consumer I want to check out some some some cloud gaming in VR or I'm a huge Enterprise and I need to onboard hundreds of thousands of employees foreign XR we didn't want to lock anybody into a certain security schema so we really rely on on cloud XR being a great streamer it supports encryption but when it comes to certificates and keys and and passing those back and forth you know groups that are using Cloud XR to integrate the streaming into their software like VMware uh we're relying on them to build the security systems and I think security systems will move quickly enough just like they have in virtual desktop and things that that's really how you want to do it is is do what you do best and we do streaming really well we do encoding really well Graphics really well and let the security people that are working with groups put their security on top of that streamer so that's the way we're looking at Securities we think there's a robust industry and and a lot of people use our tools to build security systems but we think the we'll keep focused on the streaming and the graphics and and let people make it just really easy to put security systems on not the clavics are great well uh and I think that's that's it for our time today but uh thank you very much uh Greg and crassy a very interesting conversation thank you yeah Crosby good seeing you here this was awesome likewise likewise I appreciate it

2022-12-03 19:21

Other news