Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on Demand Modelling
Hi. My. Name is Patrick, and together insomnia, we work at Burge type 1 we, will present today how did we manage to scale our simulations. And analytics, for the purpose of generating Hyperloop design so. Let's jump into the agenda of a representation today, we will first tell you what are, what, are we doing at virgin heart 1 and what is our subgroup, machine intelligence and analytics doing. We, will then show you an, example of our simulation and, what are the design challenges, of building a data pipeline, for running hundreds of these simulation, in the cloud we're. Gonna show you which where the tools we use short, demos of these tools followed by a demo of sample, simulation, run with our pipeline then. Will be quickly, jump into, one of the conclusions that we discovered, using the pipeline that is related to demand modeling. So. We first, have the one or a start up we were on 250, employees in downtown Los Angeles and we are building new transportation, system that is based on a vacuum tube and small passenger, or cargo vehicles. That, we call pods it. Offers you a direct emissions because we are using electromagnetic, levitation and propulsion and, because of that we can provide very short travel times but, not only because of that we, are also own demand so imagine, ordering your Hyperloop, ride using the web or phone app we've, run hundreds, of tests at scale and had a 500. Meter test track in about the desert and, we are constantly planning, and building new tests. And enhancement so please follow us to. See how Hyperloop, is becoming a reality and learn, about the future of transportation we are building for you. So. With this new transportation, system people are asking about questions. About safety cost, what, is the real travel time and many, many other and our, group, machine. Intelligence and analytics is, doing operation. Operation. Research or Hyperloop. We are providing, analytics products for our engineers, and business teams so, that they can show governments partners, and investors that Hyperloop. Is the solution they are looking for for, example we can provide a loop with high-speed rail airplanes, and other modes of transport. We. Are answering, all these questions using, simulation. And internal, and external data. For. Example we could answer the question, what is the optimal vehicle capacity for a given geography, and how, many passengers can be realistically. Handled, in a given scenario and. Again we answer all these questions using. Data. So. Our. Sample data flow would be to gather the man were to on a travel for a given geography, then, fits route alignment, in the land using, our 3d alignment, optimizer, and your spatial data then, we schedule, the trips for the passengers, using, the demand and Hyperloop alignments, and run, the, simulation after. The simulation we, then compute performance and, cost metrics and usually we discover something could, improve, for, example maybe we can sleep some more parameters so, over, time we run more and more simulation, from simulations. For more and more parameter, sleeps and after. A few weeks of running we have a bunch of answers to the questions asked, so, I mentioned, running, the simulation we. Call our simulation, software syn loop and it's an agent-based high, fidelity transportation. System simulation that we develop in-house it, can simulate the Hyperloop alignment, pod which. Is the vehicle the physical. Behavior of the pod control, systems interacting, with the outside world as well as passengers, and Hyperloop, stations that people portals, so, let's take, a short tour in our simulation, software afterwards Tanya will tell you more about the pipeline. Hi. I'm. Sandia so. We just now saw the Hyperloop, simulation, video now. I walk, you through the flow of the Hyperloop simulation, so, we first start, by capturing the demand and ridership patterns, of the existing, route, in consideration, and then, we, do, ML models, to predict the future demand, and with. That demand data we send it into a trip scheduler, which will then do the power allocation, in departure schedule for this we, also have a track optimizer, which, gets. The possible. Optimized alignment, for the route in consideration based on the geography, cost we, have additional, suite parameters, as well like say for example. Allowing. Weather. Ridership is coming whether. We can write in parallel, or a, ride-sharing is allowed and those, are all are fed into a simulation, software, which, then means the simulation, for us which we just saw those, simulation, results, are then analyzed, to give us some metrics transferring, assistance.
So, Imagine doing this again, and again for, different set of parameters different. Set of alignments, and files and we. Have to make sure that our analytics, platform, is able to handle this volume, and is focused enough to, run this again and again. So. While we need to be flexible and why do we need to be fast so. We, need to have accurate, answers and we need to have them right now as. Soon as possible, so, we need to have it flexible, and fast and we. Have data volume changes. Occurring. Based on different alignments, and different use cases and. Also. Here when we. Are getting data from multiple engineering. Teams and different file formats, and different, processes. We have multiple, sources, and also, see we have to add new applications to, our systems a pricing model prediction. It has to be seamless, now and being. One of the, kind technologies, and we have to keep up with the latest of all the technology, and tools we, may need to migrate between two suits and platforms, so, we, for all these reasons we need our platform, to be flexible and fast. So. If. This is not something new to set up such a robust, and dynamic platform. So. A lot of big companies like, Google and already. Shown us the way, precisely. They, like. You like adopt the cloud technologies. They. Ensure there is micro, service architecture for flexibility, they, use distributed, computing, to be fast enough. They are also auto scalable, to meet the large demands, and they, use the lambda of architecture, - -. In. Order to implement, those architectural. Guidelines we, have a little lot of challenges, being a start-up so the first challenge is that we have this those constraints, specifically. With development, maintenance we. Also have, limited, DevOps support, in order to support such a platform and in. Addition, being one of a kind technology, we need to have security implementations. In place and. Lastly. We need to have traceability. And governance, in our platform, so that the quality of our, product, is ensured, because we are directly leading the passengers. So. This is our analytics, platform, next step we. Have a couple of tools primarily. Open-source tools which help us with our text. N95. Is our data flow manager, which helps us will much in place the effects of the PMI, picture, mark a is a data, format, for columnar, it's using. It for huge, demand, and huge growing the amount of data sparked. Is at compute engine which is open source but we are using the raid objects platform so as to easels with the DevOps and the, other limitations. And, and we, use ml flow for our analysis. So. In all, these, in addition to all these we also use, other like. Cloud-based. Technologies. Like docker Amazon's. Ec2 and s3 and, Kuban accessible, these, all make a very flexible. And nice stable platform. So. This, is our overall architecture, of our analytics platform, so. As you can see on the left hand side we have a couple of funds which come as input, specifically. Say for the velocity, profiles or the portal models and vehicle, models etc the. Information. And the relationship, between these files are persisted, on our database, and we also, have an additional configuration. Application. Which. Our users, use for. Primarily picking and choosing the, different files of simulation, - one once, they hit the run the, 95 takes control it does some transformation. Cleanses the files and runs the simulations, on our, ec2, engines, and paddle and, once. The simulations, are completed, the control from specs from 95 and is then given to spout for. An analysis, job once. Analysis, jobs are complete the, results are stored as experiments, on ml flow and we, use emag flow to analyze the different batch running, results, and, compare them and see what questions. We have answered, and how, much of it is like, like, givens gives us the information and we, have a report, so. This is all overall flow, so. Now my colleague, Patrick he, will explain about how we use spark, and data breaks in our platform so, let. Me tell you our Hyperloop. Data story about, a year ago our data scientist started to grow from megabytes, gigabytes and, our processing, time started, to go from minutes to hours and, our, Python scripts especially panda scripts we're running out of memory and so we decided we need more enterprise, and scalable. Approach to, handing our data we, try different solutions, and obviously found that spark and his family at the facto standard solution, and it's great with so many connectors, and tools, around.
So. We, decided we are going to have a bunch of spark workers, and the drivers and Hadoop file system and, a stream mounted, in that file system this, is already a big infrastructure. So we already knew, we don't have enough dev ops to manage, that and this is why we are using data bricks, but. Then. We also realized that our previous code was coated in pandas and to use it with spark we would have to use either PI for occurs power and we, and particularly, me since I would have to do it weren't particularly happy about. That since we, would have to rewrite all thousands, of lines of code in. Pandas. And. We rent experts, in high spark or Scala. And this. Is where a koalas package comes in exactly, when we needed it more. Or less money one, year ago around, in our April data, breaks open sourced koalas, package that is doing exactly what we needed so. It's. Translating. Our pandas API into PI spark so we can scale our pandas, code very easily. With. With the familiar pandas, idea. So. Now let's talk about another part, of our system which is called ml flow we, use it to log track, and analyze, our. Simulation. Runs and results. For. Those of you who don't know ml. Flow is an open-source tool, that allows you to log machine learning models, its parameters, and metrics, so for example here using this UI if you run a bunch of models and log them you, can see the parameters and, metrics, for. All of them and then select, the best models then, you can deploy it to production using. Ml flow projects and models on the, top right, and on the slide, but. In this presentation we are particularly, interested, in a node for tracking so. Let's. Talk about it more. We. Found a demo called tracking service, purpose very well also, beyond, machinery, our, simulation, runs are also expecting, a lot of input parameters that we can sweep and also, have a lot of outputs, that we score, to create numerical, metrics instead. Of developing our own solution, to track query, and analyze, simulation. Runs we use ml tracking, is a generic. Experiment. Logging and visualization, tool we, just treat every simulation as, an experiment, and log it as such and we. Found it super convenient and very cost effective, we actually, saved so much time but not having to develop a simulation tracking, tool ourselves. And, it integrates very well of external tools for api's, for example there is an API to query and export experiments, to find us data frame that we use a lot and. We. Found out ml flow serves this purpose very well for tracking our simulation, and our. Simulation. Actually, contain a lot of AI hydrants, so instead. Of ml flow we actually call it AI flow. Now. I will let my colleague, Sandhya explain, nightfly. Our data integration, and pipelining, tool. So. Apache. 9 v r9. File for. The, data flow management and. For any trader driven analytics, you need to have a very good data flow so, it is data agnostic. On I Phi also supports, transformations. And data routing, has 250-plus in both continents it also has a mini Phi and an eye-fi registry. Extensible, weather, and. So. For example now say if we wanted, J merge, to JSON files which, have the same file pattern, and. Perform, transformations and, put them in s3 so, first you go and choose the gate file component, you give the parameters of the file and the pattern you want to create and then, you can. Use the merge file component, much constant component, to give the header and the footer information.
And How. You want much the, two files you, can also in the connector between the two components, you can see that you can do prioritizations. And buffering, mechanisms. And. In addition, for each confident, as you can see we. Can also have scheduling, mechanisms. Like you can put the schedule in them and you. Can use the Joel transformation, component for putting the transformations. And. Add s3, attributes, and also. For use the profile component, along with the AWS, credentials, as. Controller services, now, if I want to run I can pick and choose and run which over confident I want to so I am just run, from the first component here and I can see that the files which operate that queued in the in. The flow and, I will be able to see the different flow, content, and the. Flow attributes, which are related for that particular, file. Once. I confirm I can choose to run the remaining, part of the flow as well and, after. The flow has completed, I will be able to see. The provenance, of the particular, whether, the file has been put in SV or not and whether, it was successful and what was the content of the file which which is good so, this is the complete flow. So. Apache, 95, is also, good at real-time and in control say, for example if you want to set up the concurrency, control so, for a particular component you want to set up whether the task needs to be confident and how fun for it it needs to be and the execution. Needs. To be running on all nodes etc and, the schedule needs to be, arranged. As well you can also set. Up whether the queue prioritization, is like whether it has to be first in first out and. Buffer and etc in, addition, you can also add control, services for external, connectors, like, say Postgres or AWS and. On top of all this knife I also gives the, power to set up the. Data security, using, SSO, or SSL, visiting. The. Most, important, feature of my fights is power of provenance it gives in volt problems, and we need provenance. And lineage for efficient, traceability, in our analytics, platform, so in this screenshot. As you can see it's. A screenshot of a put file components, provenance, even and these. Are the details of that problem ceiling and the right-hand side you can see a tree like icon, when, you click it you will be able to see the complete a lineage of that particular province event, so. Each and every circle, here is a components. Problems, data. And once. You click any, one of the circles, you would be able to see the detailed, description of what, was that event what was the flow file its byte size and what, with attributes, of that particular flow file and in, addition you'll also be able to see the content, like, whether like. It's the input of output, of that flow, file or connection, and you'll also be able to replay that problems, even if you need to check if anything is going wrong this helps with good traceability, force.
We. Just saw all the different components, that are used as, a part of our pipeline so, we would like you would like to show you a small demo for. Company pipeline. And. We, would first start. By showing you the configurator, application. Which in user uses to pick and choose the files then, we will show how the control, is transferred, to night file and then, we will also show how it is simulation. Of executed, and pabulum and how we use ml flow to analyze the results. So. First we'll start off with the, configuration application. Whether user pick and chooses the files as you. Can see this is our configurator. Window you can use the. Add button to, add a particular batch and pick at the alignment for that particular package and. Based, on the, executables. Available, you can select each of executables. Which. Only you wanna run and then. When, you hit V dependency you'll be able to see the dependency, between the, executables. And what are the files needed for it when. You accept, that you will be able to pick and choose the different files whichever, was available, for that particular, file. Exits and, as. You can see you will be able to choose the different files you, can also see, in the tabs on the top of the tabs some numbers, coming up those, are the numbers of the different, scenarios which, are going to be executed, as a part of this file choices, which you are making so. If you choose different set of files, the combination, needs more, scenarios, and. As you can see you can also sweep different, parameters, and the more parameters you choose those also contribute, to different combinations, of scenarios, which you wanted so, when you hit run you, will be able to see the total number of scenarios calculated. And when, you hit you again, you will see that the successful batch has been created, so, once this batch has been created, the. Control, then transpose, to 9. So. Now we will see how Wi-Fi picks up the control seamlessly. From the back after the batch is created. And how the. Flow progresses. This is a sample, flow which we have and. Once, the ninth eye control is passed over here you, would be able to see the data flowing through and you can see the provenance, of the data which is coming in you'll be able to monitor and view the content of the data for example this is a new content and you. Will also be able to replay the content, in order to see whether if you have any bugs or anything and as. You can see here in this screen there are multiple, some simulations running, on ec2 all, talk by nightfall so. Once, it is kicked off you can also check, what yielded. To that particular, execution. By. Checking in the details. Of that particular flow file contained the attributes and the content, and also, the best part to see is the provenance, lineage, you, can see that where, the specific command.
Came From and what all are the files which yield it for that particular command to execute and how, the simulation, is executed. So. This is the cloud. Flow basically. So. Once the simulation is, executed. And. Our, analysis. Metrics, job is running on spawn all the, results, are saved on ml flow as experiments, and Patrick, will explain about it so. Thanks, Sam yeah now I'll talk about ml, flow our, metrics. Listing, and. Analytics. Platform now, so, as. You. Can see here, this is a no flow window and every row is a, simulation, run and for every simulation run we have some metrics, and rameters, and, if. We click on the row, we. Can drill down into a single simulation, run we, can again for the simulation, see, parameters. And metrics. For. That simulation. But. The cool part is for. Every simulation we, compute, a bunch of. Artifacts. Like for. This simulation for example I created, that the demand scatter plot and it's, a, very. Interactive. Here, in the window you can explore, you can, can interact, with the interactive. Cloth lip-lock as. Well as I created, here a histogram of the, trip times and so. For. Example here you can explore that. Between different cities that the trip times are different for passengers. But. Then what, if you want to compare, a bunch of experiments, so to compare, a a bunch of simulations we, select all of this, and. Click, compare and again, here we have a big table where, each column is a. Simulation. Run and we see parameters, and metrics but. Here, we also have a way of exploring. That data so if we select the sample parameter, and sample metrics we can see which. Simulation, is the best and we can also see a contour. Plot you. Like, freely plot so. You can, see, two. Parameters, and one metrics, and here we can see one of these two simulations may be perform, to the path so this. Would be our. Further. Exploration. So, I, hope you enjoyed our pipeline, run for our simulation. And now I would like to share one of the conclusions we drew using. This pipeline as I told you before Hyperloop, is on man and because. It's on demand we found that we can prove provide increased, efficiency, and passenger, convenience, but, there is one particular challenge, when that with the own demand, earnest of Hyperloop, that, I would like to discuss and analyze with, you. So, the problem. Is we, would like to predict areas of high or low demand so, that we, can redistribute our vehicles, ahead of time from, the area as low demands to the areas of hajima imagine. Network. Like in the picture on the bottom left if. There's a high demand in the center we would like to redistribute, the vehicles so that they are able to pick up passengers Wow, right when they need to and for, that purpose we feed, the demand prediction, model to our assimilation. And we. Train these models using historical, demand data we gathered for our analysis, and which trained multiple, models using in. Carousel, SDM and GRU and actually we, trained it using, horrible, inspark. And, we also run. Erima and profit, models and and swept the. Input parameters, to these models using mark. UDF as distributed. Sweeps. This is a rough example of, one of our models, and. The blue line you can see, shows. A ground, truth for, the demand for a given that's origin, destination care, and the. Model prediction, is shown by a green line and the prediction, history is strong with the red line and this. Model is trying to predict hourly, demand for, three days in advance, for. A single, origin destination pair. So. As a conclusion from that analysis, the best model, we fed into our simulations. Using. Using. The pipeline was able to improve the number of required vehicles, and cost, by. Even up, to 70% and, and this is a really, great finding, we, also found that sometimes. Simpler. Models. Like ARIMA, or prophets sometimes. Outperform. Carris and additionally. To improve, the models we correlated, the weather data and the surrounding events data from, predict taste you know, with. The transportation, demand and we definitely found a correlation that, provided, a great improvement to our demand. Models as a general. Conclusion I hope you learned a lot, about our Hyperloop project and I. Hope. You can see we're doing really cool stuff and by the way we are still hiring so please check out our careers. Page our. Story, of data breaks is an, amazing partnership but also a very lucky coincidence with, koalas and that's.
A Great company we love using their tools we. Also showed. You the design of our system, that is running analyzing, hundreds. Of data heavy experiments. And how. Did we achieve it with valid with minimal, development. Effort, and. We actually coated this platform with two to three people in six months and here I'd like, to give a shout out to our front, end engineer, Justin. He's. Great, he. Really helped us figure. All this out and it's, actually the integration, between the tools that took the most time. And. The, tools that did. The job for us we're knife wise Park ml. Flow and Parque. If you want to learn more about Hyperloop. Is reinventing, transportation, feel free to reach us on twitter using this, handle, and i believe now it's the time for questions. You.
2020-08-18 22:14
