Meet the engineers: Building line-of-business apps at Microsoft with CI/CD
Hello. Everyone and welcome to today's roundtable, on how we build line of business apps at Microsoft, with continuous, integration and continuous deployment, my. Name is Lyle dodge and I'm a senior program manager with core services engineering and operations, I'm. Joined here by some of our developers, which. Spans some of the apps that are building around path services, service fabric and data warehousing using, sequel paths so, let's take a minute for our team to introduce themselves yeah, I'm, the creme de huaah I'm a senior software engineer within. Finance. Organization. I'm, working in Microsoft, hi. I'm novelty party I'm an engineering manager and, user services, engineering team, hi. I'm Hina I'm a software engineer on Microsoft, devices, supply chain so. Thanks everyone for introducing, themselves before we get started I'd. Like to tell the audience at any time you can ask our experts in, the question, by putting it in the Q&A window I will then read your questions out loud so everybody can hear it and, then we'll have some answers from that so stay tuned after the QA because. It's the smees we'll share some key takeaways so with. That you know I wanted you to start by. We've got a diagram, up here, your. Your application. Team had. A line. Of business BizTalk, cluster that did a lot of the b2b integration, for the devices supply chain so can. You show us you know what what, we have here in this diagram on the computer and what what you guys have done with that yeah, so all our supply, chain solutions, were and on the BTB to be platform, which was the stock server and, we. Have I created. Almost, all of those 2's, here, using. As your pass and. Serverless. In, is your technologies. So. This, is a basic, architecture, that we kind of follow for order management and fulfillment, applications, that we have. We. Have lots. Of partners, in our supply chain and. They. Connect, or, we expose, our endpoints, through API M in Azure, and. In. In. API, M we redirect, the requests, to the web app and, for.
Load Balancing, purpose we are using as your traffic manager. So. API, up here is a web app here also it's just two different regions that it is deployed and then, once. The message is with. The API app it, gets into the storage and. The service per service bus just has the metadata and storage has the payload and then, the function app comes. Into the process logic, app or function app, we. Switch between that based. On the service requirement. The. Logic, and function. Apps are just the processes, which will just do the processing of the messages. So. You have you, have you have a bunch, of CI in a bunch of different path services so what does that look like. So. We have pass. Services, right. And, this. Is what it is, these. Are all the components that it has, so. What is what is it what are your CI and CD pipelines look like on this okay yeah, I can go through that. So. For all of these components, we, have. We. Have any build catered check-in for all the developers to check-in. Only. Of the unit tests are working fine and the solution is building, on. A daily basis this is the daily build that we run, this. Is we have created it in V STS if I go to the Edit here. We. Can see all the steps that we style. Copper's just you, know style of coding that the developers are doing is all, consistent. And then restoring, the nougat packages that the solution is using certificate. It's interesting because we have to introduce because of the current grid scan so, that we do not have to check in the certificates, and upload them to the build machines so. We started using PowerShell, to. Use. The base64, string to. Create the certificates, on the fly in the build and keep. It in, the drop folder and, we. Run unit tests on top of that and create a drop folder in, the build machine for. The, release to use all of those packages this, is for the web app and this. Is for the function app these two are we have two different function, apps and so, that's why you see two packages, for the function app and then we publish that drop folder once. The, daily build is run it Green Knight that, we have. Set. It for. We. Go to the releases. Triggered based, on the daily build and. You. See different environments. Here so. We. Have, in. S your we. Have, three. Resource groups for each of the service that we are building one. Is the CI continuous, integration where, daily builds are going and releasing. And then, we have and, they are in two different regions that we are using that's why you see these two and, then, the other environment, is you ad where once the feature is built tested functional, test all of that is done, that's. Where it goes to user except instance, testing. But. There. Are approvals, set for this like, somebody, has to approve for the deployment to happen here and then obviously, production. Once. User, acceptance. Testing is passed we go to the production environment so. All of these are chained, in our release pipeline. CI. West, us is the first one that gets deployed if I go you, can see later there are like 30 tasks, so. What they are doing is it is a combination all of this release is a combination, of. Running. The PowerShell scripts are the. Deploy. Using the arm templates, to do the deployment. So. Resource. Group it is creating the resource I can go in detail, for. All of these steps if. Somebody has any questions but yeah these are the so at, a high level it is a service bus deployment, this, is also interesting where, when.
Once The service bus is getting created, we need to know the connection, string and use it maybe to. Set up the functional tests and, whatnot so we, can in V STS release create a VSO. Variable, at, runtime which, will actually have the connection. String to that particular, service bus so, all of that can be enabled through the arm template if you, if we look into the arm template we, are using. VSDs. Tasks called as your resource group deployment, and. And. We, are saying that this is the template that I have to use from that drop, folder and. These. Are the parameters for that particular, template. Right. All. Of that we have are defined, in our segment solution. And. Then, this is again, the same task, but, for, the, event hub and, then. We are storing the event hub connection, string, storage. Account and then. The. Containers, needs to be deployed in the blob storage account that we are using, and. Then we also upload certain files to the blob configuration. Files to the blob that step is there so each step. Is well separated out and we, keep on storing the connection. Strings or keys that we need for the functional tasks to run. Cosmos. TBR, dr. B we have the, same step here and then, and. Cosmos. DB we need to create a collection so there is a PowerShell script that we have written for that and then. We. Are also using application. Insights so. You see, the steps for that arm deployment, as well as the PowerShell script to get the variable, out of it and the, function app there, are two function apps as I had said so. This is interesting this is a tokenize. With. X path regular exception, what it is doing is once. You have a config file checked, in into, your solution with. The with a very particular, specific. Semantics. Where underscore, underscore double, VSO variable, name underscore underscore, autumn. Out-of-the-box. Tokenized method will just put in the values in that, configuration. And. We are using key vault so all those storage connection, strings service, pasq connection. Strings are actually stored in the key world, so. This is cool. And. Then function app is the other function app that I was talking about. Once. Yeah and the web app which, is getting deployed at the primary region we we. Are using the arm template and, the, powershell command. To just update the config, for. The web app and, then. Upload, certificate, the build that had created the certificate, so this step will just upload them. And. Then, this. Is the functional test config, which we, are just using a PowerShell script to update, the config file and, then.
We Run the, functional. Tests using visual studio test. So. Once the functional tests are passing that means we are good with, our CI environment and, the, next environment deployment, starts so. I've got a couple questions here you mentioned cred scan is that is that an external internal tool what what, is cred scan right, scan is internal, okay it's. An internal tool that I think we use to go check, code. That's being checked in yes basically, to make sure that you're not storing any secrets, and, you see credit card certificates. Yes and, I think it's it's. A hard life, level. Kind of policy, that you know every. Checking you are making it, is making sure that you are not basically, checking any credit. Credentials. Over there so I think right. Now in our organization, you, create you basically, check in something, that. Particular, build is gonna trigger to say it's, gonna check the credentials, in your your. Particular check for that particular, code right so, I think if that's that's basically a policy, but you. Know organization. They will fight, policy, right now, and. This is something that VSDs, does for, you that's kind of nice I, had a question for the three of you so you. Talked a bit about here so the arm template deployments, you're using in, this point are, fairly, narrow you. Know you you can have an arm to an arm template that has everything. Everything, under the Sun in it and so you. Know where what. Have you found that are the pros and cons the balances especially, when you start to talk you, know service fabric or data, warehousing what are the kind of frozen worth what's, the balance that you found I can, answer that. So, we started with that we created an arm deployment, project in Visual Studio and we, kept, all our templates. In that, and we. Had just one task in, the VSDs release, where. Just. Everything gets. Deployed with that arm, deployment, but. The problem with that was whenever there was a single failure, with a particular component we. Didn't have much control through the release template. That's. Where we took a decision why not try out you know separate templates. As tasks. On the VSDs release so. That just worked for us but if you have a very simple. Deployment. I would, armed deployment. Project and just, have that one task and, it does everything that's the best if you don't have to change, your components, frequently, and you do not do very frequent deployments then. That's the best okay. And, so we've walked through the release trip yeah, I was just saying yeah that's totally, concur. Bertina said you know. It's. Always we, have been, depending. On the project size especially in the data warehouse world, we. Want to bucket eyes stuff right so we exactly want to know what step fail and why it failed and one, of the additional thing is from. A data warehouse point of view or a big data point of view there's something may not be available so again we also allows you to write your own components, and use them and then abridge, them all right so then others can use them so, that's kind of amazing. Feature, which, helps. Us leverage across, our better is, theirs do, you want to show us some of some, of your stuff now. Sure. Yep. So. I can showcase. Talk. About. The. Data warehouse. The. CI we have implemented, in the data warehouse so, this project is actually a sequel data warehouse. And. This. Basically you know shows you it's. Not easy that to read to be green all the time so you'll see there's some of the tasks, are orange there some of the tasks where in. Each of these columns. Are depicting, the environments, right, so. This is where the, production deployment, has happened, so you'll see the all for green but. Rest of the day it's only the integration, is happening so, some of the tasks might have failed probably. A unit test or two failed and. That basically, is a good way to make, sure that you know your code is. Completely. In place or not are you with, your check-ins are you breaking something or not right so that's kind of as an engineering team that keeps us honest so.
If I have to go open, one. Of the release. So. I can imagine yours, is the team that has it's it's not easy being green Kermit, the Frog up on the wall somewhere yeah. So. Just. So you know. So. We have like three. Environments, technically. If you look at there's one integration. But, you will see integration, environment is in two parts and there is a reason why we had to do it and then. We have the end user or we call it you bet it's a hybrid, environment we, don't have we. Were able to cut down the number of environments with the arm because of the, CIA efforts we have put in and, then the production now, if you look at the integration environment part one it actually is, what, it is doing is it actually, deploying, the entire data warehouse from scratch. So, the steps what we have here is. We. Are taking, the bill, once. It is created. After. A code. March. As, your keyword as Hina mentioned all our secrets are in key Walt not, really, compromised. By any way so, we try to fetch all the secrets. From the key vault and. Then if you look at this step is about procuring. The sequel agile so, this is a true pass implementation. Where, this, environment, doesn't exist, it only is. On the fly we created, and after, deploying and testing everything we, decommission. It so, this. Is where we'll procure our hardware, and, then. We copy, the. Database from production, so in. A data warehouse world, one of the bigger challenges, how. Do you really be sure, about whatever, you have done is really, working because. You need fraud equal, in data right but then you have a lot of compliances, where you cannot copy data from production to a pre production environments. But, this is a kind, of answer where it. Doesn't exist no, one has access to this environment, it's only a service principle which has access, it, creates, the data it. Creates the database copies. The database from, production and runs, all the test cases and some, of the important thing is out of the box some things are not available for, example, most. Of us would know with a DAC pack deployment, which. Which is very very powerful thing, creates that diff and then deploys it on top of the. Existing database, and you cannot really get away with it but, then it has a limitation let's say if I have a column, which gets renamed, because. Of business reasons the schema has changed the. Backpack, has a problem today or a gap today there it actually thinks, that there's an existing column which was renamed got dropped so, it drops that columns and creates the column again so, then your data is lost so. There are a few steps we have taken care like renaming, column before we actually go, and deploy, the dash back we, reuse, PowerShell script to rename the column so that when the doc fagged applies on top of it you don't lose data. Then. There are a bunch of other things, and, scaling. Of this server, so when. We copy the database so, if in production, we running a premium, to Edition we. Want to run, our test, cases faster so what we do we actually scale, it up using. The agile full, capacity, fabulet capability, in terms of making sure everything is run faster so we kind of scale it up to a v4, op6 scale and then, if you look at after. Doing a bunch of other operations where we have lot of encryption decryption pipelines. All, these things, get deployed and then, finally we run our, unit. Tests so, unit tests are very very. Important, for any project. And data warehouse what we have been always, kind of lagging behind so. This is a kind of an honest effort to you know how even unit test cases in place so. By. The time you run all this. Integration environment done, with it's all unit test cases you are pretty sure that whatever code. Has been added, is, kind, of pretty much tested the. Reason we have kind of and. Then obviously we do as your data did a factor deployment, so all our jobs are. As. Your data factory job from the porting.
Data From one point to other point. And. If. You look at there is a reason, we have folded to part two one, of the core requirement, for any, platform. Is to make sure that your. Functional. Test cases are. Now. The. Functional test cases in a data warehouse could, be a little tricky in terms of, you. May not be able to achieve 100, person hundred. Percent. Functional. Test case green, right, and as, you've seen that if. Even, one test case fails you actually turn out to be orange but the. Unit test case should be hundred percent right, so we wanted to make sure that, any. Day even if one, unit test case fails we should get to know about it without really digging, into it and going into it so we can't afford it, so our functional test cases are under as part of Part two whereas. The. Unit test cases are under part one so, any day we have an orange or on our very first, environment. We know our ut's are failed so the developers, actually jump in to fix it. Whereas the functional test cases sometime there is a data discrepancy, source. System probably. Has got refreshed by the time we deploy it so some of the test cases might still fail but, then you really know what is really going on and that's why it is forth and then as you see at the end of it we, decommission. The whole environment so, kind of it this environment, stays for about four, to six hours depending. On how much data we have and we. Are able to use CI to kind, of have a new, environment, but every, day with the new code which we check in and so the, having. Environment, only offer a limited amount of time with, only a managed service identity having access, gets. You covered on all the the security and release you know all of those things make sure it covers that risk. Do. You when, you you said you scale it up so then do you do your essentially, when you when you compare, your regression analysis. Is it, just purely based on the prior runs, because. You can't really necessarily then compare, your. Your. Test are you at to production if your test. Environments, at P for when. Your productions are p2 so how do, you resolve that is it purely based on past tests, so that we have kept separate the idea, is how fast, you can build the whole environment right, so the, regression test, or performance. Test as part of a separate operation itself so, we are in a DevOps model, so, we have one of the person who is the DRI or directly responsible, individual. Has. A job to, keep a good track on the performance, and agile provides a lot of performance, indicators, on its own so, we try to cut down on custom implementations. We keep an eye on how our performances. On those areas so, that's how we not, try to do, a regression on, using this, whole process because. Whether. We probably, I didn't mention that we are at zero downtime data, we know system which is very difficult to get into the, reason we are able to do it again at the same maybe our job is to make sure that we deploy faster, than anyone can. Imagine from, a customer point of view. So. We have not kept it the, performance or indicators are not part of the whole thing is kind of a separate evangelism. So you really just scale, up to make sure that you can get that deployed as quickly as possible, can I kind of bring it back to any and. It's very important like you, know when we talk about DevOps, right, I see, like there are four, pillars of it like the, very first is the planning right when you're doing the coding and the, second is CI 30, CD which, you're talking about here and, the last one is the monitoring, right so, I think if we I think, the the concept of DevOps is that, it's.
Okay To fail but, it's. Very important to recover from it right because you you fail fast but you have you, recover from that failure cookie, I think that's the beauty of DevOps so using. All these technologies, using all these automation, using bsts, we are able to achieve those things. And. So you, guys, are your your team is doing service. Fabric yeah so what, are the other things that, that. You'd. Like to show that are different, as far as when it comes to service for hybrid doin's yes yeah I think. You. Know one year back our. Application, was it. Is still one only thing but we, are from, the last one year we are kind of splitting those into, you know smaller micro services, today so. As. I, was talking about right you know it is very important, to have. To. Shorten your life, cycle, there right like. You, know if I simply ask you how, much you know time it will take for you to. Get one, piece of code. Change, to get it get it into the production I think that lifecycle, if we can reduce that that. What we want to achieve here you know as per our DevOps methodology. Right, so let, me show you like our CI. Bill. Definitions. Which. We are using so we are using a zoo, service fabric here and. This is our daily bill, definition, so, it's pretty safe straightforward like you can see we. Are building, the solution here and. We. Are running our unit tests here that. Every check-in basically we are making sure that you know you're not physically. Corrupting. Our you know the source code here, this. Is a very, important, step. In the bill definition, and you are using, a zoo service fabric so it is basically creating, a service. Package. For you which. Is basically a combination of, code and configuration. That, you have, you basically will use in your release definition. To. Deploy into a particular cluster right. These. Two tasks, are important, because if you see in, our during, our build we are using, a flag called deterministic flag. This. Flag basically. Makes sure that whatever. Input. Your, compiler, is getting it can or it. Cannot create the same output so, there if there is no change in your code you. Know your binaries would be exactly. Same as it was previously so that. Because. Later on we want to use this particular feature so that we should not deploy any service which do not have any changes, right so, because we don't want to use resources there right so, so.
You Can see these. Two tasks, are basically, is is, needed, because, TDB, files are always getting. You are using deterministic. Flag or not so we, have to delete pdb files because that, could always, get changed, right so, this. Is important for this particular step in, this particular, step what exactly. We are doing is we are updating, the version, of the manifest like version of the service services. Which we want, to deploy on, a particular, cluster so. The, idea is if there is no change there, should not be any version. Implementation. There right so, this. Particular task is making sure that. If. You you know basically check, Park this thing it, will make sure that you are only updating, if there is a change right, but for, this starts to run, you. Know this. Deterministic. Flag is very important, here so you know make sure that we are using those things and then, there are some fortified scans which we run are on. Our code, base to do some static code analysis. We. Are using a third party here, 45. Servers, which, basically scan all the code and you, know give you the report that all the things are in a good fashion, and. We, also run you, know catch scans, during. Our daily builds and. Ultimately, we copy our artifacts, so, that it can be picked up by the release definition, and you kick in the CD butter that so, this completes the, CA. Part and. We. Have the package, ready for our release for our release definition, pipeline so let me go towards release, pipelines, here so. You can see we, have a continuous, deployment here. Are enabled, for this particular daily, weight. So. If we, have like three environments you know as you know like. Development. Environment, and then we have uite and then ultimately, we, go to production, you. Can see it's. Very simple like very. Low. In number tasks like first. We what we do is we. You. Know as Hina. And Noah was mentioning, that we are using keywords for, storing. Our secrets, so, the only thing which we do in our services, is to just. Just. Print the, particular. Certificate. Thumbprint. For a particular environment that's what we do and, that's, the only part which we need to do the, rest it will basically, fetch, when the system services, will run. This. Task. Is very important, for for, our service fabric, this. Task is basically deploying. The, packages. Which you have received in your bill. Definition, to, a particular cluster. So. There. Is a field called cluster connection, where you have to specify your, cluster, which where you want to deploy these services, to. There. Are several ways that you can create these connections so, right, now we are using certificate. Based connection. Here but you can also use like. Authentication. And SP, and all those things you can use those connection. Also and, then. You have to just specify. The cloud, you. Know publish profile in the application parameter and that's it you just basically use. That particular, application package, to deploy that particular, services, to your, cluster itself and then, we simply running our BVD. Against that particular services, deployment, and then. We send out the report notification. That you know. And. This last us is also you know very important, so. Every. Time we deploy something on our, environment. We are making sure that all the cloud. Resources, they. Are secure enough and they. Are basically, following, the organization policy from a security perspective so. There. Is a you, know there. Is a task, provided, by VSDs team whose. Which basically, scans, a particular resource group and see if there is any security. Vulnerabilities. In, that particular resource or not so. You can see I specify. Two, resource groups here which basically has, all my. Resources. The, reason of having two resource group is that we want to kept. Cluster. Specific, resources, in one resource group and all the dependency. And from dependency, what I mean is like if you're using a keyboard or, we. Are using document. DB those are like my dependency. To which it has the data, so. We kept it different so that in case we you. Know tomorrow if you want to post the data, first.
The Cluster itself we. Can do that you know without worrying about our dependency, which are which is in our case as keyboard, or document, DB right so. That, basically helps us you. Know doing. The security scans for that cloud resources. So. This was all about like you know. You. Know have CI. CD, pipelines. For your service fabric, you. Know capability, and. It's, very it's. Very useful because. Right. Now whenever. We go to production it's it's not a ceremony, for us I mean you know one year back did. Them all only thing. It's. Supposed to be like a ceremony for us like you know we, have to ask. For a downtime, saying, okay you know we are deploying things here and, it's. Gonna be like town for like two or three hours but now with. This micro services, there, is zero downtime, you, know where every release. It is like a roll rolling, upgrade which. Do not have any downtime. So. I've got a few questions for, for each of you then, that's. Interesting that especially. With service cyber gets built to be up all the time yes for mobile stuff, when. You're doing your your. Your, delivery, in your release into production you. Know, for your application, or for some of the data warehousing stuff what, is that level of of is. There a few seconds to a few minutes downtime I mean how what. Does I mean for for your worlds. There's. No downtime, there. Is a staging environment where the bits vehicle and, if the functional tests are passing they were promoted, to the production. Environment in Azure. But, if it fails it doesn't, goes to the production environment. And you. You schema. Changes are things. That kind of people have to know about yeah so what so, how we actually deal with is we, build a parallel environment, well. You know current. In mind these mean, access by the, customers, the. Pal environment, is getting built so we today, to our customer we say it's a zero downtime deployment. We still give a window, can. A fifteen minute to, thirty minute will be saved there may be an. Intermittent, ex issue because, what we have to do is we have to once the deployment, is over we have to swipe it so. So. We are doing it for last one years and till, date. None. Of the customers has reported, that they they, reporting, or interrupted, because we try to do it in those those, hours where probably, no one is using so, technically from those days if you look at me a kind of zero downtime but the, swipes would take a couple of minutes now is that something that I mean just I think it it's. Probably still helps if you if you have an agreement with the business the other teams that hey yes, on, a daily, basis or on a weekly basis.
This, Daily. Or weekly recurring, this timeslot there might be intermittent, just so that way there. Is that planning cuz I think one, of the biggest, misnomer, is is DevOps means, that like there's, I hear people uncle I'm do things in DevOps I'm like well you can still plan I'm. So they're still planning involved, with everything that you do they're. Just releasing, and building quite a bit definitely for the production, yeah, environment, yes one, of the other questions that came up you knows well, Vikram. You were talking about your service fabric and how you. Could like wipe out a cluster. So. The, reality is I'm curious, are you using, is it. If you're gonna wipe out a cluster it better be all stateless services, so what, do you do have stateful services and what are you doing for those yeah that's a great question so we. Do have stateful, services, and, you. Know initially. When we thought of like you know we kept this resource, groups separately, but you know when we, launched. The stateful, services, then we thought of okay. We cannot pose the resource, group itself because we have the state with the services right now so, I think the approach which we are using right now is to take the dump our of it and you. Know in case if you want to you, know purge the environment, we have a back-up plan there. So. That we can restore all. Those transaction, from that backup it's okay and moreover you know in the recent, Microsoft. Build we. We. Saw that you know the. Service fabric team is providing, you know more backup options, now and. Which. Are most more fast right so you, know more have to like go for an external storage itself you can use the distort. For your you. Know storing your all the transactions, backup right so I think, with all those technologies we still can you know achieve what you want to like pause the environment so part of your deployment, could be to move. That back. It up and then wipe the cluster in the cluster and move know your transactions, to yeah yeah, okay, some. Of the other things that a view slightly, mentioned. Was. Really about security. And key vault so, what, are what's I, know, the. Magic key vault has made a lot of changes in the last year, two years especially, within the last year with managed service identities, and then a lot, of the other build tasks, making things easier so, can. You talk a little bit about what, what key. Vault security secrets, management. What. Are the things to be aware of or what have you had to do in your environments, and with with your stuff. Yeah. I can talk about that um so, earlier we were using the web, our app dot conflict to keep the key, vault connection. Strings and keys but, we, had to encrypt. Them with a certificate, so, that they. Are not visible right but, with. The current more you, know more, tight. Security, we. Are using MSI option. And. It's, very, straightforward, there. Was a little bit cold change required for us to say that how to get, that MSI. But. Apart from that we, don't have to now use you, know certificates, just to encrypt. Those keys and keep, it there so. That is one learning that we had just. If this MSI feature is still not available in a zoo service fabric so if, you are using you. Know to deploy a zoo service fabric we are still you know kind of I think that that feature is coming in but right. Now what we are doing is we are storing, the thumbprint of the certificate, so if you're not storing any secrets in our to these definitions. But. Yeah clusters should have their certificate, installed in. The cluster itself and then take, care of so. We are still safe we are not storing any any secrets, in our release definition, we are not storing any secrets in our code base, but. Yeah I think in future in there they have those. Features coming up in the roadmap. Yeah. Okay, let's make it easier so just to add yeah, but we, use the same way in, fact to make it more secure so today if, we look at from a BST F obviously. As point, of view from the release variable point of view we, use service principles, to install, everything, so. There is VST still has a limitation, where it still depends on a key instead, of a cert they're. Just pretty secure but. My. Team has you, know have been you know trying, to look into how do you really get rid, of that kind of. Dependency. With respect to not to have those kids because those keys are mostly, valid. For a year or two so. That's the only area where probably I know things could change in, coming, days but as of now if you look at from end to end all, our secrets, and. Passwords. Are secure, in key world without, even the engineers, having access to it. Another. Question I had because. A couple, of you talk through your, different environments, of the nightly. In the UN and prod and creep rod. What. Does what observe your, gating, and branching, strategy look, like for. Your apps. What, is that side of this for. Us every. Developer has a local branch that they can publish to the service as and when they, think that they, want to keep their gold safe but, once they are ready with their gold all.
The Unit tests at least one happy path unit testing functional test is there that's the minimum thing that we ask, all the developers to follow. Then. They can try to merge with the main branch which, we call as dwell a branch. That's. Where the gated, check in build. Gets. Triggered and, they. Will be able to check, in only if the, solution, is getting built and the. Unit tests are passing, so. That we have the sanity of the code, witches in developed branch so, that's the branching strategy and. Then when, do is, is def is developed and when. Do things go develop from develop domain or is that yes. Once a feature is built. That's. Where we want it to, go to production, okay. That's, where. There. Are checks, about, the functional tests and unit that's being complete. They. Are not automated, yet but at least I, mean. The test automation is there but, somebody has to see. That okay all the test cases are covered like. The functional tests especially and, once they are there that's, where we push. To the master branch from the develop branch okay, and master. Branch is the one which gets deployed in all the environments, okay, so essentially you have local dives on their branches yes, going to the develop which is your integration. Point for all the dabs coming that should be building and building, every night so every, night that's building to make sure nobody's breaking it everybody else's stuff and. Then from time to time. When. The business and in good engineering team deems it's appropriate, then you will essentially use release management. Yes. From developers, perspective we, keep take care of the functional desk spinkle, covering, everything but. Obviously. The UAD is user acceptance, testing by a new user, has to say ok yeah, I'm. Good right, I mean that is where we are just involved if there are any issues mostly, the our PM's, are you know interacting. With the partners. Or if, there are internal teams, and. Making. Sure that everything, is passing into an functionally, and that's, where then, we approve the production deployment, after that and, I think you. Know when we talk about their boss philosophy, right there. Is nothing called production right I mean, every. Code that you are checking in it should, be a production really right code so. For. Us it's like when, we do a feature development we create, a feature branch and everybody, start working on that and, eventually we when whenever somebody, wants to check in right so it, has to be most with master with all the sanity, checks like you're, not you're, done with the unit unit test you are you. Know checking in in credentials there right so all those you, know minimum. Barely level, checks out there when, you're checking into the master right so.
I Think, this is very important, because you know it's it's a it's about changing, the mindset right, I mean think of. While. You're checking in your chicken in the production right away because. You. Have the automation will, pipelines, with you and even, as snap it into the production right away right so I think this is very important, that did. All these automations, with, all these tools right PhDs, and everything we. Are you want to achieve that so that there, is a continuous, delivery of value to the end users right so, your team is your the developers, and the service traffic staff are committing right into a particular feature branch, which. That is actually building and deploying to. Some sort of environment on a nightly. And then, when, do you make the call that your, feature branch gets merged with with. Master and then when how does what does that look like for you guys yeah so I. Think we. Have you, know gated, gated. Release. Pipelines, so we, run our you. Know functional. Test unit test and we also see if there is no exceptions, in our systems are you know moving from one environment to another, environment. We have gates for that act like if. You found any exceptions, of a particular type for within a period of time we, cannot block that particular. So. Those features are there in BST is we are leveraging those features. So. That you know ultimately, the. Code which is getting merged and you know buried on a, particular environment, is is, all good. Enough to go into production right. And. Moreover you, know these automation, builds we, are getting a very rapid, feedback from, our stakeholders all's right so. We. Have like two weeks of sprint and every sprint we have review and with. Our stakeholders, and they provide you, know quick feedbacks, to us so, that we can you. Know quickly apply those feedbacks, using, these automation, because imagine, if they don't if, you don't have these lease pipelines, we. Cannot achieve those things right, away right so this is very important, you. Know while you. Know getting into the dev of Mythology here yeah, and so one of the other things that I am I didn't mention that we have we haven't really talked about is just the fact here for, all of our line of business applications we. All live in the same project. In the same V STS instance and, so that means any.
Of Us that are checking in code can. Go search everybody. Else's get, repository, to go find acceleration I think well, the import other important, things so that means that all the thousands, of engineers in. Core services that are developing line of business apps can, see each other's code can, go fix we can go run scans. We can go look, at see, who. Are the teams that are doing service fabric by doing a code search and. Go learn from those teams but I think one of the other really important things that are, in, RVs ts instance for example we, have kind of like some, federal level, you. Have to do this there's no exceptions, and then there are state. And city level so each of the applications, are cities they have autonomy, but, at the federal level we've said hey everybody's. Going to live in the same project but she can kind of see it up and I'm sure up from the screens you've seen that little moniker, one itvs, o at. The federal level we've set up iterations. And a default two-week cadence. Across, every one and. So our descent she are our federal level you, can't go, around this art everybody, has to use get. The. Two-week Sprint's are set up for everyone if for some reason the team needs. Larger. Than two-week Sprint's they're free to change that what two weeks are set up for everyone, and. Then the people that run RVs to use stuff are really good about let's. Let's go run credit you know credential. Scans let's, go run scans looking for secrets let's go look for all that stuff on every single building run store system and. I think today I'm. Pretty sure there are I don't, know how many git, repositories, there are but, I know we, do over a thousand, releases, all. The way through pipelines today, and we do over, a thousand, builds per day but. No alcohol completely, automated. So. No. But I wanted to get back to to what. What, do what. Is your branching strategy look like for the data warehousing stuff. It's. More or less on the same lines the only variation, we have is since, we. Don't have too many free production environments, we just have one hybrid environment, which we expose to the end-user customers. We, do not have a develop branch so. As a as, a engineer every. Sprint I branch, out from master, work. On my story, and as in I mentioned you know we have get a check-in you have the moment you create a pull request yeah, but we'll file and everything is getting.
Checked If the build is compiling. Everything or not. The reviewers. Who are gonna review they actually intentionally. Look for a couple of things. Have. We covered. The unit test and the functional test scenarios, apart. From carribean code I mean that's the kind of you know in our behavior now so, in, case if we don't find the UTSA or the functional test cases are covering the code is being written we, kind of reject. It are you talking about the pull request, yeah feature I'm talking whiskers as we didn't, get a check-in thing I and then once, everything is a proven gets merged with the master and once. It is in master, every, second, week. Which is every sprint we, deploy in our pre-production, you. Ant environment, hybrid. Environment, which is open for our product owners and even the end users to, go in test so. That, way we have tried. To keep it very lean, everyone. Knows how to branch out and March it back and what it means when you merge it back it, means it's going to go in you add yeah, so, essentially you have instead, of things deploying. To production on, a nightly basis you essentially everybody. Knows hey at the end of every two weeks, yep the. UN environment, is going to be updated in whatever the prior you environment kind. Of may be rolling to production at some point even. I told myself I wouldn't have any any, notifications. Up and then will popped up great thanks, I'm glad we were up on our on this side of that, so. What are the other questions that I wanted, to ask, as far as V STS itself. You, know you know you kind of talked a little bit about the tokenization. What. Are some of the other things that that V STS has released in the last say. Year, where. Maybe it's changed, what, you've done maybe it's made things easier, or for, somebody else that's going, down this path. Maybe, a little bit of learning here there would help them I think I can talk about one of the things let. Me show you that you. Know recently, in. The. Release pipeline they came. Up with something called gates. Gate. Is basically you know you. Know as Hina was mentioning, you, can have pre-deployment, approvals. Which, are basically manual, approvals, right, you, can assign somebody who can approve you know it, be basically, will check the sanity of your, artifacts and then you know allow, you to deploy, the particular. Art effects on a particular environment, plus. There are you. Know some feature which is called called. Gates right they. Recently introduced, these feature. In, the V STS so right now we are, using, one. Of the gate called you, know monitoring, alerts basically. What we are trying to say here is that you. Know after my, yet environment, is done and. If I am seeing no issues, in my top insights which, is my telemetry there, is no issues it, going to check like in a 15 minutes interval if, everything, looks good it. Gonna say okay I'm good to deploy into, my next environment, right so these. Are like one of the gate but there, are several types of gates here you can also. Use like you. Know external, API. For. Example you. Know in. The build they were giving example, like imagine. You, have your product, owners and, they. Usually, have to give. You a you. Know, give. You a go that you are good, to go to the production right so, they were using DocuSign. For, that so, once, P. Or basically you know sign your particular. Release it. Gonna call that API that you know whether that that is being approved or not and then you know basically. We. Start deploying on the production. Environment for that so, these, features are very you know very useful. In some scenarios. So. This, was the feature recently you know stss, provided, in the release definition, itself so, this is also a very useful feature, you know we can apply on your release, pipelines here and, then I think one of the other things that I wanted to touch on briefly is how each of the teams.
You. Know like I think that part of the build, talk. Spoke. About this who. Are the target audience for, application. Insights and, who's, the target audience, for as, your login oolitic, so a mass for your app so how. Do you how, do you distinguish, who who, your consumers are for those four application, inside it's. Mostly the, reports. That we create for the business people as well as the developers the DRI. Who. Look into that so it's solving, two purposes how. Because the telemetry app that, we have it. Publishes. All the data not, just an infrastructure. Related but also the business data, related, all, of that gets into the custom, dimensions, of application. Insights with every, stage that, we are passing through in our code and, as. A DRI, I can come and write queries on top of application, insight and say okay I am interested, in this particular order ID or, delivery. Order ID or, purchase, order ID right and I can get the data because whenever. Business, comes back to us already right it's always or our incident. Has raised it's always based on okay I have, a if this unique business ID purchase, order ID I want, to know what happened to this one right, or the, other way can be there was an error and we got an alert, so. In that error, itself, we would log the purchase of those unique IDs to be. Able to query those. Efficiently. An application, inside and also. Then that application, inside data is used to create reports for. Business. Side. Of the things, so they can actually see like how many orders came into our system. Per. Day or per hour they have all those options so. App insights is really for the, business, and then your DRI is fundable, individuals, and. Then the log analytics, is really used for the. Operations, side, of the DevOps teams to go figure out what is, that is that fairly similar for everyone yes, I think the. Point here is to do, the proactiveness, here I mean instead of you know the customer is telling you that I am facing the problem I think the in when you are in DevOps you need to actively. Look. Into those inside, and see. To, figure out like you know if he is facing that problem let's fix that right, so these. Things are very useful. Just. The, variance for a data warehouse kind of project, the, app inside is not a deployed, choice. There's. Not too, much off logging you can do with app inside so most. Of the time you have if. You're an azure pass or any, of the other as your. Offerings. For. The big data you, get a lot of logging. Available. Through, the portal, and. Obviously you, can kind of plug, it in with your power bi as well but. It's. Still being a warehouse, system you want to know how your queries, are performing, from, one stage to other stage some. Of the things that are out of box available with sequel where it gives you performance counter, and you still get to know but as as. A engineering team you want to know how much time each of these, you. Know code pieces. Or code blocks are taking for you so we do custom logging, for that but. Kind of we have an internal tool which. Is called unified. Telemetric. For our IT, organization, which provides you, with, basically ports whole dis, logging data to, app insights and through which we kind of get, to know about it so, for us it lets a twist but yeah we still use app inside through migrating. Those logs to have insights. Are there any other things that that, may. Be besides token answers how. We use app insight so if there are any other things that if. If you were talking to somebody today. And. They, said hey we're we're, looking at a sure, we. Want to start going down the Azure path and. They're starting to look at setting up CI CDE they're starting to to.
Look At doing everything what. Kinds of tips would you give somebody, that's. About to embark on the same path that your team's have just gone down. Yeah. I would say you know it's not easy to untie the way you know it, going. To take a lot of effort in terms of what you want to do with all these. Implementations. But then, it. Worth every penny in terms of what ever you putting it's gonna give you a lot of return later on once, you have it because a lot of these are repetitive processes, which. Are very important, to. Make sure your quality. Of the product is taken care of. So. I would say initially. If you plan carefully. What you need to do design, it. It's. Gonna pay you off big time so. I would say, definitely. Look, out for these opportunities from modern engineering implementation. Point of view and, I, have, seen in visual visual. Studio team system has, been adding. More and more of these features anything. You are new using, new in Azure or. Cloud, computing a, lot, of outer-box support is coming in so. It becomes very easy for you to kind of on go to those and be, very productive, and. Just to add on what noble said like you. Know VST is not you know most, of the developers think like ok when you talk about VST, s you're talking about Microsoft, technologies only right but it's not the case we use tear supports, like, all the open source we are using you. Know angular, application. Today and I I, was amazed to see all those tasks. Which are in built-in VST F which we can use to, build all the spy applications, and moreover, you can create your own tasks so, if. You have your. Own custom things, to do you, can always do in areas yet so I think, students very. Much into. You know following, your, journey. Do it dr. box yeah, and I know one, of the other teams that I've just been talking to that will have a blog post on mania blog out soon is they're. Doing Linux VMs, and. So they're they're using containers. Kubernetes. With Linux right now and, there should switching over to the azure kubernetes service with Linux so you. Know even though we're Microsoft, we still use quite a bit of Linux, and open-source internally. You. Know is there anything that you'd, yeah. Same thing there, are tons of options so, we had to do some help in trials in fact in our organization, I am working on only in order. Management and fulfillment, space but, we do have return space as well as planning. We. All all, these spaces have actually, different kind of solutions, as, your components, that we are using and we learn, from each other and we say oh you know what this works the best this. Is the design pattern that we should follow, as an organization, we standardize it then, so. They will be learning lots of and on. Top of that I, have observed that. Almost. Daily if. Not daily or every six, months there. Is something new coming up in is here and we. Always make sure that we are trying those things, out and. Keep. It in our path for. Future you, know considerations. Of how we, can use. Their capabilities, in our solutions. So. There is a team who not, team but there are few dedicated people who actually does, those kind of pcs, in, Azure and they kind of give, a, knowledge, transfer or you. Know pcs session. To others so. That's how we learned about. Those and we. Start incorporating in, our new. Designs as in when they are available, don't, it was amazing to see that you know, the. VST is and the zoo teams are working, your hands or hands whenever. You have feature available in a zoo and you, see all those availabilities. Are there in BST is to kind of get.
Into The CAC de pipelines, altogether and you, know see the beauty of that. Awesome. And. I know we. We sent out a link during. This this webcast, a, novel, to a blog, post that you wrote on modern, data warehousing with continuous integration so the. Audience has that please go take a look at that especially if you're looking around the data stuff so. We're. At the end of our time thanks. Everyone. If, we if well. We already talked about key key takeaway so thanks. For sharing those I think the the general consensus, is completely. Worth the time it's. Worth you, know not deploying from your laptop and having to deal with that it, also covers your compliance and security and those everything, so and, all the best to what you know you do your, focus is on solving, engineering problems, they're actually. Track. Of all, these side. Technical. Deaths yeah, it's it's get, get, me focused, on my happy path of solving problems and I like Vikram's. Comment, where, there, is the deployments, are no more a ceremony, yeah like, I could relate to back, so. Thanks. Everyone, the. On-demand version of the session will be focused will. Be posted, to. Microsoft.com, wack IT showcase soon you, can also find other IT show kids content like business and technical case studies productivity. Guides and upcoming webinars, on the Microsoft, comic, tea showcase site as well please. Join us for future webinars and bring your colleagues with you.
2018-06-29 21:42
