Bootstrapping Microservices • Ashley Davis & Damian Maclennan • GOTO 2024
Hi there. And welcome to today's session of the GOTO Book Club. My name is Damian Maclennan. I'm a consultant CTO and software architecture trainer with a company called Stack Mechanics. And I'm based in Brisbane, Australia. Today I'm talking with a fellow Australian and Brisbaneite, Ashley Davis. He's recently released the second edition of his book, "Bootstrapping Microservices with Docker, Kubernetes, GitHub Actions and Terraform." So Ashley, why don't you tell me a little bit about your background in the software industry and what led you to write this book? First of all, thank you for joining me, Damian. I know that you've got a long history in kind
of what we're going to be talking about today as well. So I'm really excited to talk to you about it. I've been a developer for over a quarter of a century now. So, you know, I've been around a bit. I've done all the languages, at least. Hopefully all the important ones. So I've worked across a
bunch of domains. Worked for a whole bunch of different size companies, from startups to the biggest internationals. And worked with a lot of different, I guess, ways of structuring software and microservices or, I think, more appropriately, distributed applications is, I guess, mostly where I'm at now, although I still consider myself... I've spent a lot of time in startups, so I'm a constant dabbler in anything and everything you can imagine. So I work across the stack. And you have to be to maintain a career of length, you have to be a constant dabbler and always reinventing, right? Like front end, back end. I love some of the problems that sort of come into play in distributed applications. Really
interesting stuff, interesting problems to get your head around and to find solutions that only add as much complexity as is needed to manage the complexity, really, and to keep it just simple enough... You know, not to over engineer it, basically. So we have complicated problems to solve and are trying to solve them in ever simpler but more effective ways. Which kind of leads me into... And I want to dive right into some sort of nitty gritty of microservices. But your first edition came out in January 2021, which is a little over three years ago. So what brought about you writing the second edition? Was it industry trends? Was it based on feedback or just an update of the tools and better techniques that you saw out there? You don't get to write a second edition unless the book's popular in the first place. So,
there was that. Having microservices on the cover helps with that, even though... I mean, it's a book for working with any kind of distributed application, really. And microservices is just one end of the extreme of choices, which I'd like to talk about some more as well. So, Manning, who I've worked with on a few books, came back and asked me to do a second edition. And I'd loved working on that book. Nothing like that is ever finished. At some point, you kind of have
to say, this is enough to ship it. And so I had feedback coming in while I was writing it that I couldn't address. I had feedback coming in after I'd published it. Because if people reach out to me, I'm always happy if I can fit in the time to actually jump on a call and talk people through what they're trying to achieve, and why they're having trouble if they're having trouble. So I got a lot of feedback that way as well, and made a lot of interesting connections with people that way. And then things changed. Things changed in the industry. Software was updated. Terraform reached version one. I mean, Docker's been around for a long time, but when I first started writing the
book, I wasn't sure whether Kubernetes was really going to be the king of microservices or the king of distributed applications. And it seems, as much as I can tell, that it is now at least. And since I wrote the first edition, I got a qualification in terms of becoming a certified Kubernetes developer. I'd really pushed my experience with Kubernetes further along. Done a lot more stuff in production as well. So, all that influenced it. And also, I found simpler ways of doing things. In the first edition, some of the examples are more complicated than they needed to be. And through writing the second edition, I was able to come back, revisit that, and simplify it. And it's still effectively the same result, but just code and examples that
are a little bit simpler. I think about it again and get a chance to revise it and go through all the whole feedback process as well that Manning organized, which is a really great way of writing a book, that the explanations are hopefully better, are clearer, and more to the point. When you explain the same concepts over and over again, you come up with shortcuts to saying things, you come up with better analogies. And yes, that's something that I'm certainly familiar with as well. And like you said, a lot of the tooling's improved. So, some hoops you may have had to jump through four years ago, there's a shortcut for them now. Fantastic.
It's all about finding better ways to do things, more effective ways to do things. It's finding more effective ways to teach things in terms of the writing of the book. Absolutely. So as an architectural style, microservices have gone through the full hype cycle. We had the sort of really early adopters, they gained some real traction,
saw a lot of real world usage, and then everyone wanted to jump on that bandwagon. Some of those people learned some very expensive and very painful lessons. Then we see people a little bit more hesitant. You know, so when is a microservice architecture the right choice? What makes a good candidate for a microservices architecture? That's a really hard question to answer in a general way. Not everything needs to be microservices. I think part of the problem
maybe is that microservices have been pushed on a lot of people and developers have been forced to use microservices in situations that didn't call for it. So I think you've got to think really deeply about whether microservices are the right answer to your question. I want to talk about this more in a bit, but I'm more in the middle now. So if you ask me microservices is a monolith, I'll say somewhere in the middle. I think it's a great place to basically get the most out of both
those different ways of working. And then you get to choose to use microservices for features or for aspects of the application where it matters. So it's not like saying, you know, here's an app and you should definitely use microservices for everything. It's like, here's an app and it's structured in such a way that things that can make use of the benefits of microservices can be spun out into separate services. And I've got a list of examples of things that have worked really well for me in production applications that I've built. I've worked on a couple of different applications that needed to run AI models. So pre-trained AI models,
like, something you load up into TensorFlow. And not only does that maybe can, you know, the tech stack you have to use for that conflict maybe with the tech stack you actually want to use for the customer endpoints, that the customers are directly talking to to kind of use the web application, or the mobile application, or whatever is the back end is talking to. For those AI models, we always wanted them to be separate. It was so obvious that you wanted to spin kind of an AI model out into a separate service, because then you can kind of control the performance of it and , you know, the problems that it has. Like, you can have some sort of fault
isolation. So if you spin that kind of thing out to a separate service, you can scale it, you can throw a GPU at it. Any problems that are happening there aren't going to be affecting the customer. So I think that was one of the first things that I was like, yeah, like, this, you know, a place where it's cut and dry. You can put this kind of thing in a microservice and run a dozen of them as many as you need to kind of get throughput. It's a great use case. Any computationally or time
expensive processing that you can offload onto an async process and build it into an isolated space that's fault tolerant and not so time sensitive, is going to make your general app scale better, perform better. Right? It's a sort of an easy win. One of the applications I worked on, one of the ones that had AI models in it, was a general kind of asset processing system. As an extreme example, you could upload a video to it, and then the video would... Like, there would be a process that could chop up the frames of the video, do AI analysis on individual frames, you know, pick a thumbnail for the video somewhere in the video. I mean, this is an example of something that could be quite performance intensive that if you're set up in the right way to use microservices, even if you've got a monolith, even if you've got...like, most of your stuff's in a monolith, you can kind of spin this one thing out to a microservice, and suddenly, you know, your performance problems are gone and you can kind of keep that separate. You can run it at a really low priority,
if you want to save money, or you can spin up 10 of them if you want to get the throughput. So, again, you know, performance, fault isolation. And we had say like 20 of these services that could handle different types of assets and do different kinds of processing on them. You would upload an audio file and it would spin off a service that could do the transcription of it. So that was just talking to something in AWS and basically waiting for a transcription to come back. But there was just something that you kind of like to hold off from everything else basically
and keep it at a distance as it were. So that kind of thing worked really well. I've worked for blockchain companies and there's a need for security, and kind of ring fencing certain transactions on the blockchain or ring fencing any financial data, or any kind of keys, or secrets that you don't necessarily want the rest of the wider organization to be able to access, like, the security reasons, to spin off things like that into separate services as well. So that's a really good example. I've had some similar experience working in a compliance space where we had sensitive data. And again, through building that off into separate services, we could completely isolate and refence around that data. So it's a really good example.
And of course, there's reasons to use microservices for reusability and composing systems together as well. I would say never do that just for that reason. Don't split off a microservice just because you think it might be good to reuse it. In any kind of coding, you can always go overboard with componentizing things or creating abstraction layers and stuff like that. But I mean, a really good example I've got here is, like, having a separate service for doing email notifications or something like that, or like SMS notifications. Because that's something that possibly a lot of other services in your system want to reuse. And so it makes sense to kind of spin that out as a separate service. But like I said, I would never do that just because,
you know, it's good for reusability. It has to be that there is a need for that reusability. That's right. Any good abstraction is going to come from working software. If you start with the library, or the framework, or the service first, you're very likely to get that modeling entirely wrong. Ashley Davis: That kind of thing I think can come
as you build it, because you've got other reasons like what we've talked about for performance, or fault isolation, or security reasons. And I mean, there's a list of benefits, right, to using microservices. I don't know if we want to kind of tread that path. What haven't we covered? We've covered performance. We've covered security and compliance. We've covered isolation. We've covered reuse. Do you see any others?
Well, on my list of benefits where these are personally my top two, and there's like a world of discussion probably under each of these. But one thing that I really love that I don't know how well it's appreciated or kind of understood by people, is being able to build a system or design a system so that you can kind of replace it over time. And so that's really hard to do with a monolith. With a monolith, you're stuck with what you've got for a long time...if it's accessible. Versions of frameworks. But with microservices, anything that you've carved up into separate processes, then you've got like basically a way forward in terms of replacing them one at a time with whatever, if it's changing the tech stack to something newer or it's like done really badly, like, each service, you know... One thing I love is that microservices can be
like little islands of tech debt. But it's encapsulated tech debt, right? It's like, it's isolated tech debt. That's right. And it's already set up so that you can kind of throw a piece of it away and replace it with something better. As you said, even just like framework versions. And Docker makes that really easy. Because it used to be, if you wanted to upgrade a framework version in an organization,
you would have to upgrade every build server and every developer machine. Your build chains would have to change. And now you can just say, okay, well, I'm touching this service today. So I'm going to bring it to the latest, and I'll bring it to the latest Docker image to host it on. That's a very isolated change. It doesn't affect all of the other
things. So you never have those, let's rev the entire enterprise project, which is awful. Obviously, there's some anti-patterns that can break that. Which I think we'll talk about soon. But just being able to design the system, so that the parts of it can be evolved over time in a way that the system itself doesn't break. And you can keep it running, you know, while you do that, you can basically hot swap in new services for old services. If you've got a protocol, like what I was talking about before with asset processing, you can have a running system and you can add new types of asset processing to it just on the fly by creating and deploying a new microservice.
Absolutely. So one thing we haven't touched on that is a real benefit is the people side of it, in that, you can have smaller teams that if you've got your general modeling right, can work on a small piece that has... You know, it might have some technical debt, but it's isolated technical debt. You don't need to understand the side effects of the entire enterprise to work on the one service. You can kind of fit that domain problem in your head quite nicely.
One of the common benefits that we're probably not even going to talk about is microservices allow not scalability for performance and for the granular tunability of performance, but scalability of the team and having a team of teams. That was just on my list of benefits to skip over, to be honest, because I think that's .... It's not appropriate for everybody. These patterns, like you said, can be used by anybody,
but certainly, in much bigger organizations. So as long as there is some sensible thinking around how you do that, it certainly lets you scale up your development organization. So the other thing that I wanted to cover, like just on the last thing really, on the benefits, is that microservices, you know, a lot of people think they cause complexity. A lot of people think I'm not going to use microservices because they're going to make things more complex. I've got bad news for those people because applications get complex no matter
what happens. They're going to tend towards complexity in a big organization, a big company that's got a huge customer base. So I don't think you can, like, avoid complexity. Like, you just need to find ways to manage it so that it doesn't really stunt your development process and stunt your release cycle. And I think that's where microservices can really help, is to help you manage that complexity. So it's like taking something that is going to be very,
very complicated and chopping it up into smaller, simpler, manageable parts. I think there are some techniques for that, and I actually want to get into that in a moment. I've got one more question first, but a lot of that is in how you structure your development environment, and how you structure your code base. And I know you've got some really good stuff in the book about it. So I might come back to that in a moment. But let's take it to a slightly dark place first. We've talked about the benefits and we've talked about the hype cycle. I know and you know there's a lot of people out there that have had some fairly expensive failures and some messy lessons. What are the main pitfalls that you see out there?
I think, you know, microservices, like I've said upfront, they're not suitable for every project. They're not suitable for every team or every use case. Like, you've got to think really carefully about, you know, the cost versus the benefits. So, you've got to basically justify where the microservices are going to deliver value. And if they deliver more value than they cost you in terms of investment, maintenance, skilling up, and, you know, management of the tech and stuff like that, then they're going to be useful. But if it's the other way around, if they're going to not deliver a significant value for the effort you have to put in to kind of build that larger distributed architecture, you're going to have a bad time with microservices. And I think,
you know, that's a big part of it. Like, microservices is not just for scaling up performance and development teams, the difficulty level scales as well. So, what I'm sort of liking to tell people now is that, you know, if you're having problems with your monolith and you think that microservices is going to solve that for you, or microservices is going to solve all those problems, it's not. Whatever problems you have with your monolith will be magnified. If you can't successfully, quickly, iteratively deploy and ship your monolithic app, you're not going to be able to do it with 20, or 30, or 50. You gotta solve your hygiene first. I think the experience of probably a lot of people, which results in microservices getting a bit of a bad name is that, whatever problems they have just get scaled up by however many microservices they have. There are ways to do it where you don't have those problems, but you kind
of need to solve those problems early. You know, you can't wait till you've got 100 microservices to start solving, automated testing, automated deployment problems. You've got to have a really nice process, a really nice refined pipeline. And it's got to be rock solid as well. So, you know, no flaky build processes, no flaky deployments. Those things can be rock solid.
But I think if they're an afterthought, I think if you wait till they're a problem to address them, then that's when you've got a huge problem that's going to hurt a lot just to be able to solve that problem. I think any complex system is going to be hard. It's going to be hard to manage a complex monolith, you know, as it grows bigger and more complicated, and has more people kind of touching it. So any development process, any project can be hard. It can be complex to get around. I think microservices can help, but, you know, only really if you've got your house in order, like first. And of course, with distributed applications, you have new problems as well. There's a trend towards making systems more observable these days, and being able to better understand how things are talking to each other and how messages are passing through systems. I think that's a really good trend. It's still something that we're working on as a community. But that kind of like understanding what your application is doing as a whole and
debugging the whole, obviously it scales up with microservices. I think that's a work in progress still. There's still room for improvement there. It's absolutely essential. And you are right. The advances in, observability and tracing, distributed tracing tooling is absolutely essential now to do this. You know, the days of grepping through log files on servers in a distributed environment is just not... You know, that will let you down when you need it. For sure. Any other major pitfalls that you see out there? I mean, some of the ones that I've seen are... The two really that I see a lot
of are relying on shared databases. And the other one is making things very synchronously chatty. There's a few things they were touching on as well. I'd sort of put that in a slightly different category. But the whole shared database thing is, like, you know, when you create services, you want them to be loosely coupled, right? You don't want them to be sharing details that they shouldn't be sharing. And that's how you can kind of build a system where it's, you know, easier to change one
thing, but not have ripple effects, like, across the system through the different services. So, you know, loose coupling, like having them know as little about each other as possible, obviously that's completely destroyed by sharing a database. I would say the cardinal sin of microservices is to share the database because that completely voids any benefit you'll get from microservices. I think there's a scalability issue there. Also you get a similar problem from
using too much synchronous communication between services. You get this kind of fragile system where if one part of it breaks, the rest of it breaks. I think it's what they call a distributed monolith. Have you heard that one? Absolutely. Being too synchronous, sharing a database, deploying everything in lockstep, like, there's plenty of ways to do microservices wrong. Those two, the synchronous and sharing leads to you having to deploy things in a very specific way.
One of the primary advantages of microservices is that you can deploy them independently. You can have separate teams working on them and parts of the system can have kind of small deployments of microservices. And you don't have these big bang releases that you get with a monolith. But if you've got this kind of what we're talking about, this sort of distributed monolith, where it's a distributed system, but because of, you know, the way you've structured it, like, there's so much that depends on everything else. You get these kinds of... I wouldn't be surprised if you kind of fall back to these big bang releases. Everything either works or it doesn't. And you're removing benefits like, you know, the reliability benefits and the fault isolation. That's right. The fault tolerance goes out the window. So yeah, I mean...
I've seen as well, like, what I'm thinking is an anti-pattern. And I think it's all in the name, the name microservices I think it's just kind of wrong. Because people don't really kind of get it. It's more about kind of some sort of logical single responsibility principle for the business, like, not the technological concerns. And so the natural thing is, oh, it's gotta be micro, right? It's gotta be micro. So make it as small as you can. But that often leads to these systems where the services are too small. You've talked about this. I've heard you talk about, you know,
services being too chatty. And I love the way you said that when you did. There's a massive network cost to pay for these things talking to each other. The system is more complicated than it needs to be. It's more complex than it needs to be. When you have that, when you have code or a system that is more complicated than it needs to be, more complex than it needs to be, then you're putting a larger cognitive load on the people that need to kind of understand it, and move it forward. So it's like a cognitive tax really, I think there's somewhere in between, like, really small and too big. There's like a happy medium somewhere in between. And I think where people go wrong is they get so excited about we're going to do microservices now, they forget modeling. There are some fundamentals and some basics, will never
go away no matter what the stack. And that is, you need to learn how to model your software, and you need to learn how to model your domain. Far too many people get caught up in some new fad. And they really forget about those fundamentals. Because all that is described by modeling.
It needs to be modeled around the business, you know, and what the customers actually need from the software. Rather than around technical concerns. Obviously there's going to be a ton of technical concerns that you need to deal with. But I mean, you have to come to that really through that process of modeling and to understand, you know, how are you going to kind of divide up the system the way it's going to be divided up. This has always been the case. This is not a new problem. It's just that we repeat the same mistakes with every kind of technological evolutionary step.
But the new thing with microservices is that it's scaled up to a much bigger proportion. Absolutely. So how do we do it right? In the book, I know you've said, look, here's a lot of the ways to do it. What are some of your three or five tips on how we do this right? There's a bunch of boring stuff here. We probably don't need to talk about, about, you know, getting development right generally. One thing I'm a big believer in is being able to test things
locally,before testing in dev, before testing in staging, before rolling it out to production. It's like a hierarchy of testing where it's so much easier to test locally if you can get a setup for it, that you're going to be a lot more effective if you can work locally first, before you get your PRs into dev or into the next staging after that, or QA, or whatever you call it. So I think people have forgotten that. I think people have forgotten that, you know, we can run these systems locally,
at least... You know, if it's a huge system, we might have to run it in some cut down form. We're running it on our local development computer. So there are ways to do that. But I think in general, like, people think it's an effective process to work in the cloud. And maybe it's just because we're in Australia and kind of maybe used to bad connections to the cloud. Like, that we care more about it. I don't know. Like, I certainly care about it anyway. What else? So, getting microservices right. I mean, like, I've talked about this already, but it's like, you know,
do they deliver the value that kind of outweighs their costs? Because if not, then you're guaranteed to have a bad time. And then next, you need to kind of look at your your skills and experience. Like, if you're learning how to do distributed applications and microservices while building your first production application, to learn these things, you have to make mistakes and then correct from it. Right? So you have to go through a series of
mistakes and failures. And that's not a bad thing. That's how we learn. That's how we should learn. And we shouldn't be afraid of making mistakes or failing, but it can be really hard to do that from nothing, from having no experience to actually doing that on a production application. This is where we need to bring in agile principles, right? We want to fail fast and learn from that, and iterate, and adapt as we learn, you know.
I'm talking more about, I guess, as a personal developer, like... And this is where my kind of book can help. It can help you work through some of those early technical choices and mistakes, and plot a route to production. I mean, at the end of the day, it's about how quickly you can deliver results for the customer that works for the business? And so it's about plotting a course through the technology to reach that point of success of delivering features to customers quickly. But that's a minefield, right? Walking through the design patterns, and the best
practices, and the silver bullets. I'm not just talking about microservices here, but, you know, walking through that technological landscape is a minefield that can, you know, take your legs off if you step in one of them. And again, that's what my book is for. I sort of learned the hard way and this is the book that I wanted myself, but it didn't exist. So there are plenty of
theoretical books on building with microservices. And they're great. And I've read a bunch of those and I love them. But they don't give you that practical guidance of how you kind of navigate this treacherous pathway to actually being… To get the first one into production. Well, it's sort of about being technically competent because I mean, there is...the bar of skills are just much higher for microservices and, you know... Well, that's a lot more. I mean, just in the title of your book, you've got Docker,
Kubernetes, GitHub, Actions and Terraform. That's a lot of pieces. It used to just be, you wrote code, it might've gone to a build server and then somebody put it on a server. Now, you know, there's a whole ecosystem of tooling that you need to understand and make choices around how you get your app into production. Like crossing that gap, crossing the chasm is the hardest bit. But once you get some experience and once you start to...you know, you'll build
your own toolkit of recipes of how to do this, that, and the other. And the book that I've written is full of my recipes, my recipes that I've used in production. And I point out early in the book that it's just my recipes. Once you've gone down that road for a couple of years or more, you'll have your own complete set of recipes for all the kinds of things that work for your business, and your customers, and your team. And your aesthetic, your sense of style, and your taste as well. So I mean, other than buying your book and reading your book, what should people do next if they're interested in working in microservices, in their company, in their team, and they haven't done it yet? Read the book? Yes.
What else? What's the next step? I mean, if you're just learning by yourself, then, you know, practice building something. So that way, you can kind of have the time, and the scope, and you can make mistakes and not be under the pressure of actually having to kind of release things to production. I'm always a big fan of having hobby projects. I've learned more from hobby projects than I have ever learned on the job. I constantly have a hobby project on the go. I use it to teach myself, like, the latest things, the latest ways of working, and tools, and stuff like that. Then sometimes write books about it. But all that learning sort of goes
into what I do day to day in production as well. It's just the day to day stuff is...it's gotta be a lot safer. You can't take as many risks of learning new things and bringing new technologies into play when you're working on a job . But if you do your own stuff on the side, you can take those risks and make really bad mistakes, and have really good learning from it. That's right. But in terms of bringing microservices into a company, I'd say, tread very carefully because the reality is, we have a lot of choice between monolith and microservices. Also even further, you could say that functions as a service are like a step on the spectrum beyond microservices.
There's a lot of choice there. The reality is that most production applications in the world, they're not conforming to what you think is an extreme or like, what you think isn't ideal for what microservices Reality is never pretty. It's always a mess somewhere in the middle. But I think the tools in the book can really help like
those situations where... Tame the mess. Tame the mess, tame the distributed system, be set up to appreciate that it's okay to be in the middle and have, you know, services of different sizes. Some are small, some are big, there might be a couple of monoliths in there. I actually call it the hybrid model, where you have like a monolith, or a couple of monoliths, and then a constellation of little microservices around the edge. I've noticed a few people starting to talk about this as well,
about, you know, it's not about which end of the spectrum you're at. You know, it's okay to be somewhere in the middle. And be in a position where you can leverage the best of both worlds, basically. There's a lot of convenience about using a monolith. It's all there. It's all together. It can all be tested at once. You've got access to everything without having to do kind of asynchronous messages or rest API calls. It's a lot simpler to work with. But then,
if you're set up with something like Kubernetes, you can easily pull bits out of that and make… Just pieces off as you need to. As you need to, as a business need arises for it. I think you're set for hopefully success, if you can position yourself to not care about these religious wars of monolith versus microservices, and actually take whatever the advantage is, whatever the benefits you can from, you know, every possible situation. That's why I think actually the word microservices has probably outlived its usefulness. I've talked about this before on different podcasts that, you know, it should be called right size services. It's like, whatever is right for your situation, there's no microservices police out there that are going to come and look at your code and tell you your services aren't small enough. So stop worrying about that.
That's right. And there's a lot of those terms that just get so much baggage attached to them that they lose their usefulness. Right. So look, to wrap up, what's next for you? Are you working on new books? Any big projects in the works? What's next? I'm currently finishing my third full book, which is called "Rapid Full Stack Development." So it's kind of a bit like how bootstrapping microservices, sort of cross cuts the world of distributed development. I'm trying to write a book now that basically cross cuts the world of...where the app application development, front to back, and also talks a bit about using electrons for desktop applications and capacitors for mobile applications, using web technologies. But it's not really about what you're building, it's about how to be more
effective and more productive at doing it. So that's "Rapid Full Stack Development." That's what I'm finishing up at the moment. It's available now, like, for a discount. So if you want to get on board, like a couple of chapters before it's finished, please search for "Rapid Full Stack Development." And one thing I'm toying with the idea of doing is making a video course from
bootstrapping microservices. So, if anyone is interested in how the book might translate into a video course, like, a very hands-on practical thing, just like the book, then please take a look at bootstrapping-microservices.com. Sign up to the email list and you know, we can chat about how I can basically help you understand how to build distributed systems.
Amazing. All right. Well, I think that's all we've got time for today. So look, thanks very much, Ashley Davis. That was some really interesting chat. And good luck with the second edition of the book. It's amazing. Thank you so much.
2024-08-28 02:21
