Check out Ashley Davis's book 📖
Bootstrapping Microservices with Docker, Kubernetes, and Terraform | http://mng.bz/YGge
To save 40% on Dane's book use the DISCOUNT CODE ⭐
What are microservices? Why and when to use them? How to build microservices with Node.js? In this video, Ashley provides answers to all these questions, and more.
📚📚📚 Bootstrapping Microservices with Docker, Kubernetes, and Terraform | http://mng.bz/YGge To save 40% off this book use discount code: watchdavis40
About the book: In Bootstrapping Microservices with Docker, Kubernetes, and Terraform, author Ashley Davis lays out a comprehensive approach to building microservices. You’ll start with a simple design and work layer-by-layer until you’ve created your own video streaming application. As you go, you’ll learn to configure cloud infrastructure with Terraform, package microservices using Docker, and deploy your finished project to a Kubernetes cluster.
So if you'd like to see more of what I show you today and in more detail and with me guiding you through the process, you can definitely find that in my book, Bootstrapping Microservices. Before we get into the theory, Erie, I'm just going to give you a little bit of a demo, a little bit of a taster of where we're going to get to at the end of this. So the end result of this is we want to be able to kind of update our infrastructure through code. We want to do that basically by pushing code to our hosted code repository, having our continuous delivery pipeline automatically kick in and do the work for us. And I'm just going to show you what that looks like. So I've prepared some example code for today's demonstration. You can find it on Bitbucket Bootstrapping Microservices. Bit Bucket is a little bit like GitHub. It's a place where you can host your code repositories. I've got two example code repositories here, example one and example two. Originally, I was going to go through both of these code examples, but we're probably not going to have time in this live stream. So today we're just going to be looking at example one, which is an example of using TerraForm to build a Kubernetes cluster on Microsoft Azure. And then example two is an example of how to build and deploy a microservice to that Kubernetes cluster. So we're mostly just going to be looking at example one today. I'm just going to drill down to that. So this is a code repository. You can see here a bunch of example code. Anything ending in a TF file is a TerraForm code file. We're going to be looking at the code in there a little bit later, and I'm going to be explaining how it fits together and what it means. For the moment, I just want to show you the end product, which is the continuous delivery pipeline in action. If I click on the pipelines tab here, you can see all the continuous delivery pipelines that have been invoked for this code repository. Obviously, there's been quite a few because I've been practicing for this demo. Now, this code repo, it's just like super simple. It's not a complicated infrastructure, but it's good for learning and it's good for practicing and it's good for demonstrations like this. So at the start of this, back on the original page, on the first invocation of this continuous delivery pipeline, that was where the original Kubernetes cluster was created. So what I have now here is an existing Kubernetes cluster, and what I want to do is I want to update it through code. I've already got a local copy of the code which I've opened in Visual Studio code here. This file is Kubernetescluster TF. By the way, if you do want to follow on, you probably won't be able to keep up with me live. But you can get this code repository later. You can clone this or you can just download this directly from here and you can follow along. Let's just go back into that pipeline stab. I have to be in the pipelines tab just to show you what it's going to do. So back to the code. This is the configuration of my Kubernetes cluster. I've got it set up with a node pool that just has one virtual machine. So this is a Kubernetes cluster that's powered by one virtual machine, not a very big cluster. We want to make this a little bit more powerful. We can change the node count to two. I'm just going to save that now. I'll flip back to my terminal here. I'm actually running this on Windows Ten today, but all the commands I'm going to show you will work on Linux and macOS. I've actually got Git bash installed and that comes with a whole bunch of Linux commands that you can use under Windows terminal. So it's a really nice setup and it works cross platform. So you can see here I've done an LS. You can see all the TerraForm files that I've got. The one that I just changed is this Kubernetes Clusters TF. Now what I'm going to do, I'm using Gift for version control. I'm just going to see what I've changed. I've changed one thing like a good developer on any team, I'm going to dip my change just to make sure that I'm not breaking anything. So you can see here that I've changed the node count from one to two. I'm going to commit this change, put a nice commit comment and now I'm going to push this to my hosted code repository. Now what you'll see now is that this is going to initiate a pipeline. So you can see we've got a pipeline pending here and it's got my commit comment. Scaling up the clusters as who knows? Now it's in progress. We can drill down into this and see what it's doing. You can see here it's running a shell script called Deploy. Sh. This is something that's important. I'm going to show you this shell script a bit later. You can see here in the output of the shell script that it's running TerraForm. And indeed it's actually now talking to Azure. So you can see down here at the bottom of the output, it's talking to Azure, it's making changes to my live Kubernetes cluster and it's updating the cluster for me by the time it's finished. We're not going to wait for it to finish. We're just going to get on with some theory and then the full demonstration. This is just really a taster to show you what infrastructure as code is all about. You make a change to your code, you push the change and it automatically updates your infrastructure for you. So I'm just going to leave that running. So that was just a taste of where we're headed today and where we're going to get to. By the end of this, we're going to be looking at obviously, the theme of this talk is infrastructure as code. We're going to understand what it is, where you would want to use it, and how to use it. We'll be using TerraForm as the tool and the configuration language to implement infrastructure as code. And I'll be giving you a live demonstration of how we can run TerraForm locally to create infrastructure in the cloud, and then how we wire that into our continuous delivery pipeline to then put it on automatic before diving into infrastructure as code. Let's briefly take a step back and understand what I mean when I say infrastructure. Infrastructure is simply the platform on which we run our application. For this particular example, the application is one that I built using microservices, because that's what I do. That's what I wrote my book about. It doesn't have to be this way, of course. This is just one particular example. You can see in the diagram that we have a bunch of microservices running on a Kubernetes cluster which is hosted in the cloud. You can see off to the left that a user is interacting with our application through a front end in a web browser. This talk, it's not about front end stuff at all. You could replace that web browser there with a mobile application. It doesn't really matter how the front end is delivered. What I'm talking about today is back, and it's completely servicide. Nothing to do with the front end. But typically the application that you put on your infrastructure is going to be accessible through a web browser or through a mobile app or maybe through a native desktop application. Even the infrastructure in this example is running on a Kubernetes cluster, and that's the computing platform that our application is running on. So Kubernetes is an orchestration platform that allows you to run containers, and I use it for my projects to run microservices. If you want to build an application like this, the first question you need to ask is how are you going to build the infrastructure? How are we going to build this Kubernetes cluster? Now, we could build the infrastructure manually. And what I mean is that we could go into the web portal for our favorite cloud vendor, and we could create a Kubernetes cluster using the GUI, or we could also build it using the command line tools from our cloud vendor. That's a really great way of doing it as well. And in fact, if you want to try doing this manually using commandline tools, I have a blog post for that if you go to my blog. Actually, I've already got this open. My blog is the Data Wrangler. If you scroll down a bit, you can see here the quick guide to creating a Kubernetes cluster. If you drill through that, it will show you the quickest sequence of steps to manually create a Kubernetes cluster on Microsoft as you are using the command line. Getting a little bit sidetracked we'll get back to the slides It's perfectly okay to start a new application by manually creating your infrastructure. Building infrastructure manually. It's a good way to learn. It's a good way to practice your skills. It might also be all that you need. Like if you're building a small application which has a small infrastructure that isn't going to change much in the future, then it's completely okay for you to do that manually. If you are building that small application and you don't see it as changing, you don't really need to automate your deployments. Most infrastructures aren't going to be like that. Of course, in the modern world with modern applications and there's a bunch of problems with manual deployment. As you might imagine, the applications we're building these days are growing ever more complex, and as a result, the deployment is getting more and more complicated as well. And the more complicated you make the deployment, the more potential there is for human error. The risk of error means that the manual process isn't very repeatable. It's also very labor intensive. So if you need to do multiple deployments and you have an actual person going through all the steps all the time, it's going to be pretty time consuming, not to mention boring. And if it's boring, the person likely won't be paying as much attention as they should. And that further increases the risk of mistakes. Ultimately, manual deployment just isn't very scalable. As your infrastructure grows bigger and more complicated, manual deployment is just going to work less well for you. Of course, the cure for manual deployment is, of course, to automate it. We'd like to make our deployments reliable. We can only do that by taking the human out of the equation. Automation also makes our deployments more efficient. And why would we want to pay a human to spend hours doing something that a machine can do in minutes? Automation also makes the process repeatable. And that means we can make multiple copies of our infrastructure if we need to. That can be useful for performance and fault tolerance. And being able to make new copies of our infrastructure on demand is really useful for safely releasing new updates to our customers. It means that we can employ a technique called blue green deployment that utilizes two copies of our infrastructure. It's also useful for us to be able to make multiple environments. So often on a software development team, we find it useful to be able to create multiple instances of our infrastructure. And for example, we'd like to have an instance for development, a separate instance for testing, and a completely separate instance for our customers so that they can use only the tested version of the application. Having automated deployment means that as a last resort, if we have infrastructure that's not working so badly that we can't fix it we can simply nuke it and run our scripts again to recreate it. Now, that is completely a last resort option, but it's something that automated deployment allows, and it's quite amazing, really. In the end, we need automated deployment for scalability the amount of time it takes a human to build the infrastructure, and it should no longer be a constraint for application design. The software industry has moved well beyond that. Now. We can automate our deployments, and that means we can deploy whenever and wherever we like and as often as we like without having to wait for a human to put in the hours that would have been required before we automated it. So that brings us to infrastructure as code. And to make a long story short, infrastructure as code. It's simply the technique where we manage infrastructure creation and updating and management through code instead of creating that infrastructure manually. In a moment, I'm going to explain a little bit deeper about infrastructure as code, but for the moment, let's just take a look at this diagram and we'll understand what I'm going to demonstrate today. You can see the developer on the left here is using TerraForm locally on their development workstation. They're invoking the TerraForm tool to run their infrastructure code, and that code is reaching out to the cloud to create and update their cloud based infrastructure. In the demonstration today, I'm going to be showing how to use TerraForm to create a Kubernetes cluster on a zero. And the process in this diagram, actually, it shows you the way that I prototype real infrastructure. I write and test code on my local computer, and that code is then making changes to the infrastructure in the cloud. This diagram shows where we'd like to end up with infrastructure as code. You can see the developer here on the left is writing and testing his code locally, and they're committing code to their local Git repository. When they push code, their hosted repository is automatically triggering the continuous delivery pipeline. In this example, we're using TerraForm inside the pipeline. It's running the code, and it's creating or updating the infrastructure in the cloud. Terraform is not the only tool you could use here. You could build your own tools. There are other tools you could use. This diagram shows a simplified pipeline. There's no checks or balances in it. You can't see any automated tests here. Typically, in a real pipeline, you'd have those stages, those things, and you'd have multiple stages, possibly for manual approvals and whatever other testing, automated compliance testing, automated security testing, whatever other testing you want to have in your pipeline to be sure that safe code is getting rolled out to production. So I've really simplified this for today's demonstration. It doesn't include any of that, and it's a lot simpler than it could be. This pipeline makes for a complete automated flow of infrastructure changes from the developer all the way through into production, and hopefully you can see here why it's called infrastructure as code. It's the act of changing code that triggers the pipeline. That is what makes changes to our infrastructure. Once we have a pipeline like this in place, we are now updating our infrastructure through code changes. So I'm just going to get into a little bit more detail now about infrastructure as code. We'll dive a bit deeper. So infrastructure as code, it's where we manage our infrastructure through code. Not only is our code the source of truth about our application, but it can become now the source of truth about our infrastructure. And rather than being like a manual or physical task, the act of creating infrastructure has been converted to a coding task. So anything that's useful for coding suddenly becomes useful for building infrastructure. That's important because we've already amassed a large toolkit of techniques for coding that we can now apply to creating infrastructure. Infrastructure as code usually means that the management of our infrastructure is automated. And this is a huge win for any modern business that is trying to scale up. Because what we want to do, we want to grow our application to support our expanding customer base. And we can't scale to any large extent without automation. Infrastructure as code, then it provides part of the basis that basically paves the way for massive scalability. Why should we use infrastructure as code? It's all about the automation. We're automating the creation and maintenance of our infrastructure, and that has many positive benefits. Automation can really empower a small team. It's what allows us to achieve big scalability with only a small workforce. It reduces the amount of manual work we must do, and this, in turn, frees up time for us to work on the features that are important to our customers. It reduces the risk of human error. Again, this frees up time because errors usually involve quite a bit of time to correct. The fact is that automation makes deployments cheaper, and this is what makes small deployments more cost effective. And actually, this is one of the reasons why micro services have exploded in popularity recently because they have become cost effective to build and run. So all this serves basically to reduce the risk that anyone deployment will fail. And infrastructure as code, it's making our updates more reliable. The process is faster and more repeatable. And the knock on effect is that we can safely increase the frequency of our deployments, and thus we get more features more quickly into the hands of customers. So we're increasing the pace at which we roll out features. That's, of course, the reason why we're doing any of this. We are delivering application features to our customers, and anything that streamlines that makes it safer makes it faster. That's going to give our business a competitive advantage because our infrastructure code is the source of truth about our application. The code is acting like a kind of a documentation. It documents the way our application or our infrastructure is supposed to work. And it not only describes it, but also running the code is what brings our infrastructure up to date. So we know that our documentation or this infrastructure code as documentation is always going to be up to date because it reflects what we have in production. Another big reason to use infrastructure as code is because it makes a coding activity out of infrastructure management. Because it becomes a coding task. We apply all the tools that we normally use while coding, things like automated testing. We can use that time and time again to automatically validate that our code is working correctly. We can use version control to manage versioning of our infrastructure. As developers, we often take version control for granted, but it's amazing, really. It allows for a kind of time travel, but we can go back into the past using version control and we can reconstitute our infrastructure as it was in the past. That's really useful if you want to reproduce a bug or a feature or something like that from the past, you can also use version control to roll back changes, maybe broken changes, roll back to a working version. We can use branches and labels to manage different versions of our infrastructure. And one of my favorite things actually is to use a test branch that deploys to the test infrastructure, a Dev branch that deploys to the Dev infrastructure, and a prod branch that deploys to the production infrastructure. So we can manage multiple instances of our infrastructure using version control. And of course, infrastructure as code. It's a key enabler for continuous delivery, and that is what allows us to automate the deployment of our infrastructure. So the question now is how do we make use of it? As with anything in our industry, there's so many different ways we could do this. There are many different tools we could use. And of course, today I'm focusing on TerraForm, which is one tool we can use for infrastructure as code. We'd like to have our infrastructure defined in code so that we can put it on automatic in a continuous delivery pipeline. We need to trigger it using a version control system. Of course, I'm going to be using Git. That's the modern industry standard for version control. All that we need to do now is choose a cloud vendor. I'm going to be using Azure. I'll tell you more about that in a second. What TerraForm? Why would you want to use TerraForm for something like this? Terraform makes it easy to reliably and repeatedly configure cloud infrastructure. It's a declarative language, which means you describe what you want rather than how you want it built. So it's much easier to code than, say, an imperative infrastructure as code language. It's incredibly flexible. Its functionality is extended through plug in providers, and that means it's multi vendor. It works with any cloud. So you can use TerraForm with I'm going to be showing today how we use it with Azure. You can use it with AWS, you can use it with Google cloud and some of the smaller vendors as well. The cliff form also supports the ability to configure the internals of your Kubernetes cluster so you can roll out containers. You can deploy microservices to your cluster using TerraForm. Terraform is really smart so as I said it's declarative so it figures out how to create what you're asking for the sequence of steps to go through to create that and it gets even smarter because once you've got existing infrastructure and you want to make changes to it you don't have to do any kind of like step by step migration of your infrastructure. You just change your code so it is the way you want it to be. Next you run TerraForm, TerraForm figures out what is the difference, what is the Delta between what it was and what you want it to be and it applies those changes. To top it all off TerraForm is open source so if you wanted to you could actually get into that code base and contribute to it. As I mentioned TerraForm works with any cloud vendor but to demonstrate I need to pick a side. I like Azure, I've got a fair bit of experience with it so that's why I'm going with Azure. It's one of the big three of course so it's one of the ones that you would pick but I find as you are pretty beginner friendly it's pretty easy to use. I think that's how Microsoft are really competing against AWS is in Usability and when you sign up for the first time you get $200 starting credit so you've got a bit of time to experiment before you start spending cash. Now you're going to make sure you clean up all your cloud resources after you've done your experimentation of course because you don't want to burn your credit and after that you don't want to burn your cash. I'm going to be showing today how to deploy a Kubernetes cluster on as you are using terrace.