Video details

Kubernetes Security 101 - Best Practices


This workshop aims to give an overview about how Kubernetes works and provide some best practices to secure your cluster whenever you are deploying a new cluster on your own or via managed services such as GKE, EKS or AKS. We are going to cover everything from the Control Plane or the Master Node, starting with the API server, including etcd, RBAC and network policies. Then, we’ll cover the worker nodes, kubelet, audit logs and pods best practices. We'll talk about the CIS Benchmarks for Kubernetes and the default configurations you need to worry about when deploying a new cluster. We'll show how to use RBAC and assign roles and permissions to your cluster users. We'll demonstrate how to enable audit logs for better visibility and later we'll set up some network policies to avoid communication between pods and prevent any lateral movement from attackers. Are you starting to use Kubernetes for container orchestration? Do you need guidelines on how to start securing Kubernetes in your organization? Do you want to find a way to increase the protection of your Kubernetes clusters without increasing the complexity of the infrastructure? Do you need to use Kubernetes clusters in a safe, efficient and affordable way? Everything in a practical way with a focus on security best practices? Then this is the workshop for you!
PUBLICATION PERMISSIONS: Original video was published with the Creative Commons Attribution license (reuse allowed). Link:


The agenda for today is what is Kubernetes, right. So I'm going to assume that you either heard about Kubernetes or at least have seen someone or an organization that uses Kubernetes. But in this workshop, I'm going to assume that you've never used Kubernetes. So the idea is to start from scratch. So what we're going to do the way that this workshop works is we're going to set up an environment on AWS. That's why there was a prerequisite of having an AWS account. And we're going to use Cloud Nine, which is like a Vert developer environment for AWS. So we're going to deploy Cloud Nine instance and use that to deploy our cluster. Right. So we're going to understand what is Kubernetes, the Kubernetes architecture. I'm going to talk about all the small components, the two main components, the control plane and the worker nodes, and then the H little component that's inside those major ones. As Job said, we're going to do some remodeling, right. I don't know if you heard. But this year, in April this year, the Miter released the attack framework, the attack matrix for containers. Right. And we were part of this work with Miter. So we partnered with Miter when they called the community for help, and we provided some data regarding our honey pots. We have some Docker and crates planets that we use to analyze attack data on those environments. So we're going to talk about that as well. It's then later we got to talk about after we set up our environment on AWS, we set up a cluster. We're going to deploy application containerized application. There a vulnerable application. So we're going to attack it. Right. So we're going to understand what's going on. How can an attacker compromised cluster one of the many ways that that's possible. And, of course, this is a misconfigured cluster. Right. So that you understand the fundamentals of why it's important to keep your Kubernetes cluster well configured and secured. And if we have time, then we're going to talk about the same in Verne. Right. So what are the main issues on that? You can order the best practice, right. Either, regarding the server and the whole process of deploying a container, generating a container right from scratch and deploying it into Kubernetes cluster. And there is many things that we can talk about here. So yeah, I hope you enjoy and feel free to ask questions. I know that people are monitoring those questions and they can send them over to me. So if you're struggling, if you're trying to follow along and something's not working for you, don't worry. Try a few few questions and we can make this very interactive. Okay. So before we start, I just want to say that since I started working with Kubernetes last year, I created this kind of GitHub project, the awesome Kubernetes security list. And this project has has, like, a bunch of links and information that I use myself to learn Kubernetes and understand and has a lot of stuff about Kubernetes security. So let me sure that here with you so that I can take a look as well. So first I started with that as like, kind of my own thing when I was studying saving links on a text file for myself. But I said no, maybe some people, other people can benefit from that as well. And that's why I did this GitHub repo, which has almost almost a thousand stars already. Right. And I created that in October last year. So it hasn't been even over a year yet and has a lot of information. I know there's some stuff that I need to update and update it soon. So. So yeah. Feel free to start to fork it. And if you have any other links that are either related to Kubernetes Kubernetes security, you can submit a PR to me, and then I'll take a look and add to the list as well. There's a lot of information here. Okay. So I'll give an overview about what is Kubernetes so that you can understand. And then we go to set up the environment and I'll continue the slides while the environment is setting up because it takes a few minutes to set up the environment on. Aws. So let's do that first. So what is Kubernetes? So what is your understanding of Kubernetes if you have seen it if you have used it, right. Yes. Kubernetes is an open source system for automating deployment, scaling, and management of contrarie applications. Right. But what does that mean if you remember when Docker was released in 2013, right. There was a lot of type about using Docker and using containers and containerizing. Everything. Right. But quickly we learned that by doing that, there was a lot of increases the overhead of management of those applications. Right. Because when Bucker was released, there was no way for you to easily manage multiple containers running on multiple servers. Right. And so Kubernetes was created as an idea to help manage those containerized applications. So it's a container orchestrator, as we say. Right. And they just released the latest version, which was 122. And it has before until last year it was. New versions were released four times a year. So every quarter. And I think now it's three times a year. So it gets updated very quickly. So you need to follow along and keep yourself updated so that you also can keep your closer updated. Right. So some of quick facts and fun facts about Bernetti Her it was created by Google in 2014, and it's based on internal projects from Google, which are called Board and Omega, which are closed source, and they decided to. Google has been used containerized applications for many years now, and that's why they work on this project. Kubernetes means it's a word from Greek, which means helmsmen. Right. And Helmsman is like the person who kind of drives the the ship, a big ship in kind of an illusion to Docker, which are the containers and the person kind of steering the wheel of the ship there. That's the helmsman, right. Sometimes can be the captain. Sometimes it's another person. So that's why. And that's another reason why the icon for Kubernetes is a hell. That's what we call this kind of steering wheel of a ship. Right. It's a helm. Yeah. Another fun fact, or Kubernetes. It's also called a right as. And the reason for that is just because there is eight letters between the K and the S of Kubernetes, right. If you count the letters there in between the Bernet, it's eight letters. So that's why an abbreviation of that. That's the way that they do. It another thing Kubernetes developed in Golang as same as Docker and many other Cloud native applications as well. So. Yeah. Okay. Let's move on. Why Kubernetes? Right. So since it's important to understand what Bernet is, it's also important to understand why Kubernetes like, should you use Kubernetes in your organization? You don't want to go go to your company on Monday and say, no, we should move everything to Kubernetes because Kubernetes is the next big thing, right? There is a lot of a lot of management required to run Kubernetes clusters. Right. And as I said, I think personally, from my perspective, it's a very complex system. It's hard to understand, and especially if you don't have the background of what Kubernetes do behind the curtains. Right. So once you understand that, that gets easier to know what he's doing, but it's like you need to have that background, right. So, yeah. Just kind of a comic here from Dilbert, right. It's not everything that needs to be a Kubernetes doesn't mean like that. You have very few people that really, really understand Kubernetes properly, especially even Kubernetes security. If you want to start moving your applications to your Kubernetes cluster, then you're going to require not just budget resources to do that, but also skilled people to maintain your closer. Right. And think about that before you go into okay. Yeah. Let's move everything to Kubernetes. It might not be the solution, and it might not be required for your needs and your organization needs. Okay. What is the CNCF, right? The CNCF is the Cloud Native Computing Foundation. It's the sub foundation of the Linux Foundation. It helps in maintaing many different projects from what they can. Is there Cloud Native? Right. So Kubernetes is just one of them. There are many different others that you can see here, and some of them are even part of Kubernetes as TCB and Core DNS, and also help to package Kubernetes applications as well. So there's many different ones. Right. So it doesn't mean just Kubernetes runner talking about Cloud Native CNCF, the Cloud Native Computing Foundation and Cloud Native Security. We're not just talking about Kubernetes, right. This is the URL there at the bottom, the L CCF is if you go there, it's going to show you a lot of different projects that are related to cloud native. Right. And just a quick thing here about what means to be cloud native doesn't mean that the application only runs in the cloud. And we're going to probably going to talk about that in the next few slides. But just just to add to that being cloud native, is there's different characteristics and applications that they have and they're created to run in native cloud environments, but they can also run on premises, their organizations and many different companies that run Kubernetes on premises. Right. Okay. Yeah. So that's the slide here that I said what it means to be cloud native. The definition here is from the CNCF itself and can be found on their GitHub. Basically, as they say to an application to be cloud native, it needs to have this characteristics on the last year seven characteristics and some examples here that you probably are using or you probably heard about containers service matches microservices in general, Immutable infrastructure or infrastructure as code and declarative API. So these are some examples of cloud native applications. Okay. So I'm going to stop here before I go into the architecture and explain all the details. I'm going to share my screen and try to have everyone follow along to set up their environment. Okay. So let me stop sharing and see if it can ensure again 1 second. So. Okay. Can you see my screen? Yeah. Good. So one of the prerequisites of the workshop, as we mentioned, is having an AWS account. Right. So if you're not logged in on your AWS account, please go ahead and log in. And what we're going to do, as I said, like we're going to deploy a Cloud Nine instance, which is this service from AWS, and you're going to find out very soon how it works. It's basically an online version of Vs code. So if you use Vs code before you're going to find it very easy to follow along. And from that the idea of why I'm using this, right. The idea is that I want to make sure that everyone has the same environment. And so I didn't want to create a VM and deploy and send it to you to download and everything. Let's do it online. And using Cloud Nine is a good way to everyone's going to have the same environment if we use the same configuration here and then from there, we can set up the instance to deploy the cluster afterwards once we have the cloud mining set up. And that takes that very quickly. But then we configure the cluster on AWS as well. So let's start here and I'm going to create one from scratch. I have some created here, but I'm going to create a new one. So let's see Cloud Village, right. Workshop. So the name you can give any name that you want. Right. So just remember if you have different Cloud Nine instances, but you can add a description as well. Here you have the settings of the environment of your cloud lining. It's going to deploy an EC two. Right. So the way that Cloud Nine works, it's deploying an EC Two, but it's going to configure it. That's going to work as an online developer developing platform. Right. So it's basically an online vs code that's hosted on AWS, so that you have all your code there if you want, you can use that as well. So yeah, basically it's going to create a news to environment. Right. The idea here you can use the two micro, which is free, free tier if you have free er. Yeah. Just another note here before we move on. There might be some charges because not because of this Cloud Nine, but once we deploy the cluster and EKS and everything, depending on how long you leave the cluster online, there might be some small charges on your credit card on your account. Right. But at the end of the workshop, I'm going to show how to remove everything so that you don't leave anything running and don't get charged a lot of money. Okay. Sounds good. Okay. So here if you're all there, basically, we're going to create a new site instance. It doesn't matter. Like I don't need a large instance for the Cloud Nine is because I'm only dealing with code and executing some commands there. So that's fine. The platform can be Amazon. Looks true. And I can leave this. Everything default here after 30 minutes of idle time. If I'm not using the Cloud Nine instance, it's going to shut down automatically here, so I can leave everything here as it is right by default. And I'm on the right region. I'm on us one. So I'm going to deploy on us on the cluster. I can deploy anywhere, and we're going to see how to do that. But basically for the Cloud Nine is make sure that everyone is on the same region and they have access to the same features. That's the only reason for that. Okay. Here just an overview of the settings that I set up. Basically, everything is default right now. We're going to create a new row for my instance later. But let's leave it at that right now. Okay. So it's creating my environment. It might take a few seconds to minutes. Right. And this is going to be like if you use Vs code before, if you work with coding and programming vs. Code, you're going to find it very interface is very similar. There is a few things that we need to do here once the environment is set up. And so creating an Im role for our instance and also disabling some credentials. So I'm going to show here once the Cloud Nine is completed and deployed. Right. Let me see. See if it's taking a lot of time. Yeah. Let me go back. Yeah. I don't know if we have any questions so far, but feel free to ask them if you're stuck. If you miss something until now, feel free to send the questions over through the form. The Google form. Okay. No questions. Sounds good. Awesome. Thank you. So let me wait here. Okay. Now the Club nine, it is set up as you can see, it's basically an ID platform online. I have a terminal here. I can add new files and new terminals as well. Right. Okay. So what we're going to do? No German. Okay. Is the font good enough for everyone to see? I don't need maybe. Let's leave it. That okay. So this is basically the environment that we're going to use to deploy our cluster from. Right. Okay. So a few things that we need to set up here and I'm going to show you. So it's on the settings here on the top right corner of my interface, a gear icon called preference. Right. And in this year icon, I need to go to AWS settings. And on the AWS settings, I'm going to disable these AWS managed temporary credentials. Right. All I need to do is click here and disable that. And that should be fine. Okay. Everyone got that preferences, go to AWS settings and disable the AWS managed temporary credentials. Good. So after that, let's see. Now we're going to let's create. Now we're going to go back to the AWS console to create to create the IAM role that I need to give to this instance because the Cloud Nine is an easy to instance to be able to have permissions to talk to EKS and create clusters. Yes, the workshop is being recorded and it's going to be shared later as well. So fine. Let me go back to the AWS console here. You open another tab. I think I need to share. Stop sharing this one and share the other one. Sorry about that. Okay. So on AWS console, what we're going to do is create an Im role and please don't do this on your production account, because what we're going to do is create an Im role with administrator permissions, administrator privileges so that we can give can give the cloud dining and permissions to create the cluster and everything. Right. So that we don't have any issues, we don't face any issues. We're going to give this administration role. So I'm going to IAM and let me go back. So what I did was I went to IAM, which is the identity and access management service from AWS. And basically what we're going to do is create a role for that create role. I go here to roles on the left side and create role. And here I select AWS service. That's selected by default is two. Okay. And then I click next in permissions on the permissions it's going to load. And I'm going to give this role administrator access to my account. So be very careful with that. Don't use that in a production environment. We don't want that because that's too permissive. I don't want to give the Im role. This is just for the workshop so that we don't have any issues deploying the cluster. You can add Tags, but that's not needed. And row name in Cloud Nine instance growth. Okay. You can put any name. Just remember that name because you need to attach that to your Cloud nine instance. Okay. So what I did was just repeating what I did was I went to Im. I went to Role and create a new role, selected EC two and gave the policy's administrator access and created a name for it. Right. That's it. Okay. The role has been created. So the next thing that I need to do is go back. I'm going to the EC, tune out and attach that role to my institution as my Cloud nine is two instance. Alright. So I'm going to easy to here. That's why I talk to and to see my machines running. I see that there is an instance running. Make sure that you're on the same region that you deployed your Cloud nine instance. And basically see there is this instance running here. That's why I have the Cloud Nine running as well. So select the instance here the EC two and on Actions Security modify IIM row that we're going to attach that row that we just created to distances. So again, select my instance, go to Actions Security and modify IAM role. And here I have my row that I just created the Cloud nine instance row. And I'm going to give I'm going to attach that to the instance and I'm going to save. Okay, if you got the message successfully attach your role to the instance. That's great. That's all we need so far is anyone behind. Does anyone have any questions? Anyone struggling with something feel free to ask. Okay. So now that I set this up, I need to go back to the Cloud Nine and install a few things on my Cloud Nine instance to be able to deploy the cluster. Right. So some things I'm going to stall. Is it some other packages? So we're going to follow along. Let me share that with you. Let's see if I have it here. Okay. Yes. Let me go back to my Cloud nine is I'm not sharing that. Sorry for this switching here, but I don't want to share any. And it says get information, right? Yes. Okay. So I'm here one of the things that I need to do here on the Cloud Nine. I've attached. Remember if you haven't done the AWS settings disable that as well, just let me show it again the temporary credentials and yeah. So you can see that there is a kind of a directory structure here. It shows your environment. I can have many terminals and all that stuff. That's great. So now we're going to use this project on GitHub to download two files. That's all I need to set up my cluster, basically. And I'm going to stall some package and some tools, but basically those two files, it's all I need to set up my cluster. So let me show you here. We clone and this URL Teresa 101 workshop, if you download that, you can see that there's basically two files there. And I'm going to explain what they are. So and I'm going to increase the font and close this. I don't like this there. Yeah. So you can see that there's two files here. Two Mo files. Right. And if you haven't played with Mo before, that's something that Kubernetes uses a lot to manage to deploy applications and objects on Kubernetes, right. They're all usually emo files that turn into JSON or sent to the Kubernetes API server. And I'm going to explain that soon if you haven't clone this. So the command is de clone. Let me post the whole command there. Clone a shirt with the people participating. And if you clone that, if you have those two files on your cloud nine instance, you're good. Okay. So far you're good. Let's move along. So now this cloud nine instance is brand new. Right. So if I try to use Cubectl, which I'm going to explain some of what it is, but it's basically a CLI to talk to your Kubernetes cluster. So I don't have that right now. I command com that's not installed. So I need to install that. So the first command to install Cubectl here. Let's see. Okay. Let me post that on the chat. So this is the first command here and I'm going to share with you. So basically I'm downloading that from AWS. Let's see if that's done. So now I should have give it permission. Of course. The second command is to allow execution permissions to this binary. And don't worry, I'm going to explain everything once we start deploying the cluster, which is the thing that takes longer. Then we go back to the slide and I'm going to explain some things that I'm doing here if you're not able to understand yet, but you will be able to understand. No worries. Okay. Yeah. So TTL is working. We have that already. Now keep to controls the Kubernetes cluster manager. That's great. Other things that we can install that's going to help you with our interaction with the cluster. So let's start you a couple package that can help you. And those packages are Ju and gas tax and batch completion. Right. So basically what I'm going to do is pseudo stall those packages, right. Because they don't have that on the on the cloud lining stance by default. Great. Now I'm going to configure a batch completion to CTL. So that helps me when I'm managing my class and using command with Kubectl. So basically keep CTL completion bash completion and another one here. Let me post those two commands to the team as well, so they can share with you. Yes. So those are two separate command that you're doing. Let me clear here so that I moved it. Okay. It's better. Okay. Any questions so far? Anyone struggling? Any comments that you don't understand? There's just a couple more things that we're going to do now. We're going to install it. And what is another binary to interact and help you deploy EKS clusters? And what is K is the main Kubernetes services from AWS. Right. So it's like a Kubernetes that's provided to you by AWS and AWS managed set. Every major cloud provider has their own managed Kubernetes services. You have EKS for AWS, you have. As for Azure, you have GKE from Google and many others. Right. Okay. So we're going to install the EKS binary, because that's what's going to use to deploy the cluster once the cluster is applied. Excuse me. We just use Cube CTL, basically. Okay. So we set that up. This is the command. I'm posting that to the team so they can share with you, too. Sorry. Okay. Moving that to the local being and also enabling batch completion for Et, and that's it. Okay. That's working too great. So far, we've only downloaded a Sea package, install Cube CTL and install EKS CTL. That's what the two main binaries that we're going to need to deploy our cluster. People may ask, why don't we need to install on, right. We're not running our cluster here on this instance. We're going to deploy actually one new instance to run our cluster. Right. And the way that EKS works is that on they run, they manage the control plane of the Kubernetes cluster, and we only need to deploy our worker notes. And that's what we're going to deploy. As in situations that's going to work as our worker note to run our applications. Okay. Let's see. No questions so far. Oops download link truck. Sure. Let's see. This is the whole this is the whole link. Maybe if it's truncated. Yeah, that's the whole link. Basically, I'm downloading the bit binary if you want to download that from directly on Kubernetes, the Kubernetes website, which is Kubernetes IO. There is also a Kubectl there on downloading that from AWS from a free on AWS that they provide. Right. That's better. I don't think you're going to have any interest to download from the Kubernetes IO website, but if that doesn't work with the link, please let me know. Okay. So far, we figure just to give you overview of what we've done. If you just joined the workshop, we deploy the cloud nine instance. We configured an Im role for this instance, and then we downloaded the two files from GitHub that we're going to use to set up the cluster, and then we stall Cube CTL and EKS. That's all we did. And don't worry, if you don't understand some of those names and tools. I'm going to explain that as soon as we start deploying our cluster and move back to the slides. And I just don't want to waste too much time. So I want to do the set up first. If people haven't been able to do that, and then we go to the slides while cluster is being deployed. Okay. Let's see here. So basically, what we're going to do is to create this cluster I'm going to use. Let me show the files first. Let me go back to the files. Okay. So the first file that we're going to use that we're going to interact with is this Es cluster demo. Right. This is going to deploy my cluster on AWS. So that's why we install the KS that's going to help me. And here on this fire, you can see the command here already. Right. That's what we're going to do. Basically run this command on the terminal, of course, pointing that to the file that we have and basically deploying a cluster configuration. Right. And this is the name of my cluster. This is the region. So here you may be able to change your region if S is available on the region that you are in. And it's also a good idea, because if we deploy all don't know how many people are following along and deploying clusters as well. But if we all deploy our clusters on the same region, we might run out of resources. So the way that it works on AWS, they have a limited resources for availability zones. Right. So inside the render as different availability zones. And we might have some horse saying that you don't have enough resources to deploy that cluster in that region on that availability zone, because EKS CTL, they're going to choose from this configuration here. They're going to choose which availability zone on us east one region. They want to deploy our cluster. We don't manage that. Of course, we should add that to the configuration file, but I don't want to add a lot of overhead to the configuration. So you don't need to worry about that. And on this cluster, I'm trading a managed node group, and I'm calling that manage node group one, and I'm deploying an instance of the size to two small. Right. And here I can say the minimum size in the maximum size. That means that my cluster can have only one worker node, only one instance and maximum free. Right. So if I need to increase the number of worker nodes on my cluster, I can easily do that. And if I can change that to ten, it doesn't matter what's going to matter. Here is the desired capacity. Right. That's what the cluster is going to start with. If I change that to free, for example, then it's going to deploy three different in two instances of the same size in the node group. And I don't want that. I don't want free. We're only going to use a small web application, so I don't want that. I just want one so that I don't need to pay a lot of money for deploying dis cluster. Right. This is the volume size, the hard drive and some labels. The worker labels, the worker role. And basically that's it. That's all that is on my cluster. So here I'm going to use this command. Let me go to the workshop. Here is my ECS cluster. So I just enter the directory of the after GitHub project that I just downloaded again. It's just if you haven't disease, do you like sure. Okay. Yeah. Maybe the chat let me share the the whole file here. 1 second. Some people are struggling with the commands the first command and then give me a few seconds here we put that drive and making sure that 1 second. Hang on. I should have done that before. Sorry about that. Okay. Okay. Let me share on this file here. I created a Google docs, and I will share the link of the Google docs if all the commands and and then we can share with everyone participating. So then we can check. Let me just one one workshop that quickly. Okay. I'm sharing on the Google docs. So if you can share that link with everyone, everyone can see that link. And these are the main commands that we did to set up Cube, CTL and ETL on the cluster. So if you're having any trouble, some of those are two commands. So remember that step one, step three. So follow that along and check out the commands there. If you're still struggling me after following that, let me know. Please let me know. So I'll move on here and I'll deploy my cluster. And while that's going on, as we're going back to the slides, can help anyone that's struggling or answer any questions, right? Yeah. As I said, I want to create my cluster and to do that, I'm going to use Set to do that. And the command here is a create cluster, and I'm passing a file, which is the one this file here that I have. So this is the command and now add that to the spreadsheet as well so that you can follow along if you get stuck or something. That the spreadsheet so that everyone can see one workshop. I'm adding that to the sorry to the Google docs. Just adding that there so that you can see. Okay, let's see if that's going to work. A live demo or live workshop is always hard. So here I started creating my cluster, and as you can see in curiosity, always doing this job. It's creating a subnet, creates a VPC and then does everything for me. So that makes things easier. Right. This is what takes a little bit longer. You might face some errors here because of resources. If you're doing on the same region, if everyone is doing on us, this one what you need to do, basically is either try on a different region or try again, but change the name of your cluster so that it doesn't. What this is doing. Actually, ETL is using cloud formation to deploy your cluster. As you can see here, it's waiting for the cloud formation stack and all that stuff. And so if you try to deploy and it didn't work, then if you try to deploy again with the same name, it's going to complain to you. There is already some cloud formation template created with the same name. And let's hope we don't have we don't face the issue these issues. I faced that on the previous addition of the workshop that I did. But let's hope not. It seems that no issues so far. So I'm going to leave this running here. I'm going to go back to the slides so that we can continue just the explanation of the Kubernetes architecture. And then once the cluster is deployed, we come back here to deploy the application and now play along with the attacking the application and all that stuff. Okay. Stop screen and share the slides again. Okay. So yeah, we stopped here. We stopped here. Right. The Kubernetes architecture. And so as we can see if you're struggling, feel free to ask questions, I'll stop and answer them. Why? I'm explaining no problem. Or as we can see, there are two main major components here to use the Kubernetes control plane. It used to be called Master node, but for inclusive language, usually we don't call it anymore, even on the new version Kubernetes, it does, I think it does say, but it's being deprecated the name Master node. Right. So I'm going to call it from now on Control Plane. And so this is on the left side here with five smaller components on the right side of the slide. We have free worker nodes, and the worker nodes are the ones that we use to deploy our applications. And today in this workshop, we're deploying only one work, only one in situ that's going to function as the main node for deploying our vulnerable web ification we don't need three. But if we need it, as I showed there, with the Max size and the desired capacity on the YAML file, we could easily change that there. Right. Okay. Awesome. Yeah. I should have thought about the Google Docs on this. Let's look at the left side. First, the Cube API server, the Cube API server here in the middle. We see that's kind of a big thing. It's a main component, one of the main components of the control plane. Every other component is talking to the API server, and you can see from the arrows there. It's not just talking with the components from the control plane, but it's also talking to the worker nodes as well. Right. And the Cube API server, it's basically an API server and Rest API server. If you played with API before that, receive and forward all the communication from all the components of Kubernetes, right. It has basically everything that you type from it. It becomes an API request, an HP request that goes to the API server of your cluster. Right on the bottom here. On the bottom left, we have a CD. A CD is the main database data storage of your cluster. It's a key value store where all the components of Kubernetes Kubernetes cluster is stored there. And so everything gets saved there as as an object, as a key value store. And then Kubernetes tries to check that and tries to create whatever you told it to create that saved on TCP tries to create that on your notes. Okay. And that's something very important here because the the Kubernetes works the way that Kubernetes work. It's called desired state. So Kubernetes is smart is smart enough to see that it is constantly checking if whatever I told you to create, for example, nodes or application or services, whatever. I told you to create their own CCD. It checks on your notes and see. Okay. Is this application run already? Do I have three replicas of that application? Right. If I do. Okay. Good. That's the desired state. If I don't, if whatever is on a CD, whatever configuration is there doesn't match what's running on the cluster, then I need to fix it. Right. They need to match. Right. And so. So let's say I only have one replica of my application and I need three. Kubernetes is going to check them. And it's going to tell the cluster to create two more applications to more containers, right. From actually spots. But it's going to create those to match those in CD. So I need to have three replicas running on my cluster. And it doesn't matter on which worker node depending on your configuration as well. But I need to have that matching. Right. So it's constantly checking. If you check the logs of Kubernetes, it's constantly. There is many different components that's constantly checking to see the desired state. If whatever is on Ted should be reflected on your whole cluster on your work notes. Right. So you can see here that the API server and see, there are two major components Oster Kubernetes clustering. That's why you need to protect those. Then we're going to talk about security, the security part of it and the best practices as well. But be very careful with those components. Right. If an attacker has access to your TC and they can change objects there. Right. Basically, whatever they change to Kubernetes is going to just follow along. Just Kubernetes is going. Okay. If you want me to create deploy a malicious container, I'm going to deploy a malicious container. Right? It's not checking who put that information there. It's only checking the information sed is matching whatever is created on the cluster. Okay. And on the top left here we have the Cube controller manager, and the key controller manages exactly this object that has a bunch of different controllers, the pod controller, the service controller, and other controllers that keep talking to the API server and checking on CD. Is it matching? You have the right number of pods? We have five here, but we need ten. Right. And so it's constantly checking that to specific controller is doing that. The cloud controller manager on the top right of the control plane is the one that talks to cloud providers. That's the reason why Kubernetes can work with different cloud providers. You have the cloud controller manager. If you want to use disk volumes or load balancers, Kubernetes doesn't have those objects, doesn't have those services. So you need to use the one provided from your cloud provider, and we're going to do that on EKS date. We're going to deploy a load balancer to expose our application to the intern. Okay, good. And on the bottom right is the Cub scheduler. The keep scheduling. Exactly the one that tells the worker nodes that fly the fault. Right. Start the containers. Right. Talk to the Kubelet on each worker node to schedule a pot. It can be scheduled, like instantly right in it can be to be talk to the Kubernetes runtime engine. Create those spots. Okay. So those are the five main components of the control plane right now let's talk to let's take a look at the worker notes here. On the right side, there are three main components on each worker node. I have a Cube. Let I have a Cube proxy, and I have the container runtime engine. Okay. So basically, the cubelet is the agent that runs on each node that talks to the API server and starts any containers and also get some statistics and health information about those containers. Right. I'm saying containers here, but it's actually pods, but I'm going to explain that once we move to the next slide that the smallest unit of Kubernetes is actually POS. Right. There is also the Qu proxy component that handles all communication, networking, communication inside the cluster, but also externally with outside the cluster as well. And the container runtime engine, which by default was Docker. And there was a lot of discussions about when they're talking. The doctor was going to be deprecated in Kubernetes and all that stuff. I don't know if you heard about this story, but you can still use your Docker containers on Kubernetes. Nothing is going to change. It's. Just that the reason being that Kubernetes to be able to run Docker. Kubernetes developed kind of another component called the Docker Shem to be able because Doctor doesn't follow the to the open container interface. And Kubernetes was created to run any OCI standard containers. Right. Okay. So you can deploy that. The Cube is going to talk with the Kubernetes runtime engine, and that's what's going to really create your containers and run that on your notes. Okay. Any questions about this architecture here so far. So we're going to track right what we're doing here today on this workshop. We're using the Cloud Nine instance to talk to the control plane. Right. We're using that. And we're going to tell the control plane to create some objects for us. Right. We just with the city or command that we just use that's creating the cluster. Let me just quickly see that creating if they get your command that we use to create the cluster. Kubernetes is doing that. And the way that it works on EKS, we don't have access to this control plane because AWS handles that for us. And that's a good thing, because you also don't need to worry about the security aspect of the control plane. All you need to do, you talk to the API endpoint of your control planes and you tell it to create applications, deploy containers and all that stuff. If you're still having issues, deploying the cluster, if it's saying that's not available, it doesn't have any resources on that availability zone. Try to change regions, right? If it's still not working under different names, it's still getting a horse. I'm going to show once my cluster is completed, I'm going to show you the the output of the console and see if you're getting that to try to change different regions. It's not working on the region. Sure. The OCI is the open container initiative, right? It's like a standard for it's the standard for creating containers. Right. And the Docker doesn't apply. It's not in compliance with that standard as far as I know. Please, please correct me if I'm wrong. And so Kubernetes couldn't run Doctor containers by desal. They had to create another software that's called Bucker Chin to be able to do that. So the problem is that these components that allows Kubernetes to run Doctor containers, is it's maintained by the Kubernetes? Yeah, sure. Okay. Sounds good. What's going on is they didn't want to maintain that component because that component was, as they call it. The Docker. Shim S-H-I-M. Is kind of like it's not a good practice what they did, right? Because Kubernetes should be able to run OCI standard containers by default. But Unfortunately, Docker is not compliant with that. So they said, no, we're not going to maintain the Docker shame anymore. And the Docker she was going to be deprecated. Right. So that's why this caused a lot of stir on the Internet and on Twitter a few months back because people thought, oh, maybe like Kubernetes is deprecating Docker, I'm not going to be able to use my Docker containers and pets anymore. What's going on, right. Because pretty much I would say 90 to 95 or 99% of Kubernetes clusters running Docker containers and not the other ones. You can have different runtime engines. You can have the containerd cryo or Padman. There are different ones. But since Docker was the default one, people usually use that. Right. And because Docker is very well known and people have used soccer before, even before Kubernetes, that's what they use. Right. So I think that's it with the OCI. There is a blog post I think let me see Docker Bernette it. There was a blog post on the Kubernetes website that talks about that. That whole insurance. Let's see. Yes. Let me post that link here. Dr. Shin FAQ. Okay. Let's move on. If everybody got the kind of the overview of the architecture here, let's move on with that. So as I said before, right. When we deploy a container on Kubernetes, we're not deploying a container itself for deploying a pod. Right. So the pod is the smallest unit on a Kubernetes cluster. Right. The pod can have one container inside that pot or multiple containers. Right. So as the diagram here shows, you can have one, two or many. Right. And the reason being is that the idea of containers is to run only one process on that container. Right. So if I have my application run on my container and I need something to let's say, collect the logs and ship those logs to a central location. I don't want to install that application inside my application container, because that's going to break the principle and the idea of containers. So I deploy another container on the same pod, usually on the same pod, and that other container talks to my first my application container and collects the logs from there to ship it to another location. So that container is usually called a sidecar container. It can be used to collect metrics logs. It can be used as a security container as well. Some tools, some security tools deploy or the way that they work. They're deployed as a side car as well. So there's different usage for that size are container, and since they're on the same pan, they share the name spaces there. So other objects that you can have internet is a deployment, right. You can deploy a pod by itself, but you want to use the features provided by Kubernetes. Right. And the way the deployment works, it gives Kubernetes or your Kubernetes objects, scaling updates and row back abilities. Right. So let's say if I need to create another pot to handle traffic and there is a lot of traffic on my application, I can easily do that with deployments if I need to scale up or scale down, or if I'm not using all those containers and I can save money and reduce the number of containers that are running, then the number of pods I can do that as well. With a deployment deployment is just an API Kubernetes API object. It's not something that that exists on the node itself. It only exists inside my cluster. It's just a configuration. And the way that they use, there are many different objects that I'm going to show on the next slide. We're not going to be able to cover all those objects here, but just to give you an overview. Okay. And on top of that deployment, we have something called namespaces, and those are different than the Linux kernel namespaces. These are Kubernetes namespaces. And the way that the namespaces work here is that they work like you're just a logical separation of different applications that I have on my cluster. It's basically folders inside my cluster. There's no kind of a security boundary between those namespaces, so I can have a namespace for my developer environment. A name stays for my QA environment and name stays for my production environment. I can also have a namespace for my developer team one, and my name is this for developer team two if they're both using the same cluster. Right. And we're going to talk about namespaces as well here, which is another part in another slide. And on top of the namespace, there's nodes. Right. Node is basically the server that's running my application, so it can be a VM, it can be a missions, it can be also a far gate on AWS or other kind of serverless applications or serverless runtime or workload, so it can be anything. It can also be running on Prem and can be bare metal as well. There's also some companies that provide Kubernetes running on bare metal. Okay. And let me go to the API objects. So as I said, there's different API objects that you can create on your cluster. You have a pad. As I mentioned, there is replica set, which is used by the deployment to replicate your pod. There is Demon sat stateful set, right. The deployment itself as a nation there's service that's used to expose either Port or my application to the world. They can expose that through a load balancer as well. Her job and Chrome job, which are used to the job, contains basically a pod or a container that runs only once. The job. And the Chrome job is basically a container that runs according to the Chrome tab. Right. I apply the configuration there and it can run periodically like every other minute, every other hour, whatever I choose, and whatever I configured on the crontab configuration of that container, there is also config maps and secrets. My config maps and secrets are usually used to store information. Config maps is for configuration information that are not really sensitive and secrets are supposed to store secret information. Right. Sensitive information. But I'm just going to say here that secrets are not really secret in Kubernetes because the way that secret works by default is that all the sensitive information is stored on CD and it's stored on encoded as pay 64. Right. So you're probably aware that basic 64 can be easily decoded, and so that's not really protecting that information. And also and by default, all the all the information is stored in, etcd. Is stored in plain text. It's not encrypted. So there is another issue there as well. So there's different ways that you can use secrets on Kubernetes. You can maybe use, like, a third party solution like HashiCorp Vault or use the one from AWS as well. It doesn't matter. So just be careful if you're using secrets with Kubernetes. Don't use the default settings, some configurations that you can apply to at least encrypt your secrets on that CD. Yeah. There is in Grass and many different objects. Right. And those objects are they all work. They can all be applied as a Yamo file. And I tell that and apply that to my cluster using Kubectl. And depending on what I'm creating on that file that gets three another cluster. Exactly. Because of the desired state. I don't care how Kubernetes is going to do that. I just tell it. Okay. I want this object created on my cluster and then apply that to the CD once I send that information via Cubectl that goes through the API server and get stored in, etcd. And now the controller is checking that something changed here, and they want me to create this object. Okay. Now deploy this object and everything happens really quickly. Okay. Any questions so far? Let me see here. I think my cluster got created, so I'm just going to talk about it. And then we're going to go back to the cluster. Anyone still having struggling or still having issues? Please let me know. So basically the Keeps CTL, as we did on the Cloud Nine instance, right. Is the CLI tool that allows you to control your Kubernetes cluster. Right. It's a config file. There's a config file that can be found on your home directory under Cube, and I can show you where is that stored on our online instance. Right. It's very similar to the Docker CLI for doctor containers. Let me see another question here is an API object like request perimeters to the appropriate API endpoints. Request perimeters. Not sure I understand the question. Api objects on Kubernetes D R. They're just like different objects. They have different functions. Right. So basically, when you talk to Kubernetes and we talk to the API server, there is an endpoint for each object. Right. So let's say my Kubernetes cluster and API Socreate, right. So there is different endpoints. Everything gets get convert from the API request and get to the SD, right. Once that store an SD and depending on which type of object that gets created on my cluster. Right. Not sure if you can try to rephrase the question, please. So that I'll try to answer that better. I'm not sure if I understood correctly, but yeah. So the same tax of the Cube CTL, it's very similar to the Docker command, the Docker CLI, and the reason being as well that they did that. So. So that people that were using Docker already could easily shift and start using Kubernetes without facing too many challenges. Right. So, yeah, that's it yeah. Before we move on, let's go back to our cluster and let me go to all nine isthere. Okay. Sure. Okay. So you can see from the command that I did. Right. Create cluster. It did a bunch of things here, creating the cloud formation and everything should be working. Right. So to make sure that everything is working. As I said, if you face any horse, if there was something going on or there was not enough resources, either try to change the name of your cluster and try again because it could be just a temporary thing or the availability zone that EKS chose for you. There wasn't enough resources. And once you try again with a different name, it's going to try in a different availability zone, or you can just change regions and do the same thing and see if that works. Unfortunately, I don't know if there is a way around that if someone that works for AWS can tell me about it. But as far as I know, there's no way unless you create your own subnets and VPCs and configure that that you know, that there are resources available, but that can be like a soft limit from AWS that I'm not very aware of. Hey. So let me just clear this. See. Okay. Good. You guys see my screen? Is there another question there? Okay. No questions. So basically, let me do keep CTL get notes. Okay. So I can see here. This is the first command I'm getting all the notes that are part of my cluster, and there's only one node, which is this instance here that's running and it's selling the Kubernetes version here 120. And yeah, it's ready. So the instance is ready. If I go to the AWS console, I can see another instance created there as well Besides my cloud nine instance. Now. Right. Let's see here on the home directory. So tube, there are some stuff that was created the config file, right. If I catch that, there is some information about my cluster, how to access my keys and all that stuff. I'm not going to show everything, but basically that's it. It's there. You can see that there is some certificates, the IP of my cluster and all that stuff. One thing that you need to be aware of on ETS clusters is that by default, once I create my cluster, the API endpoint the API endpoint for my Kube API server. That's public by default. Right. I know that's a long URL, and it's hard to guess, but there are attackers people on the Internet scanning the Internet for looking for those API endpoints. So if you don't need that exposed to the Internet, be aware of that and make sure that you configure that to be private. And we're going to see that very soon. Okay. So now that my cluster is created, we're going to deploy the other objects, actually, the objects of my Kubernetes cluster right now. My Kubernetes doesn't have anything I say. I still got no resources. Fun. There is some resources from there. The default resources of Kubernetes of the ECS cluster. There are some stuff toward DNS AWS Cube proxy running, but there is no application run. You can see that they're all running on the Cube system namespace, which is the main namespace of my control plane components. Right. Let me see Tibetans or namespaces. So these can do like this namespace or NS as a shorter version. Right. And as I said, namespaces are kind of like basically folders for organizing stuff from your Kubernetes cluster. And by default, I usually have those four namespaces when I create my cluster, either the managed or unmanaged version. Okay. So now let's create the objects. Right. And let me go to this file here, and I'm going to describe everything that's there before we apply this object there. So I'm creating a new namespace on the cluster objects so you can see here. And this is our different every the way that Yama works. Right. Every free dashes here. It's it's basically a separate file or a separate object, but I put everything together so that's easy to understand. So I have one object being created here. That's the namespace called web app. So basically I'm creating a folder to deploy my web application. This is an RBAC configuration and creating a cluster on a cluster role binding, telling that the service account. And I'm going to talk about our back and services account later, but basically telling the service account that I'm going to use to deploy my web application, my pods running on this web app running as admin. Right. And so that's a bad thing. I'm purposely doing a misconfiguration on my cluster, but that's for the exercise. That's for the workshop, right. By default, you don't want to do that. You don't want to do that. There is also a deployment, and I'm using deployment to deploy my web application. I'm creating like, a Grupo deployment here because we're deploying a duo web application, and you can see here down below, that the specs of the containers that I'm using. I'm downloading Grupo eight five0. And since there is no specification of which container registry I downloaded from, that's getting that from Docker hub. Right. So once I deploy that, apply that object to my cluster, Kubernetes knows that. Okay. I don't have that image here or whatever you want. I'm going to download the by phone. It uses Docker hub to look for images. Right. Unless you want to apply. Of course, that's not a good practice. That's not something that you want to do, because if someone compromises the doctor hub or put some malicious image there, then you're downloading an image to your cluster. So the best practice is to have your private container registry and history images your approve images there for you to download on your Kubernetes cluster. Okay. Sounds good. There is also a service I'm creating a service here, which I'm going to use services or objects that I can use to expose my containers. And I can expose just like there's different types of service. The type that I'm creating here is a load balancer, which is going to create an AWS load balancer. And I expose my group of web application, which is running on 480, and it's going to expose that on a Port of the load balancer as well. So as I said, there's different types of services. There is cluster IP, there is no Port, there is a load balancer. And I think one more. Yeah. I forgot the other one. No problem. I'll remember. And this is basically I'm adding just a kind of a flag here that I use this kind of setup to deploy a CTF challenge. Right. So I have a flag here. I'm creating a secret, and I'm storing that secret as a flag name CTF on the Cube system namespace. And this is the result of the flag here. Right. The value of the flag. And it's basically basically for encoding is just a flag that we're going to practice and try to get that once we compromise the cluster. Okay. Sounds good. I hope you understood all these objects, and we're going to apply that to my cluster. And you're going to see that that's going to be very quick and go back to the environment. Let me go back to that project that we downloaded from GitHub, and that's where my cluster underlying object is. Just keep in mind that for the first file. I don't know why I did this, but the first one, I use Dash and the second one I use underline. So let's see if the command is correct. Yeah. It's using the underline. So that's fine. So once I do this, Kubernetes apply. Right. So F means that I'm passing feeding it a file. And so it's going to look whatever is in that file and apply that to the Kubernetes cluster. Just find that. And you can see that all the objects that were on that file were created, the namespace, web app, the service account, the deployment, the service, and even the secret everything is created. So if I do now keep I get namespaces. I can see that there is a web app name space is there on the bottom. Right. If I do. Yes. Pods and web apps. So now that I just want some information from a specific namespace, I'm using the parameter and to tell Kubernetes, I only want the pods from this namespace. Right. See, there's one pod running. What happens if I get this, like, Kubectl get pods. No resource is found. Because if I don't call Kubernetes the Kubectl specific namespace, it's going to look into the default one. The default one is this one here at the top and it doesn't have anything there. That's why it's not going to find it. Right. So you need something that people that are starting to work with Bernie, many main may forget and they think, oh, I just created this. Where is it? Right. If you don't specify anything, then all the objects that we create are going to create on the default namespace. But as you can see from the object that we use here, I created this new namespace and I told to create all these objects on the web app namespace. Accept this one, which is on the Tube system. Okay. Okay. Let's see another question. Where are these API objects going to this get sent to XD and then some pulling service. Check for new additions resources using APIs. Yes. So the API objects, they go to SC as like a key value store. Right. So the store an SC and the controllers, the Cube controller manager with their controllers. They check out CD and see what change do I need to create this object? Oh, yeah. I need to create a service or this load balancer or the pod, and then it talks to other objects. Like if it's a pod, talks to the Cube scheduler and tells, okay, create the spawn on my cluster. I don't have that on my cluster right now. And I need is telling me TV is the source of truth. Right? Is telling me that I need this object there. So create that for me. So some objects in Kubernetes are how can I say they have a physical representation? Right. So for example, Pod, it creates itself there when it's created on the cluster. Yeah, I can see it. I can see that there is a pod and there is like something running. There is a container and all that stuff. Some other objects are just kind of how can I say they're more abstract? They exist there. They exist in Kubernetes and stuff, but there is no kind of physical representation of that object Besides the configuration. Right. So it's just like some stuff that Kubernetes due to to kind of encapsulate that object in some specific configurations, such as deployment stateful set. It's just a way for you to help you deploy containers with different characteristics. Basically. That. So. Yeah, I hope that answers your question. If not, let me know. Okay. So now I have a web application running and we expose that web application to the Internet via the load balancer. Right. So now I can get the service of my web application, and we see that there is a service called Drop SVC load balancer running here. And this is what we need. Right. The external IP. So this is the IP of my web application that should be running there. So if I click on that, if I copy that URL and open on my browser and I'm going to show you right away, let me share the screen again to change change here to my other tab. Six. Okay. Here. Oops. Yeah. So you can see here that there is a drop web application set up on this URL and I need to install it. I need to configure and that's what we're going to do. Right. But yeah, basically it's running. I'm exposing the Troupe container to the Internet with only that cluster on that continue. Right. And with the service load balancer. Of course, if you go to the AWS console where the load balancers are, you're going to see that there is a load balancer created there as well. I'll show that later. Yeah. So that's what's being used. My container is running inside my cluster, my Kubernetes cluster. I'm using the load balancer to expose that, and I'm accessing that through that long URL. So don't use the same URL that I'm using because your is probably different than line depending on where you deploy your cluster. So let me install here. I hope that nobody messes with my drop. What I want to do here is choose the language, use the standard one. We need to configure the web application needed running so that we can exploit it. I'm going to use Sequel Light, which is the database here so that I don't need to set anything. And yeah, it's going to install. And there's just a few minor configurations there, but basically we all need to do that if we want to exploit and play along with our exploiting the application. Right. So Don can I can set anything here like this? Doesn't matter. This configuration is just to complete the configuration of my website, but. But yeah, Cloud Village, that's Con. Com. I'm not going to use any of that. So don't worry about remembering the username or password or anything, because we're not going to use that. We only want to do that to set up our drop web application. So I'm going to use just a strong password just in case that anybody tries to mess with my application. But yeah, that's it. Okay. A fine. Now I see the group what application is running and you can see that's working. I logged in as admin. I can post. I can do anything. It's a regular group web application is just using an outdated version because that outdated version has a vulnerability. Right. And now we're going to exploit that vulnerability. Okay. Other questions. So let me go back now to to the slides. I'm going to talk a little bit about how we're going to exploit and the vulnerabilities that we're going to do, and then we go back to this web application. True. Okay. See train slides. You're good. So what we're using today, and as we mentioned in the beginning, right. We're using trap modeling to attack this web to see what vulnerabilities are in the application, and then we see it. Right. So that's one of the things that I do during my daily job, right. Analyzing new technologies, especially club and container technologies. Check the possibilities of any kind of like a hypothetical attack and create a threat model for it and then try to PC it, try to create a proof of concept for that as well. One of the things that we like to use is the Miter Attack framework. Right. And as I mentioned, the Miter Attack framework is a very well known framework among the infosec community. Right. It's like a knowledge base of tactics and techniques based on reward scenarios. So you have the Miter attack for Enterprise, you have the Matter attack for Line for Windows, and you have even one for Cloud, which is now called IAS Infrastructure service for the all the major cloud providers. But we didn't have one for Containers and Kubernetes until this year. Right. So what we did was there was one Matrix released by Microsoft in April last year, April 2020, that they kind of use the Miter Attack framework structure of statics and techniques. This matrix approach to create their own matrix based on the data that what they were seeing on Azure on their AKS clusters they created. Ok. So deers are the techniques that attackers are exploiting in Kubernetes environment. Right. But that's not an official Miter Matrix, a Miter project. So in in December last year, the Miner team for Cloud and Containers. They released the publish a blog post asking for help from the community to collect data about rewrote scenarios and what attackers were doing on those environments. Right. So either previous attacks that happened, we award attacks that happens to organizations such as Tesla. And I think there's I don't recall others now, but there's many other EKS as well. They use as a baseline. So they're trying to gather real world data and real world information, or honeypot information from what attackers were doing there. So since we do this research on a daily basis, we had a lot of data that we reach out to Matter to provide them with that data to help build this matrix. So the work started basically last year, beginning of this year. And in April this year they released the official Miter Attack for Containers. And I kind of and Kubernetes as well, because that's not how Might calls it. But there are some techniques here that are very specific to Kubernetes environments, right. Orchestrated container orchestration environments. But yeah, basically, this is the matter of the official Matter Attack for Continuous that was released in April this year. So you can see that there is the common tactics there. And there is those techniques. Some of those already existed on the Miter framework, but it didn't have the context of container in violence. Right. So they add that reference or the technique specifically or that tractor. And some of the techniques are brand new. They didn't exist before. And so that was kind of a cool project that I did that I helped build with Meyer, and there's the link there with the Matrix, but it's official or Matrix for the Metatax framework for Containers is this one. And yeah, there are different techniques here. Different actors is actively exploiting those environments on one honey pot that we deployed. It took less than 24 hours for the attackers to compromise our environment, and we didn't public size that we just deployed the Kubernetes environment with a vulnerable web application. And in less than 24 hours, the attackers had compromised the cluster and even broke out of the cluster and compromised the cloud environment that we set up the cluster in, and they started deploying be instances to my cryptocurrencies. And that's usually the end code here with Docker and Kubernetes environments. When attackers compromise those environments of like or 90 of the time, the and Go is to either compromise already the containers that are ready running or deploying new containers to Monero cryptocurrency. Right. That's usually what happens most of the time. Okay. So what I did with the trap modeling part was even before this was done. This diagram here was done even before the mother framework was released. It was done last year. And what I did was exactly create this step model diagram based on the Microsoft attack, made the Microsoft Kubernetes trap matrix. And so I created like, a scenario where okay. Which are the steps that an attacker can do to compromise a web application running on a vulnerable web application running on a Kubernetes cluster. Excuse me. So here we have the initial access, right. The web app running in a pod. So we have that already. We set up that already on AWS. And the web app has an application vulnerability. And you're going to see that really soon. We're going to talk about the vulnerability that this Trupo outdated Trupo application has, and the attacker exploits this RC, right. They exploit this vulnerability, and then they get a shell inside the pod inside the container. Right. And from there they can do other things. Right. And so this is kind of the diagram that I created to help me understand what an attacker was able to do. And that's when I did the Doc to validate that. And of course, then that's where I kind of this workshop came from right from this diagram there as well. So I don't know if you can see the whole diagram if it's too small, but I can share just the diagram link later with you. So yeah, attacking Kubernetes if we have set up like a call the website Dodge coin if you have the drop set up now and you should see kind of a similar screen a string to that too. Okay. Okay. So what we're going to do? What are we going to do today is exploit this. Yes, I can share the attacks and error diagram. I don't have the link handy right now. Someone is asking if I have these attacks and error diagram uploaded anywhere. Sorry, but yeah, I do have it. I just don't have the link handy with a higher resolution version, but I can share that together with the slides. No problem. So, yeah, the initial access here. So that's when we start now attacking Kubernetes. And I'm sorry it took too long to get here, but I wanted to everyone follow along and be everybody on the same page to set up the environment so that you understand the basics of Kubernetes. So now that we can start attacking it, and that's what's important here, like not understand the basics now is just practicing different scenarios and different attacks. And maybe you can do that on your own later. So two main entry points of a Kubernetes cluster. Right. One of the entry points here is a vulnerable web application. And we're going to see why is that this vulnerable web application has an RCE, the remote command application, as you can see the CV from 2018. And we're going to use that exploit code on that GitHub to do that to exploit that vulnerability. Right. Other ways that Kubernetes clusters got compromised before was the exposed dashboard. So in previous versions of Kubernetes, there was a dashboard that was created by default that allowed you to kind of manage your cluster in a in a graphic user interface way. Right. But that got deprecated, and it's not provided by default anymore. You can still do it, but it's not applied anymore by default. And that dashboard has some vulnerabilities. Some issues. That's how one of the one of the cluster attacks that happened a few years ago on a Tesla. That's how they compromise their environment. They expose the dashboard. And so that's not the case anymore. And even the reason why, if you look back at the Kubernetes trap matrix by Microsoft, they kind of deprecated that technique. Right. Because since it's not deployed by default, let me go back. You can see here as the initial access expose, the dashboard is just kind of deprecated. Right. There is other options right. On the mutual access there is using cloud credentials, compromise container registries and all that stuff. But I'm going to focus on the application vulnerability here. It there is also the Cube API service. Right. And if the Cube API services exposed and it's not properly configured, you can even deploy pods through the API server, the API endpoints. Right. So yeah. Same thing with the Doctor Demon API. If that's exposed and there is no authentication and the person in which with access to that we can have access can reach the API endpoint, then they may be able to do some things of get sensitive information there. Right. The Kube API server endpoint is public by default in and managed services like EKS. Sally, ask you the question here. Do you have any issues if AWS having vulnerabilities in your clusters in this workshop, right. We're not. We're just deploying a vulnerable cluster. We're compromising it ourselves. We're not stalling anymore. We're not running scans on AWS, and we had some issues in the beginning, but with our honey pots but we changed the way that we deployed and also the environment. And I don't think we're going to have issues with that this scenario today. But if you do, please let me know and we're going to shut down everything later. So it's not going to be up for a whole lot so that you don't get your cluster compromised by someone else. Right? We don't want that. Okay. So one of the main things, one of the main issues that I said is the API endpoint reaching the API endpoint externally. That can be a problem. You can see two examples here. Accessing on Unmanaged cluster. The API server of Kubernetes runs on Port six four for free and on EKS, it runs on Port for for free. And if I have the URL of that API endpoint, I can just do a third request to that you are. And I can get some information about my API endpoints. So let's try to do that first before we go into exploiting that. Let's try to do that first. Let me go to the so let me share my screen here. Sorry. Keep forgetting that. So yeah, I'm here on my AWS console, and I went to Elastic Kubernetes service, and I'm going to check my cluster here, and I can see that I have a cluster running. It should be at least only one if it's a brand new account or something like that. Right. And that's the cluster that we created for this workshop. So this is kind of the console of your control plane. Right. State that I'm using an updated version. There's a new version available. I can see my my configuration here. Right. So this is my API endpoint and that's usually exposed by default. Let's see if we can open a new tab. You can see that. Yeah. Okay. Let me show you here. We can either do a curl or or just open a new tab in the browser. I'm going to show you here. So yeah, you can see here that I got like, an eight. Let's see API version or just version. Okay. Yeah. Just accessing the URL there. I got an error. Right. And I got like, this is kind of a JSON error and a 403 telling me forbidden user system anonymous cannot get past. Right. Okay. I don't have access to that. And that's fine. But you can see that this API response is very specific to Kubernetes. You can already tell that by the format that there is a Kubernetes cluster running here. And if I go to a version which is usually open to anyone by default, I can see the Kubernetes version Atrani I can see the Go version of my Kubernetes and the platform and all that stuff. So this is kind of a information leak, right. That it's exposed by default in many Es cluster. If you deploy the cluster like we did by default with the default settings, it's going to expose once someone notes this endpoint. Of course, it's a long URL, but you can see that this is running on EKS. And basically what changes is this this kind of sequence, this string here. Right. And so people can try to attackers can try to scan the web and scans for any kind of open clusters that may be running a vulnerable version or like, exposed by default. So they want to look for that. So be careful about that. Let's see. What else can we do here? Let's go back to the back to the cloud nine and stuff. Now we're going to download, download the exploit that we're going to use and and use it. Right. And basically that. So let me grab that again. 1 second. Yeah. So this is the URL of the exploit and share here with everyone. Let me share the screen to sorry. Okay. So this is the GitHub of the exploit for the CV in effect drive. So if that's the one that we're using, just a couple of things here before you just kind of clone this. If you get clone and download this to your cloud nine, you need to have Ruby running installed. It should have that on your cloud nine, but it might require some dependencies, which is a gem. And I'm going to do that with you together. If that happens, we just install the dependents and we can run it. So all we need right now is the the load balancer URL from where your application is running. So let's go back there. Let me go to the cloud nine. Sorry for the switching screens. That's safer. Okay. So yeah, I'm going to meet that URL. So just save that for later. Save it like a load pad or something, because you're going to need that for exploiting the cluster and you can do that from any other machine. I'm just going to do through the cloud nine because it has Rub installed already. So it's going to be easier for me to do that. So I'm going to add that command as well to the Google and that I clone. Let me just get out of this. Hey. So I downloaded the flight, which is returning Rob from GitHub, and now I can see that here. Right. If I take a look at the description there, basically I just need to run the exploit and set up a target a URL of the vulnerable to application, and that's the URL of the load balancer that we have already. Right. So let's try to run that and see if that works. If it doesn't, then we install the dependency on Ruby and should work as intended basically to get on the RV and the URL of my load balancer. And let's hope that works. Yeah. It's nice Highline. It's missing independence here. Cannot load file high lining Port. I think it's starline. Yeah. So another command. Let me add that to the Google docs. Yeah, that already there. Okay. So it should work now. Fingers. Let me just figure this. Fingers crossed. Not yet. Why did it work? Thank you. Open the terminal again. Justin came. Goodness. Let's try one more time. If that works, if it doesn't, I do from my own machine. No problem. Find. Okay, start. Okay. You know, maybe I need I need the path, probably. Yeah. Let's try. Let me show you from my machine, and then I'll figure out how to solve that for everyone. Okay. Let me put a terminal here and I'll share my screen to give me a second. Okay. To be. Yes, that works. Let me share my screen better. Sorry about that. Is the problem with live workshops like demos. I'm going to share my terminal. Okay. I went to the Drupal get on folder. Where were exploits? Right. And because I have the Ruby and Dependencies already installed that worked already here. And basically I did drop get on the RB and with the URL of my load balancer where my application is running. Right. So I think there's some issues with the path of ready and on the cloud nine. And we can figure out that soon. So let me show you what's going on here. So the exploit is running is looking for the drop stuff it's collecting. So you can get information about my web application where it's running and all the stuff. And I get a shell here. Right. So see, we mind it's running as the user the data from the web server or where this dis containers running. Right. Let's see. I see. I can see from the file already. If you've known Dupo before I know the fire structure. Dupo, you can see that there is a drop application running. Yeah, I am inside where I am right now with the exploitation of the web application in a shell inside the pod that's running the duo container. Right. So what else can we do here? Right. What else can we can we see? How do I know that this is a cluster is running on a cluster. Right. So something that can help you understand, where is it running the environment, right. Checking the environment variables. Right. I see a lot of environment variables, right. I don't want all of them. So let me do and grab I Cube. Let me do that. Now. I see there is a lot of Kubernetes environment variables, and I can see them all together here. And I can see the IP address of my API endpoint my API server. You can see the Port that's running it. And so there's a lot of information here. So these already telling me that this is running on a Kubernetes cluster. Let me just add that. Just add that to the Google docs. Thank you. What other? What other things that I can do here inside a Kubernetes spot? What can I do? What else can I do? It's not just like it's running inside a container. Right. And but what information can I have? Another thing is that Kubernetes stores their service account token on every pot, and that account token is used for the pod to talk back to the API server. Right. And so if I go to the always on the same location in the same directory. So if I go here to or run secrets, I think I need to increase the font a little bit. Secrets Kubernetes dot IO have it account. See if it don't show me. I don't know because of my shell. See, she is not great. I should probably reply a new share. Okay. Yeah. Now I can see it here. I can see that you receive free files here. Right. The certificate, the namespace, which is basically where the vulnerable web application is running and the token. And this can be used for me to talk to the API server. There is another tool that can be used as well, which also helps get information about the containerized environment. And this tool is called AMI Container. We post this when I post that to the Google docs. Come on, secret and and step nine, we're going to look at the Mi container. So what we're going to do is use download this to the file that I'm running and execute it. Right. So let me go back here. 1 second. Where is it? Okay. Yes. Let me share that. So let's see if that's going to work. I have the command somewhere here. Yes. Yes. Here. So basically, actually I want this one. Yes. I posted that to the Google docs so that everyone can follow. I'm going to try to do that through my shell. It works. So basically I went to the older I use Curl to download the binary of my container from the GitHub. Right. And change it permissions there and execute it. Right. So you can see this is the output. This is the output of the containment tool, and it shows that the container runtime is Cube. It shows it has namespaces, the ID namespace through username space fault. It shows if there is some if there are any protections, your Linux security modules activated like a farmer or Lenox or so. Right. So you can see that the app armor is unconfined. There are some capabilities that you can see here as well and also see calls that are blocked. So I know that as well. So this gives me a lot of information already. Right. It's good for an attacker to kind of do like a Recon of the internal environment. Right. I can even see if I can curl. I think I can even grow from here. Yeah. The same thing that I not going to work. So the same thing that I need on accessing the tab, the URL. I'm doing that from inside the pot. And since those are environment variables, it's getting those and getting the Kubernetes the cluster information. Right. So that's interesting. Another thing that I can do is you probably heard about the API metadata, right. The instance metadata from from cloud providers. They all have one. Aws has one. Google has one, and Azure as well. So from a compromised pod, if that's enabled, you can reach the API metadata as well. And I'm not going to show my I'm not going to show my API keys, but I can show you some stuff that works. Let's see. Where is it? A second. Okay. Let me grab that first and find my sheet here. Sorry about that. Where is it? Okay. Okay. I'm not finding that. Yeah, I'll show the API metadata phone. Don't worry. So another thing that you can do is you can from the part that you're compromised, you can talk to the API server too. And as I said here, like I just did the current to the API server. But if I do it like just API, you can see that I got for free as well. Right. So that forbidden because I'm not sending any kind of authentication, any kind of token. It's not working. But as I showed that there is some credentials, some service account information, the certificates and the tokens inside the pot that I compromise. I can use that to fit into my car request to talk to the API server. Let's see. Go back there. So I'm going to create two environment variables here and see if they work. I'm going to create the token environment variable that in getting the token from that directory and the namespace as well, getting the namespace from that directory. And now now I'm going to do a car request. But with that request, I'm sending the header authorization there token to this API server. It didn't work. Maybe it didn't work. My so I need to copy from the copy. That fine. Yeah. I'm going to show my can. That's fine. We're almost. I think we're almost at the end now if you're going to have time to cover everything. But yeah, let me try to run that again instead of token. I'm going to send that here. Right. Let's see. Authorize something change a worthy no problem. There is fine to go back to slides and I'll figure out what's going on. Okay. Yeah. So let's see where we are right now from the line. Okay. So we've done the reconnaissance. It's like not sharing yet. If someone adds lines, can I see these lines on my screen, on the screen? Okay. Thank you. So, yeah, we did some internal. In essence, we inspect the Kubernetes environment. We inspected the environment variables and the service account token. We even used a tool from outside externally on GitHub downloaded that too, when executed? Yes, probably. There's probably some issues. I'll check it out. So I just want to get stuck because we're almost out of time. I want to complete that. But I'll take a look here so we have more time yeah. So we did an overview of that. Right. There is many stuff that you can do here on your container. There is also a possibility of doing because the way that this cluster is configured, there is also a possibility of doing, even like a post exploitation or processes on the cluster via privilege escalation. Right. So there is this technique posted by those equally who works at Is Valent. Now, he used to work at Apple where he created. He posted this kind of command using Cube CTL to deploy a container with the privilege container that has access to the worker node. Right. And on the right side here is kind of the structure of the JSON format that he is using to override that contained that he is deploying. Basically what's happening here. And he's using, like, host ID equals true. Right. He's using the privilege container through as well as the security context. He's doing a lot of, like, different techniques to access the worker nodes. Right. And that's one of the ways to do that how that happens because I'm allowed with that service account that we created, which is giving me a lot of permissions. I can do that. I can deploy a new pod, and that's one of the ways of escalating privileges in a Kubernetes cluster is deploying a privileged pod on that cluster. If I have permissions to do so. Right. Of course, I would need to download and install kept on that pad first to run it and do the privilege escalation. If we have time, we can try to do that as well. That can be a little kind of tricky on manage clusters because the way that they handle those previous Es commissions. But let's see if that works. So let me talk about the defenses before we do some other attacks. Otherwise, we're not going to have time. How can I protect my cluster from attackers? Even Kubernetes secured by default. And where do I start? Basically, we can see that Kubernetes is not secured by default by many reasons. Right. So let's take a look at some different things that you can do to protect your cluster. One of the basic things is taking a look at one install the cluster, and this applies only to unmanage cluster where you have access to the control plane. But one of the things is just integrity monitor understanding what files are created when we install Kubernetes. The recommended ownership and permissions, and those are from the CIS Kubernetes benchmark. And if you have the integrity monitor you set up on that node, anything that changes on those files, either permissions or ownership, then you should be alerted. Something suspicious is happening because usually those don't change very often, and those are the recommended permissions. The CIS Bernet is benchmark if you haven't heard about it is like very extensive documentation with best practices for setting up your cluster on Kubernetes environment and setting up a configure. It correctly there is also another kind of recent document that was published by the NSA called the Kubernetes Hardening Guidance. I think it was released a couple of days ago. So it's really recent, but as far as what I look from the document has some has some issues. They talk about spot security policies, which is something that we're going to talk about soon. But those are being deprecated or deprecated already. They don't mention much about the Encrypting Ted, which is the database of Kubernetes. They only have a small example at the end, and they're using the Docker files, which the tag latest, which I don't think it's a good idea using the latest tag, and that's something that you learn when using containers. So yeah, I think that the CIS Benchmark document is a better one to use for deploying your cluster securely. There's also the same thing with the worker nodes. Right. It's just different files that are installed on the worker nodes with different ownership. And that's from the recommendation from the CAS pitch Mark one five, one. There might be a newer version, but yeah, take a look at that on the CIS Benchmarks website. I talked about IP system namespaces already and how they can be dangerous. Like, as I said, there is no security boundary between namespaces. So the Cube system namespace, which is the main namespace of your Kubernetes cluster, and it's like an inception, right. Kubernetes uses Kubernetes to run Kubernetes. So all the main cause of your control plane are on the Cook system namespace, and so if an attacker has access to those, then it's likely that they can do a lot of damage. Right. So be careful with that. Keep system namespace. You don't want to give access to that namespace to every user on your cluster. And the idea is that you can use our back, which is role based access control to protect that namespace from being accessed. I already talked about the API server, how that shouldn't be exposed, and how we can get some information from the API server and see if there is a cluster running, right. Yeah. There is some information that you can get running. Some comments if you have access. In this case, we don't have access to the control plane, right. Share access. So we can see the configuration of the API server, but you can do that if you do. If it's an unmanaged cluster that we talk about the Cube on the architecture diagram, the agent that runs on each node in the cluster, make sure that our containers are running in the pod, and two main security settings for the Cube is restricting cubelet permissions and rotating the cubelet certificates. There is another thing that there was a blog post that I published earlier this year talking about how attackers using a Cube exploit to compromise different clusters. So the way that it works, just as a quick overview, and I'll post the link for the blog here. It's like I'm not trying to promote anything is just that's a technical blog on what we analyzed from a very famous tractor. The way that they do is they compromise your environment, and once inside your environment, they scan your whole interior network for Kubernetes clusters running inside your network, and they specifically target the Cube. Let we get your blood exploit that's being known for a while. Okay. So take a look at that blog post if they can share the link. But yeah, there is something that's happening in the wild, as well. As I said, the Kubernetes Benchmark is a guidance for establishing security configuration poster for Kubernetes. There is over 120 security checks for Kubernetes cluster, and it has been developed by Cloud native and Kubernetes security professionals with much more experience than I have. So Rory Macoon from Aqua Security and Release Rise from Is Valence and many other contributors. Right. And there's also specific ones for EKS me. And I think there's one for AKS, if not already, because the way that the managed services work, they are a little bit different, a little bit different than the standard unmanaged Kubernetes. So yeah, it's better to take a look at the specific documentation for that. And I think that's the best guidance out there for protecting your Kubernetes clusters. I see you don't want to if you don't want to run, if you want to check your benchmark manually, right. Because it's a long document. It has a lot of checks that you need to do. And the way that the benchmark works is that you have you have the setting that's recommended. Okay. Here's the setting. Then it has how to chat if that specific security setting is properly set up, and then if not set up, it has another command for you to run to change that setting to the proper configuration. So the benchmark is a PDF file. It has a lot of stuff and many best practice configurations, but it's very hard to do that manually, and especially if you have multiple clusters that's going to be probably unfeasible for you to do. So. Aqua Security, which is also a company that works with Containers security products they provide. They created this tool called Cube Bank, which is open source, and it's free for everyone to use that. They check whether Kubernete is deployed securely or not. You can validate it against your CIS Kubernetes benchmark, and it's developed in Gold. So it's open source. And let me post the link here from GitHub bank. There's a dash up on it. Yeah. Okay. So you can use that. Right. So I just have an example here. I don't know. I don't think we're going to have time to run to Benton on our own cluster, but it's very easy to do that. And you can see an example of an output here showing that my cluster has a lot of Vulnerabilities, or maybe it failed a lot of best practice from the CIS benchmarks, right. A lot of fails and warnings and whatnot and it checks everything from the master node, the API server, the CB, and worker notes as well. So it's interesting too, if you have closer running on your organization and you want to see their status and then how they look if they were configured or not. It's a good tool to use and get a quick overview if they're compliant with some bad acts from the benchmark. Other stuff that you can do is use some image scanning right before you deploy your containers on your cluster. The idea is to scan them for vulnerabilities or outdated dependencies, and there's different image scanning tools out there. I'm not going to talk about them. There's different open source and enterprise ones. All those here are either three or open source, so you can take a look at those there's, even like a native Dockers can command that you can use now, which is based off a snake. But yeah, there is different others. Okay. Now you scan the container image and you deploy that image on your cluster. Right. But how do I know that someone has compromised my cluster? Some compromise that container that out, right. That's when the cloud native runtime protection comes in. And probably if we had a tool such as like Falco, which is an open source to Israel that belongs to the CNCF. Now it was donated by Cystic and it probably would detect some of the attacks and the shell that we got inside the container. Right. So what Falco does is Falco parses Linux kernel calls. At one time it's using a technology called EDF Extended Berkeley packet filter, which is kind of a technology that many container security companies are using to create their own tools, their container security tools. Right. And it has a lot of visibility on the system, so you don't need to install agents and like a match with the container deploys sometimes deploy five cars and all that stuff. So Falco is a rule engine, right? It has an easy and powerful room engine. You can create your own rules for your Vulnerabilities. It generates alerts based on the trap detected, even using the Falco Sidekick T shirt today that I got as a configurator to the Falco projects, and it detects an expected behavior on a cluster. And the site case is another project that runs with Falcon gives you better kind of visibility. What false on the logs and everything here? We're almost out of time. But there is other settings that you can apply to your bot. Right. Of course, CPU and memory is limiting resources to avoid denial of service. That's something that you can do. You can apply a security context as well. There's a different set of of settings that you can apply on a security concepts. Some of those are here. And as the Linux security modules, the kernel features so armor and Linux can also be used to apply that to your Kubernetes. Be careful because Docker has some default profiles for those for Armor, for example. And so. But when you use that on Kubernetes, Kubernetes doesn't inherit that from darker, so you need to apply that as well. And Port security policies is something that got deprecated. It's not being used anymore. It was a way to apply the security context as a whole on a cluster level, and there is new features now new tools that you can use. Some alternatives, such as OPA Gatekeeper or Hi Berna, which are open source tools as well that you can create policies as code to say, okay, only deploy this container if it doesn't have any higher critical vulnerabilities. Right? There is a a new Pod security and I forgot to update the slide, but there's a new Pod security. It's not BSD, but there's a new version that's being implemented by the Kubernetes is off and six security team it has been approved already. Yeah, I won't have time to talk over our back and a network policy and let me finish with the basics and I'll share those lives with you. And if we want to talk about those other things, our back network policies and August logs, we can chat later. I'm available on social media, Twitter, or LinkedIn, but basically what I wanted to give you here is just an overview of Kubernetes, an overview of the security issues, and also an overview of how to secure your cluster. It doesn't mean to be is not an extensive workshop. It's not in that workshop. So keep that in mind. So I just want to leave you with a few a few basic rules with Kubernetes, right? First thing is update your environment version earlier and often. As I said, there's new versions being released very frequently now three times a year, and the version 122 is the most up to date one recently that was released recently. Don't use the cluster admin user for your daily work. I treated like root, so the reason being why we got to exploit that vulnerability and then get a shell on the cloud and then do bad stuff on the cluster was because the service accounts of that pod had too many permissions. You can even deploy new Paws and escalate privileges and all that stuff. So be aware of that if you can use managed Kubernetes services such as Asks or GKE because they have better security defaults and you don't need to worry about the control plane because they take care of that for you. And as I said, check out the CIS Kubernetes Bank Mark for more security best practices. I hope you enjoyed this workshop. There are some references here and I can share the whole command list that I use and you can try to replicate that on your environment. And I'm sorry if some of the commands didn't work, but I'll take a look and make sure that's fixed? And before I share with you so Yeah.