Docker has become an increasingly important subject when we talk about containers. However, it's made to be used by humans. What if we needed containers to be controlled by other applications? Is it possible to do it by using JavaScript? In this talk, we'll build a container control application that will communicate with a container runtime through gRPC and will manage all our containers. Welcome to the new era of container management!
PUBLICATION PERMISSIONS: Original video was published with the Creative Commons Attribution license (reuse allowed). Link: https://www.youtube.com/watch?v=y1D8UD5TX_I
Hello, everyone. My name is Lucas, and I'm thrilled to be here to be presenting you on this awesome event. Well, let's get started to it, so we don't lose any more time. So today we're going to be talking about integrating containers into JavaScript in several ways. I've been doing this. And how did I get to this and how we can leverage this power. Okay. So basically, these are my social networks. This is my email. My name is Luca Santos. I am a developer advocate from Apollo, Brazil. Here Microsoft. And if you want to get in touch with me in any way, just put my social network from Lucas lsantos Dev. So if you want to know anything about me at all, if you want to know any other social networks, just go to Info lsantos Dev, then let's get started. So containers, they are a thing now. They are. Containers are widely used for a lot of things. They used in several areas with several tech technologies. Kubernetes is year Dockers here, and there's a lot of stuff that actually uses containers, and it's something that's been really active and really hot in the technology area. But containers are a new the exes since the 80s or something like that. They started in Linux, and we are not actually telling the whole story of containers here today. What I'm going to show you is just how we can make this work with JavaScript. And why did I choose to do that? Right? It's easy to control containers, right? It's pretty easy. Doctors here to show you to show you that we can actually control the via, like a CLI or something. And basically Docker does most of the work for us. You don't need to understand how the underlying infrastructure actually works because this is basically not important for you to get your work done. So controlling containers mostly have been done by using CLI or other graphical user interfaces or something like that. But it's easy if you're human. Most of the CLI and most of everything we use now they're made for human beings. They're made for people. So Docker CLI is awesome. It allows you to do a lot of things. And there are other toolings that are actually pretty good and what they do to allow us to spin up containers. Kubernetes is here to show you that we can have a good car with control and a lot of other things. But the point of all that is that we we need to create things as if they didn't didn't were created for humans. Right. So all these tooling has been created to be used by humans in the command line interface instead of programatically if they were an API or something like that. But what happens if you're not a human? If you're like a program on an API or an operator or something like that, you might face a hard time by. I doing that for a lot of reasons that I'm going to show you. And now we are going to just understand what we're actually working with. So to present this to you, we're going to use a technology called Container is actually a tool that was built by OCI. Oci is the Open Container initiative. It's built up a lot of companies, including Docker and Microsoft and a lot of other companies. The basic idea, the goal of the OCI is to create a standard interface for all containers and all images that we use today. So Containers is a high level run time. Actually, it's more like a runtime manager because it controls something called runs inside a Linux machine. So what we're going to do today is that we're going to spin up a Linux machine. So I'm going to use a Linux machine with containers installed and runs installed so we can run our demos or examples here. So basically this is why containers are actually widely used in Linux and not so much in other OSS. Right? It powers Docker. So Docker actually uses Container underlying its implementation and infrastructure. It was part of Docker in the beginning, and then it was split up into several other stuff and container differently from Docker or other doing. It was created to be manipulated through an API or an SDK client or whatever. Di. Container D also has a CLI interface called CTR because it's actually pretty easy to manipulate containers using the CLI. And basically Container D actually does that for you. If you have a CLI or something, you don't need Docker installed in your machine and you don't need anything else installed. So it has a live despite being created to be used programmatically. But this is not important. Actually, having the CLI is one of the parts that will allow us to control it a bit better. But that was one of the attempts, but Container D is actually widely used. Most of the container runtimes uses container in underlying infrastructure or run, seeing some way. But it's not so easy for JavaScript developers. So if you go Google or whatever search engine or whatever documentation you might find, you're going to see that most of the container things are done using Go continue is actually built. Using Docker is built using runs is built using goal. And there's like a lot of other things that we can do, but they are mostly aimed to Golang developers. This is one of the examples. There are a lot of other examples, but basically what we're seeing here today is that we need to import all the packages and you see that in this right hand, most image here that we have have a container, the client that is actually well integrated into Go, and you can just import it and actually create containers and the lead containers and pull images and so on. Just using the Golang CLI, which is not actually present in JavaScript. But then I actually came across an article that made me think of it. This is actually not wrong, right? It's not actually wrong to be written in Go. It's per formatic, and it's a great code, but it's not intuitive is not that intuitive for those who doesn't know going or whatever other language. So I came across this awesome article by Marcos how to get a browser to communicate with Container through GPC. So I thought, well, if container has a GOP interface, maybe we can actually integrate things into JavaScript. So I thought, well, what if I did this using Node JS? Right. I know how to integrate it into Java interfaces using node JS and know how to do this using the JavaScript command line. And I know how to use this on JavaScript itself, that we have a lot of Libs that allows us to create a JavaScript plain JavaScript Java Se interface, and that are very, actually good to do this. So I tried a full attempt. Okay. So the first attempt I did was sometimes good, mostly bad, but it kind of worked, and it was ugly at all. But the first thing, this first attempt is just a proof of concept to show us that our two actually, there are two ways or even more ways that we can integrate container into node. It's actually a simple way to, you know, avoid being so over bows and trying to actually not to know all the concepts. You don't need to know all the concepts behind container to make this work. So the first attempt was to use chat process. So basically what I did was that I created a child process, and I spawned this child process to actually create instances of the CTR Continue interface. And, well, this worked very, very well. So this CLI called CTR, which I already told you, and it's able to do a lot of stuff. Maybe, like everything that Continue can do is you were able to do via the CTR command line interface. So what I did was just put all these commands into a node JS process and try to build something out of it. I'm going to show you the code, but let's talk about the pros and cons of this first. Right. So it might be thinking, well, this is easy. I can do this home. Of course you can do this at home. And I encourage you to do so. It's actually very nice to see these things working. But the pros of this is that it's super easy. It's very easy to integrate using a command line interface. And using CTR using child process. Child process is actually amazing to use. And you have a lot of tooling on top of that that you can indeed use. And, well, this is one of the pros. The other thing is that it takes advantage of all the implementation that's already done in the CTR in the face. So you don't actually need to be so into containers or you don't need to know how container actually works. So you can make everything work inside JavaScript. You just need to know how to call a process, and you just need to know how to control that. And well, it's a faster implementation. So basically you can do things very, very fast. I did this. I did this container decline in like 1 hour or so. Right. And can be integrated to an API. Basically, if you can manipulate this through the code, you can integrate this using the APIs. So it's actually so simple that I did this. I did this amazing container, the API, which is basically a, you know, vanilla JavaScript Renes modules in the browser, and it doesn't need anything else. Just some CSS and some HTML. And it kind of works. So as you can see, I can create these images. So let's jump to the code and I can show you this actually working. So this is our code. And basically what I'm going to show you is this small server it's built using Cola. So as you can see, it's just a few routes, delete route, a put route and so on. So we can create containers. This is our course, and we have a body parser and allow methods and so on. So this is a base Pi, and this only calls the CTR. Ctr is a container decline that called container D because of obvious reasons. And this is actually everything that does to work. So it's already running here. I run both the API and the application. So this is the static application that I'm running using the Go static web server. So this is all the same just to serve the HTML. And basically what you can see here is that I have my implementation of container D basically using just child process promises. And this is actually yet. So we have these actions that I freeze. We have the list name space action. So I just exact sync. So I just run an execution and sync execution of CTR client. This is really, really just a proof of concept. Right. So if we have a stand and error, then we just error out. If we have a another thing that is not understanding error, we can pass this string output. So basically this is the idea. So I can pass the string output. I can run everything that I need into this name. So basically, this is the amazing container API. As you can see, I'm a back end developer. I don't have a lot of experience, and I can do good front ends. So I can change the phone name. Space. Container is based on namespaces, so I can change the phone space. But I won't do this. I'm going to download. I have this network manager here open on Dev tools. So I'm just going to show you how these things are working, but I'm going to pull this image. So I'm going to pull the global API, which is a simple image. It looks like it wait, more like five megabytes or something like that. Then I can create a container from this image, and I have this website already open for API. This runs on eight, eight. So as you can see, nothing actually happens here because the container is not running, but I can actually run a task to run this container. So once I create this task, my container is going to run. And as you can see, I can actually use my API, and I didn't need to integrate anything into anywhere, so I can just kill my tasks, delete my container, delete my image and everything is going to be back to what we had before. So this was my first attempt of creating something that was going to be used as an integration for container. Right. So this it's pretty good, but we have some cons, as you may have seen on that. The first thing we have is that it's heavily dependent on the environment you have. So whatever you were doing, you can just run this client if you have it installed in the same machine as your container binary. So if you have, you have a machine with continued inside it and you need to run this API in somewhere else. Basically, you can do it because it needs to be installed in the same machine as the API. Of course, you can actually continue run this API and then run a reverse proxy or something on top of it to communicate externally remotely to this API. This is completely feasible. Right. But there are other cons on this. Basically, we don't have any control over the process. So if by any chance, the CTR runs into a problem or something that we haven't anticipated. And being honest, it's really, really difficult to anticipate any sort of errors that happen in a command line interface because it's not meant to be giving you all the details of the error because it's basically built to be used by humans, not machines. So you can control all the process. You just control its inputs and you receive all these outputs. And this is everything that it's bad about it because you receive an output that's basically a giant string, and you need to parse the string to figure out whatever if you had any errors or something. Some of the clients, like continue declines, are they have a quiet mode that doesn't output headers or something, but it's, you know, poor. I can't actually get the container status or the image status or ask status because these these informations are actually hidden behind a parsing or a table like structure, something like that. And basically what you have to do what we have to do is to parse this giant string and use string parsers and so on. So we can put this thing into data structures that we can manipulate. Okay, so this is the first part, but the second part is that this problem doesn't allow us to give a proper error handling because we just know that this is going to be an error. When we we have a standard error output and the standard error output is meant to be used for humans. So it's a human readable error, and it's not a machine like error. It's basically what we do is just receiving the error string, the error message, and we can't know for sure if that error can be corrected or not. Instead of like we can just parse it. We can parse it by reject. We can use whatever thing that we can. But this is basically basically running all the errors in one place and trying to figure out how to parse everyone using regex and using every sort of hacking that we can do strings. So this is really bad. This is one of the reasons why this API and this front end doesn't actually return a good error handling. Sometimes when we run into an error like the image resort image doesn't exist. It returns like a cars error or something like that, and I can actually return this because I'm mirroring this out and this doesn't return my error properly. Right. You need pseudo to be run, although this is actually possible to be removed. You can you can configure it container to run this as another user. So there is a config file, and I think it's DC container config Tomo that allows you to change the default user ID and the file group ID. The container is going to create its socket, and it's going to create all this process. So you can actually change this to allow you to run containers without pseudo, but it requires extra configuration. Right. And actually, this is the most important part. We had the error strings. We had all the output strings, which is kind of boring. It's kind of complicated to treat, but it's doable. But the only thing that we need to actually be aware of is that this is a huge security failure because we can input anything into that. So if we run into any attacks, or if we run into any hackers or something, we need to sanitize all user inputs for API because this otherwise can be passed on to our container decline. And this can run some things that we don't want to run inside our machine. Right. So you actually have to be very careful in that. So the conclusion is we can't integrate containers into JavaScript. This is the first way. This was the proof of concept that I needed to make sure that this was actually possible, but I wanted to do this a real integration. Like I wanted to integrate this without manipulating external stuff and manipulating things that are already there. So I wanted to actually create something that was going to be native. Ok. But not so native because it's just gRPC. Right. So the second attempt I did was to read on to the container D Japes interface. So contained was meant to be extended. You can extend contained and it has a job interface that allows you to do so. So if you have these are all the proval files and you can see these are all the concepts that Container D actually has. Containers count in des, event images, introspection, leases and namespaces, snapshot, tasks, and whatever thing else that you want to do. And this is like the image service. Okay, so we have a get a list to create an update image and so on. The problem is you need to know how to do it because the interface needs you to control every aspect of a container creation pipeline from downloading images to downloading blobs reading manifests, creating containers creating an OCI spec for both the image and the container, creating all the bindings you need from the file system to the container, creating all the flags needed and everything else. If you have anything that you run, run like you need a lease for resources, need to create leads and need to create a container and mix that up together. And in other words, you need to understand how containerd works under the hood. So in every aspect of the container and creation and every aspect of how containers actually work for these, you might need to read the OCI specs for image, the OCS specs for distribution or the OCS specs for the artifacts or containers, and whatever the runtime spec. For instance, this is the example I want to show you if you need to create an image like the list RPC here doesn't actually get any parameters is just a filter, a string filter to filter the images, but the create one actually needs an image type. So the image type is not just a full image. You can just download an image and pass on that image to the PC interface. The image type is just a descriptor. So basically the descriptor is just a name, some set of labels, a descriptor, a CI descriptor a timestamp for creation and update. And basically the image is just a pointer to a set of blobs. Actually, the blobs a pointer to a set of images, but the image interface itself. I can create an image without even downloading that. So I can create this image. It's going to be there, but there's nothing behind it. There is no root file system, there is no file system at all. So in order to fully create the image, we need to download the content. So this is the service for content that we need to download it. And these downloads are blogs. As you can see if I run like CTR content list, we're going to see that we have all the layers from our image. Like it's just three layers. But those are the blogs that I downloaded when I pull the image. And in other words, you need to understand all the flux of how to download an image, which is not that difficult, but this is not very well documented. The documentation is a sparse. It's actually scattered everywhere. Right. So it's possible to do it. This is what I did. I integrated with the GPS interface, so let's jump into the code so you can actually see this working is not that immense example or something that's going to take a breath away. But in other words, it shows you that it's actually possible to do. So. So what I'm going to do here is just I'm Loading the proto file. So the profiles are here. They are completely descripted, and they are completely downloaded into this thing. So basically, this is the content profile, and it's very well commented. It's very well documented here inside the profile, but not the whole pipeline process. Basically what I'm doing. I'm Loading these profiles using the full gRPC module from NodeJS. So this is it's a bit laggy because I'm using X to execute the host code outside of my Linux VM. So this is running inside a Linux VM, so it's a bit laggy, but I'm using PCs and the proto loader to load an image definition and the content definition here down below, as you can see. So I'm just Loading the content proto and I'm Loading the images proto file. I'm creating a client. So I'm creating this image client from the Container Services Images V one images and I'm binding it to the Unix socket. So this is the address that we're going to bind to our gRPC server. Container actually allows you to expose this as a TCP and over Http server. And I can create this without any job requires you to have a certificate because uses Http two under the hood. So we can actually create an empty certificate using this. And we can add this metadata is just a header, basically. So it's our header named container namespace, and we are going to use this namespace JS CTR, JS continuity. And this is the payload that I can use to create an image. So it's the name of the image, the target. This is the descriptor, the OCI descriptor, which is basically a size, a digest and a media type and the creation date. So I can create this using this payload and that namespace and I can list all the images using a filter, which I'm just not going to use for the creation. I'm just going to use the content definition. Right. So in order to do this, of course, I'm going to run this for you down below here in Start and as you can see, I can download this image. So I downloaded that. So this image is now here. And if I try to download it again, I'm going to have okay, because the image is already there. As you can see, the image already exists. You can see this is a very good error. Details. It's a very good error code. So as you can see, we have a code, you have details, we have a lot of other things. And if I execute this on my amazing container API here, but changing the namespace to sir, we are going to see that this is actually what we just did, but I cannot create a container from this image because it's just a pointer. Right. So this is what I did right now. This is what I had time to create for this presentation because these two content definitions does not need actually any inputs. So this is the output from the container as the content definition. As you can see, this is just a blog file that is put into JSON, and basically this is what we have when we download images. Okay, so now back to our presentation over here. You're running out of time. This is every one of those two attempts have pros and cons. So the first pro is actually we have proper error handling. As you could see, we have a basically basic interface for errors and a basic interface for handling these images and these messages and everything else. So we can actually return a proper error to the user. We have proper return codes so we can actually return proper errors to the user and allows you to fully control all the aspects of the pipeline. So if you have anything that you want to be better or optimize, you have caching. You can do whatever you want from this pipeline. You can download the manifest and put them into cache and then download this for further downloads. You can just look into the cache and then download all the blogs again or whatever. You can optimize it. If you can find blobs into different images that have the same digest, you can just use them mutual, or you can add whatever features you want. So it's basically the whole control the best of the control you have and does not need to be in the same machine. You can. As I said, config container to output the API interface into a TCP Port, but it needs you to have a CRT certificate, so you need to have a certificate file. You actually have to have a certificate file in a private key file to be able to do that. But once you're once you have this, you can connect to your gRPC API from whatever external machine you have. So it doesn't require pseudo because you're actually connecting straight continuity, and it's practically native. Okay, so you're just using a native module. Non native module. You're using a native protocol. You can actually extend this. This is the standard way to extend continue. Okay, so the cons are that it requires a lot of knowledge. I don't even know how to start describing this because I didn't even discovered how to actually execute this in a pipeline completely. I'm doing this job. I'm going to tell you about this by the end of the talk, which is already in hand. The documentation is a sparse both for Jrpc in JavaScript and continued. So you don't find this very easily. You need to actually read the source code for goal or not for Google, but for continued Ncti to actually learn how these things work and how to build the OCR spec. It's way harder as you can see and might be a problem to connect external servers, since you need these certificate files. Right. And and everything to actually connect the external servers. So what are the next steps we can do to this? Actually, this opens up a very good opportunity to integrate JavaScript into containers and make CI interfaces or whatever. Actually you want to do using containers. So what I did is that I installed starting a project. As you can see, you can actually go into my GitHub account and you can help me do this because it's really in the start of this project. It's Container DJs. There isn't anything that is actually like it. So I'm building this from scratch. I'm going to use everything that I've been learning so far and everything that I can find on documentations to build a very native and very usable client for continued JavaScript, folks can use this to build their own tooling and to build their own better version of containers, Docker and leverage all these incredible tooling that we have into other languages as well. So let's send more knowledge and actually create more documentation for this. So if you know how to do it, create more documentation, post about it, blog about it and actually share this knowledge. So we can create a battery ecosystem both for containers and both the JavaScript folks and both for those who actually love containers in love JavaScript like me. So basically, this is what my goal is to create more documentation into Microsoft Dogs and whatever other platforms. We have to make more people able to create containers and to understand containers and to show them that containers are not that hard as it seems to be. Right. So these are the reps that I took. So basically all the GitHub files and all the repositories that you're going to see here. All the code that I showed you is in these repositories. So this is the integration example using just going and continue with a client. This is the JS container, the example that I showed you right here. And this is the ongoing work to the Container DJs client. This is Mark article as web interfaces, and he has a lot of good articles in his medium. I really suggest you to go there and read it. It's amazing how he can integrate this into several other languages like Java. He can do this using a web. It's amazing. Well, Mark is actually one of the people that gave me this idea to try to integrate this into JavaScript and make it work. So I can share these things with you if you want these slides, it's on slides. Lsantos Dev is actually integrating containers into JavaScript, so I'm just going to leave this here. Thank you so much. And it's a pleasure to be here if you want to talk to me, if you want to Ping me out on anything info into Dev and well, I'm always available on Twitter, into Facebook or LinkedIn whatever in GitHub. So if you want to help, just come on and Ping me. So thanks a lot. And I hope you have a great day.