Serverless and microservices patterns produce many small parts for cloud-native systems that increasingly integrate services, or functions. See how effective visualization matters in solution design and implementation.
This presentation was recorded at GOTO Chicago 2020. #GOTOcon #GOTOchgo http://gotochgo.com
Lynn Langit - Cloud Architect Building Serverless Bioinformatics Cloud Data Pipelines
ABSTRACT Cloud-native systems increasingly integrate services, or functions. Microservices and serverless patterns produce many small parts. See how effective visualization matters in solution design and implementation. Understand emergent visualization by example [...]
TIMECODES 00:00 Intro 00:37 Why do system pictures matter? 00:57 Cloud infrastructure 04:17 Bias - do icons matter? 09:21 Next - can we solve a more difficult challenge? 10:04 Bias - solving difficult problems 12:59 What went wrong? 13:55 What makes good information design? 17:17 Visual grammars for cloud systems 19:48 Dimensionality reduction 22:15 Perspective - User, Developer, DevOps 25:50 Conclusion
Download slides and read the full abstract here: https://gotochgo.com/2020/sessions/1271/visualizing-cloud-systems
https://twitter.com/GOTOchgo https://www.linkedin.com/company/goto- https://www.facebook.com/GOTOConferences #Cloud #SoftwareArchitecture #Backend #Programming #CloudNative #Serverless #DevOps
Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at http://gotocon.com
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. https://www.youtube.com/user/GotoConferences/?sub_confirmation=1
Welcome to Visualize and Cloud Systems, I'm Lynn Leggett. Pictures are going to be important, this talk, and because of the world situation, this is the picture I have of Chicago. Unfortunately, we're not in Chicago. We're doing this virtually, however. A picture that is more foundational to my talk is this one probably not familiar with what it is yet, but you will be by the end. As developers or developers, professionals or architects may be wondering what is even the purpose of this talk? Why do system pictures matter? Don't we write code, don't we write Daboub scripts? Pictures really just aren't that impactful. I would argue that they are and I'm going to provide examples throughout the talk when we build a cloud infrastructure, cloud pipelines. We often think in terms of writing some sort of code and we have several choices, I've represented some of them here, you know, we could use the Web UI and we could click, but that's not generally what we do when we're doing the production. We generally start with a script, whether it's using the cloud or something like that. Often we'll use templates. Bane of our existence, right of US confirmation templates, chickpea deployment's, Ali Baba Cloud Fund templates, sometimes you want something a little bit more vendor agnostic like TerraForm. Some of the work my team's been doing has been in bioinformatics and there are some specific infrastructure languages such as workflow, definition, language, Seawell and Next Flow and others that are sometimes used in place of terror. Former cloud formation, also some verticals, bioinformatics again have web wise that can replace the underlying vendor web UI. An example is Tear It Up bio from the Broad Institute at MIT and Harvard running on GCP Galaxy Project, running on eight of us. When we think about system architectures, let's warm up by looking at what's there. So this is probably recognizable as a standard Amazon architecture. It's a store. It's got three different layers and API Gateway, a compute layer with serverless and server based services and a day layer with both serverless and server based services are a server based, for example. It's pretty quick and easy to visually parse. That's important. Let's look at a similar but not identical reference architecture from a competing cloud vendor. This is Alibaba and notice no attention is paid here to stylizing the icons by color, which really impacts how quickly we can look at this and understand. We have to read the words function compute to understand that's the serverless layer. Which one's easier to understand this one or this one? I would argue the first one is substantially easier. And I think in part and I have a few examples in the talk, that's why Amazon is often a dominant cloud provider, because they provide reference architectures that people can build more easily. Now, in the world of Amazon and this is a snapshot from their console, along with other cloud vendors, it becomes increasingly difficult to parse what they offer. This is a list of all their services and working across different clouds. As I do. One of the services that my team provides is to help customers to pick different implementations of services because there are so many. And I can help or not. This is a partial representation of Google cloud platform icons like Alibaba, they chose not to include color, which I think was a mistake. If you contrast this with a subset of the Amazon icons, you can see the level of detail that Amazon has included, which makes it much simpler to pass the type of service. Orange, for example, is compute red is file storage, and then the choices within the different types. I think it directly leads to adoption. I've actually had customers, for example, who asked me to use Amazon style icon so they can, quote, understand what they were saying. And I believe there is a bias in some of the cloud companies that icons don't matter. They're an afterthought. And I think that's unfortunate. Another set of visualizations that my team did was for a fintech client and they were looking at machine learning and machine learning has lots and lots of services. They wanted to understand between the vendors which machine learning services were available, and that's going to help them make their choice. And they really couldn't pass it all from the list of services or from the vendor icons. And so I tried this approach with them and it was really successful, someone to share it here. I made a visual representation at the level of service. See, the top is the APIs. So those are the endpoints, things like recognition for Amazon, where you can just feed it some photos and we'll give you the labels. And then I did the same thing for Google so that there could be a comparison point by point, section by section, and the customer found this to be really useful. So this is a technique that I have used in high level architecture discussions with sea levels and senior technology leaders and companies. I've abstracted at a higher level for point of comparison, different services for quick visual consumption. Oftentimes I draw this on a whiteboard, but you can see I can more easily compare. So the next set of visual examples is around work that we've done in bioinformatics. And it's interesting area because of the potential impact to human health and because the amount of data and the industry really just is not been able to keep up from a process standpoint in building optimized cloud pipelines because the amount of data has just exploded. So it's a fun problem. Space to work in, but challenging and visualization can really impact success or failure. So one of the first projects that I worked on was the tool that a team built for enabling faster, faster, crisper editing research. And it's called G.T. scan to. Here is an example of the actual live tool. If you click on it, it's going to show you metadata about different sections of the particular sample, the genomic sample. And this will help the researcher to understand if the area is a good candidate for CRISPR editing Green or a bad candidate because of aspects of the sample. And again, not getting too deep into the bioinformatics. This is a BASTABLE search problem because you were looking around at different areas. You'll have different amounts of samples. So the team, when they built this solution, which is running on Amazon, attended an eight of US conference in their region and they saw a reference architecture, classic serverless reference architecture, kind of similar to the one I showed in the beginning of this talk that had API Gateway and Lambda and Dynamo DB this saw picture, and they were able to understand it. And more importantly, they were able to build it for the solution. The underlying architecture of the solution looks very much like that first picture. This is what it is, was published on the Amazon blog. And again, I have links to all of the pictures at the end. I created it just so you can read the blog post if you want to understand more about this. But the point from a visualization aspect is the team was able to understand what could be built and then they were able to build it really quickly with no cloud background. Very, very impressive. One of the first cloud native solutions and bioinformatics. This is three years ago now. This is where I entered the picture. I happened to meet up with a team. I was interested in talking with them about how they did this and how this was working. And another visualization story about G.T. is when I first met up with them, it was working pretty well. But there was sometimes it was bottlenecking. And I said to them, oh, well, have you had a chance to review your logs? And they said, well, yes, but we can't really find where the problem is. And I said, well, there is a new visualization tool at that time called x ray. And here's what the solution look like. Using X-ray after we optimized it the first time we ran it, one of the circles which represented one of the services turned red because of latency. And so it was super simple to identify that problem area. It was log aggregation, visualized the team, rewrote the business logic, fix the lambda, if you will, redeployed it and got an 80 percent performance improvement within twenty four hours. And this is a problem they had been working on off and on for a week or a couple of weeks because they hadn't had time to go through the logs. Again, the visualization of the system, not only at creation time, but at design time, at runtime, across a life cycle of the system is really key. And particularly as we build micro services and serverless based architectures with so many parts and pieces, people can just not pass it without a visualization. In addition, because there was an effective picture, one of the team members worked both in Australia where they were based and in China, and his other team in China wanted to build a scan on the Alibaba cloud for use in China. And I would argue that they were able to do that because they use the reference architecture picture as a basis and they were able to get it up and going. And this is what it looks like on Alibaba. And again, I have links if you want to read about the process, but it was relatively quick because of the effective visualization. So the next problem that the team gave to me and my team was to take a library they had written in Schola for a specific type of genomic analysis called Variant Sparke, and it was running Runnable on their HPC cluster, but it was taking five hundred hours. And they challenged my team to come up with a solution that would make use of cloud scalability. So what we were able to do is have a dramatic positive result. And just to kind of get to the TDR, we got it down to a 10 minute run. But the process that we took and the role of visualization, I think is very germane to what we're talking about here. So the first thing is, although we did solve the problem in that they were able to get a runnable solution that was running on the cloud at much faster at a reasonable service cost. There was revealed in our work our bias towards directly jumping to solve difficult problems without having intermediate steps. And this is represented visually by overly complicated architectures. So I wrote this whole process up. It took over a year because we didn't work on it full time on medium. And again, the links are there, including all the source code. So if you're interested in what we actually built and how we built it and looking at the templates, you can do that. But from a visual standpoint, the first thing we did was not go to the most cognitive optimal solution because we found that it was just too much. We tried to start using docker containers and data lake and it was too much for the team to learn. And they didn't even have a full time dev ops person. They were pretty new to cloud. So we step back and we emulated an HPC cluster by using a cloud cluster, if you will. We selected a classic map produced on Amazon and we're able to get the rundown to ours, which was pretty significant reduction because we could use things like spot instances and dispatch. And then we also use this as an opportunity to get dev ops practices into the team. So it would be repeatable and reproducible because it's a key aspect of infrastructure for bioinformatics. When you published, you want to have the compute processes need to be reproducible. So we got the team using information templates and having configuration buckets that were properly stored and we were feeling pretty good. But again, our bias for complexity, we really wanted to get to that next level solution, which we built and got us down to minutes. So we really substituted the EMR. We took that out and got rid of the file storage. We used the data like we used S3 and we built custom containers with Sparke on them and then we use Carbonetti as our controller. The problem with this is that in our own drive to use the latest and greatest, we weren't communicating, working with and listening to the customer and providing them with something that they could really understand and use, although they were able to run it when we were sitting there. We also used TerraForm so it could be run on other clouds as well. The amount of information and the complexity of the solution was just too much for them to maintain comfortably. And I would argue if you look at the diagram, it looks complicated. No matter how many times I tried to simplify it, no matter how many times I tried to clarify it, it just looks complicated because we were successful with that. We wanted in some ways, we wanted to build a full pipeline, even add more processes to it, which we never got to, because the team that was using the solution really was still working with the IMA. So we kind of paused the work. But this was the dream to build a preprocessing section with lambdas to pre-process the incoming genomic data. So what went wrong with this project? A lot went right. We built something that could run in 10 minutes. The problem is the customer wasn't using the ten minute solution. So it's really thinking about this. And one of the aspects we always got on the terraform code was what does this code do? We just learned confirmation. We really can't understand this. And so I wrote a lot of instructions. I tried to record videos and it just was too much for people to understand. And I was really obsessed on this idea of making this architecture. But I really had to accept that the way they were seeing the architecture was like this. It was just too much, too much new information. It wasn't useful. So then I started thinking, based on a different background that I have, my educational background is linguistics. I started thinking, is there something that my team's doing less than optimally in terms of communication? So some kind of way we can communicate differently. So I thought of this quote from Bitcoin Shine, if we spoke a different language, would perceive a different world. And I'd done in earlier in my career, I've done a lot of work around business informatics. And I think business informatics is more mature than cloud architecture, art, cloud architecture. It's been around longer. So this is from information is beautiful from Data McCanless, who's a very famous business informatics designer. I've attended some of his seminars. It's been really interesting. I'm the only developer there, which I think actually is a strong point. Everybody else is a designer. I think there's just really an opportunity for us to learn from some of the other more mature industries on how to improve our visual communication. I was reading a bunch of papers and again, these are all referenced. A lot of them are about understanding the dimensionality of visual space because I'm trying to figure out how to make it less complex and just convey in a simple and effective manner, way that can be reproduced. Another person who I've drawn inspiration from is Brett Victor. He's a technologist, used to work at Apple. He says artists draw because they want to convey something they can't describe. This is really kind of getting a little esoteric, but he's got a bunch of prototypes, executable little mini programs that he talks about, direct manipulation for creating animations. And I find them quite inspiring in terms of a direction we might want to go, especially in building solutions for cloud system visualizations. And maybe you will, too. Also, I did some work in the past in Zambia, in Africa, and one of the things that struck me is this proverb, if you can walk, you can dance. If you can talk, you can sing. There's no concept of your bad singer about Dancer, because, of course, I was feeling I was a bad visualizer, like I'm not an artist. I just can't do this. I'm not capable, which is limiting the thought. So as we well know, if a person wants to improve skill in an area, you simply have to practice in that area. So how do you practice visualisation? Well, for me, it's been through casual photography and sketching. So as I travel around the world, I enjoy looking and taking casual photographs and trying to make them interesting. So here's a trip that I took. Thing I added to it is this idea of sketching. So how would I then draw a triangle? And at first, honestly, I felt silly. But again, how do we improve our code by coding? How would we improve our visualization? By visualizing. So it's kind of a go do for all of you listening here, get a sketchbook and add this to the skills you practice. So another thing I found inspirational is as I try to learn bioinformatics concepts, I read about them. I study them so that I can service my researchers more effectively. But I watch videos and I found this one to be interesting. I'm going to start playing it. Just play it really a little bit of it. It's about eight minutes. And if you're interested, you watch the whole thing. But you'll, I think, see what I mean. When I start playing this now, it goes on and on. It shows a lot of different processes. But I'll tell you, I literally had a revelation when I saw this animation for gene transcription. I had read about the process, but when I saw the animation, I was able to comprehend it because of the richness of the visual environment. And it really struck me that we have a long way to go in terms of visualizing our systems, our cloud systems, because we have nothing like this, as far as I know. So, again, I wanted to include it in this talk as an aspirational goal. We can't all become professional animators, but maybe some of us can. And I think it would benefit the greater community. So in the previous section, I was really thinking aspirational, but I think it's important also to think practically so. One of the aspects of this talk is to talk about some of the techniques that we've applied that I think have been effective, that maybe you can, too. So I think in terms of improving visual grammars for court systems and I think about creating dialects for systems, given my linguistics background. So particular form of the language that's particular to a specific region or group, so a regional language, it develops language, a developer language that's visual, a local language, something that's GCP specific, something that's specific vernacular, something that's bioinformatics specific, so on and so forth. But visually. So one of the first actionable areas is not only to have an entry point or hello world, but to have a visual hello world. You may remember earlier in the talk I said we have this bias towards complexity. And in the variance problem, you know, the idea was to go directly to Building Container's well, we did do that. But what we did first is we created a compelling fun visual executable hello world that had the impact of growing the resources on our team. We made a Jupiter notebook and the team that created a very smart, creative, fun example couldn't use cancer data or something like that because it's for privacy. So they created a what's called synthetic phenotype, fake disease, being a hipster or not, and an genomic basis for it with four different genomic traits, Monaro hair and beard, preference for checked shirts, which is a property of your retina and how you process coffee consumption. And so a fun visual, the info graphic and executable on the cloud. For this particular case, we chose the SAS solution so that you could get up and going with no effort, just spin up a cluster by clicking and figure out if this variant spark algorithm was going to be useful for your research. And we did this because we wanted to get more resources for the team and it really worked. So here's the actual example out on the data community cloud. And to run this, you'd have to just create a cluster just here and it tells you exactly what to do down here. And you could then actually run this interactive demo on the cloud, try it out visually as a hello world. I find this is something, again, with our bias for complexity that a lot of developers and dev ups people overlook. And if you have this, you're going to have much greater adoption of whatever it is that you're building, if that's important in your particular scenario. The next visual grammar that we've had success with is dimensionality reduction, a different project that my team did was to create getting started examples for Here Technologies, which is an API company with mapping solutions. And again, we publish this on Medium. And I had a lot of feedback from readers that they really found my diagrams very understandable. And something I did in these diagrams is something I brought from the business information world, which is I call the three to five aspect diagram. So only three, four or five areas. No more, no less. That's well known in business informatics. That that's a sweet spot. In terms of reports, if you have more dimensions than five people kind of lose the focus if you have less than three, it's not interesting. So I brought that to architecture. Now, kind of embarrassingly, I somehow couldn't bring myself to to apply it to the variant spark problem. This is that second architecture that I had a sense was too complicated, but I couldn't find a way to visually simplify it. Interestingly, when the team that built this went to bring to a wider audience the intermediate solution, the one on EMR, they actually simplified the diagram and they published this diagram on Amazon, which is really a three aspect architecture. And what they did is they published this into the marketplace. And again, they're very innovative. They're recognized here. The team that built this is CSIRO in Australia. They'll need Dr Denis Powers in the front row, Brown here. They were celebrated. They've been the first public sector organisation in the world to publish machine learning based health product on the marketplace. And I think that effective visualization throughout the project, it's one of the reasons not only that they more than doubled the size of the team, but they were able to be successful in getting this reference architecture out there for reuse. Now, I had said in that very Innsbrook project that my team had the aspirational goal to have a more complete pipeline. And so I was looking around for examples to help the CSIRO team build out that pipeline. And I found one in at the institute. It's called Terra Dogpile, and the system runs on Google Cloud. So it's a it's a Web UI that they created to replace the GCP, Web UI that has an underlying language. So it's a very powerful and complex system. And one of the interesting aspects of working with the teams there is to see how they are using visualizations for different perspectives for the user, which in this case is a violent robotics researcher for the developers and for dev ops people. So for the team at the broad and beyond who actually build the terror platform, they have a series of chickpea based architectural diagrams and a technique that I did here is I snipped a corner just to focus on a couple of areas. And this is a dimensionality reduction for a complex architecture. When you want to discuss just one section, you can't zoom, zoom out, literally snip a corner and cut it out as a separate drawing. It may seem really simplistic, but it actually helps focus people's attention. This is one view of their system, a view of their system. For the greater organization at the road is the general diagram, which shows the terra system, what part that plays and how it runs on the Google cloud and some of the key pieces, such as the language and the Craughwell orchestration engine, which is another open source tool written by the team at the road and other researchers as well. Now, when you get to the level of cloud workflows, it's interesting the way the language is capable in terms of configuring bioinformatics tools. And that's those listed here, which I'm sure unless you have BioThrax background doesn't make any sense to you configuring the inputs and outputs. You can think of them basically as a chain of functional tools, take some file in, do some process, put some file out, continue. The language also has the ability to allow the researcher to configure the underlying infrastructure. To a certain extent, they can set the size of Google cloud platform, virtual machines, the amount of memory, whether or not they can use preemptive machines, which is the kind of version of a spot similar, but not exactly the same. But both of these things can be done in a language it's very, very powerful language for configuration and building. But one of the challenges Frode is having is both within for new AI researchers and importantly for partners who are new to the language understanding what the code does. So, again, the role of visualization at the appropriate level. Some of the work that my team has been doing is to help to provide visualizations at the user level, which has been quite compelling, particularly lately, because, of course, in the world of bioinformatics with the global virus outbreak, there's been just a surge in the need for researching to work on this problem. So very interestingly, the terror team was able to put out a pipeline and very proud to say that that pipeline includes a visualization at the user level, which is something that we tried out in their health world, adding these visualizations, because even the simplest hello world has multiple processes and they had some good success with and in fact, we look over this is Tara running and we look over at the dashboard of this pipeline. You can see there is the visualization. And then inside of here we have our workflows. And then they I ran one of them. Just to show you another aspect. They have like the x ray reader that we saw for scan. Tara has a log reader so that you can understand the overhead. This is called the job manager. So, again, interesting to see how this tool has both visualizations at the construction of the workflow and visualizations at the execution of the workflow. Again, I think this represents where we're going as an industry in terms of visualizing cloud systems. So because of the timing, I only covered examples in the first three here of recommendations visualizing entire systems, including the entry points using visual grammars that account for bias and audience and reducing dimensionality and the question period. I'll be happy to talk about examples from the show and verify from work that we've done. And I said that I would explain this picture. So the picture is from the book Gene Machine by Venki Ramakrishnan, who won the Nobel Prize with a number of other people for creating this picture. It's a visualization of the ribosome and they won the Nobel Prize in chemistry because the importance of getting the visualization. It's a fascinating read and it helps to understand how progress in visualization comes into an industry. I found it to be, again, very inspirational. I have all the presentation links here. I'll make the slides available and I have bonus content to the right, which we can talk about in the question period that talks about specific services and products that I think are interesting in the cloud space for visualization. I'm handling it. Thank you.