Video details

Rachel Stephens: WhyPy? Understanding Trends Driving the Usage of Python


How does Python fit among programming languages? We’ll start with a zoomed out view of programming language trends and then drill into Python specifically. Our goal is to explore where, why and how Python being used, and then discuss future opportunities and threats to the language.


Here we go. Rachel Stevens I'm an analyst with a firm called RedMonk, and our job is to understand technology preferences of developer populations and how those preferences help drive technology adoption patterns. So part of what we do is kind of qualitative exploratory conversations with developers and vendors and enterprises. We try to back that up where we can with quantitative data and that usually looks at publicly facing data sets around developer services. And the goal overall is to try to triangulate any available data source that we can get to understand what a developer preference and trend looks like. So I'm going to share some of those. Some of this is going to be qualitative, some will be quantitative, but the goal is to understand developers. And we have actually done this as why do we care about developer preferences? And we have found, and if you're interested in more detail, you can read my colleagues book, The New King Makers. But the goal is to understand and help both developers, and the managers of developers in particular, to understand that developers have unprecedented influence. And part of this is because hardware and software is accessible in a way that it's never been before, and developers don't need permission like they used to. So if you think back to early two thousand s and you had to go through an entire procurement process to get something racked and stacked, and you have to go through the process of getting licenses, and that's just not the case today. In a lot of instances, developers have more choice. Part of that choice is driven by the rise of open source software where people can kick the tires on things, bootstrap projects, get things started, and then present the case to their managers about why they may or may not want to use the service without having to go through the entire procurement process. Part of it is driven from cloud technology, where we have eliminated the need for upfront capital expense to get projects started, and we've made new classes of hardware virtually accessible and on demand resources easy to procure. And part of this is our ability to collaborate as communities more effectively. We can share best practices, we can transfer our knowledge, we can see and share code, and we can collaborate. So all of these things together have meant that there's been a recalibration of practical authority in technology choice. And technology adoption is increasingly driven from the bottom up. And this has ramifications for the technology industry overall. And that's why Redmunk is really here, is to help people understand what those developer and practitioner preferences are and how that impacts what's being used in the space today. And language choice is an interesting thing for RedMonk because one of the primary ways that developer populations identify with the world is through their programming language of choice. And so choice of language has kind of profound impacts on developer adoption patterns, not only within the languages Themselves, but Also In The Tools And Frameworks that Kind Of Exist Around Languages. That's why I'm super excited to talk to you about one of our projects. And I realized that this is super hard to don't have to worry about it. Python is up there on the top right. But one of our big projects that we've been doing since 2012 is trying to attempt to correlate language usage and language discussion. So we don't view this. And I'd like to reiterate, this isn't like a definitive ranking system. It's just trying to go into publicly available data sources via GitHub and via stack overflow and see what's being used, what's being talked about, and how do those correlate together. And then what we found is that people don't like that kind of wishy washiness of saying, okay, we're just correlating. So we ended up, by popular demand, actually doing it as a ranking system. And you can see Right Up there. Python has been top five for the duration of this analysis and is currently Ranked At three with A Steady trajectory. Way To Go, Python. So I think that it's interesting. Every time we run this and we try to run it two times a year, we get a lot of questions. Some of these questions are you use stack overflow to measure discussion. And that means that the languages that are harder to use are the languages that get more questions asked. And that means that there are problems with your analysis. And We Don't Disagree. We get questions where people say, oh, like my community does all of their work via slack. And so you're not capturing the real spirit of the community if you're only looking at stack oversville. And We Don't Disagree. But it's an imperfect measurement system, to be sure. So take it all with a grain of salt. But this is just trying to look at two really broad developer communities. What Do We See? And we see Python as a healthy and engaged community. Well, in theory, that's a picture of a sailboat. I wanted to talk about headwinds and tailwinds in the Python community, and this is a discussion around why Python has done so well, and some areas where other competing languages maybe have different things that are helping them do well so that we can kind of understand and compare and contrast overall communities at large. So we're going to start with some things that Python has going for it, and one of it is its extensibility story. So rather than having all the functionality built into its core, python, by design, was highly extensible. It's compact. It's got a small core and a large library. Easily Extensible Interpreter. You have the joy of the pip install, which may make your life easier. Or may make it harder. It Depends. But it's something that has been designed so that the language can be very general purpose. We have traction in areas like web development and scripting and data, science and machine learning. But really what's cool about Python is the extensibility allows people to meet their needs through the language in a way that a lot of other specifically designed languages can't do. So the general purpose language means that people can do what they want with Python, where they need it. It can be used as a glue language, it can be used as a scripting language, it can be used all over the place because people have the ability to have the language meet their needs. And this leads to mastery. I think we talked about design of Python this morning. And one of the things that is amazing about Python is that it has this minimalist philosophy. It's simple and readable. It has a way that a lot of programming languages view cleverness in a positive light. That's not the case for the Python community. We want things to be comprehensible readable and this gives people the ability to read what their code is doing. And it gives people the ability to come into the programming language community much easier. Python is routinely one of the languages that's used as a first programming language learned. And that's because it's something that people can understand. So it's not just machine readable, but it's human readable. This gives people the ability to get to a flow state. You can code in fewer steps, less code is required to get to a solution. And there are prescriptive ways to do things so you are not guessing at what you should be doing. There are ways that you can do this in a way that makes sense for people. And all of this combined leads to an approachable learning curve. Choice. I feel like choice is not something that is typically associated with Python based off of that Zen of Python thing where we have ways to do it. Ways that people enjoy solving problems is usually autonomy, independence and creativity. And I think that when we try to have a more prescriptive language, sometimes we can forget that people have a lot of choice. But one of the things that Python has going for it is because it's so general purpose, there's a lot of choice and coding methodology. We have like object orientation, we can use it in a procedural way, we can use it in a functional way. And so even if it's not something where you're going through and you are coming up with clever solutions, because clever is not the thing that Python code should be. It should be simple and beautiful, but you can be really oriented to your code in different and flexible ways, which I think is something that Python has going for it. And then the community, I think the community around Python is uniquely amazing. You have such a positive reputation in the world and the programming language communities can sometimes be hostile to newcomers to different ways of approaching things. And the Python community overall has a very positive reputation of being friendly, being welcoming, being a place where people can grow. And that's not just me saying good things to all of you. Amongst programming language communities, you are viewed as one of the communities that is friendliest and most approachable for people. So that's all of the things that Python has going not all of the things, but some of the things that Python has going for it has kind of tailwinds that are pushing the language community along. Now we're going to go into the part of what's happening in the world around Python and what is causing people to choose something other than Python when they're creating projects. And in the spirit of understanding that sometimes our greatest strengths are often our weaknesses, some of this is probably going to look a little familiar, extensibility. I think one of the things that we see at RedMonk all the time is that the best tech doesn't always win. I think we tend to think of programmers and engineers as highly rational beings and we're going to pick the biggest test, it's most reliable tool and we're going to find the exact right tool for the job and we're going to choose the technologically superior solution. And what we see time and time again is that's just not the case. The best tech doesn't always win. And it's not just what we're delivering, but how we're delivering it. So how things are packaged and how they're consumed and how they're accessible are extraordinarily important elements of adoption. And the library compatibility and dependency management story of Python is notoriously difficult for people to manage. The compatibility is a huge thing for making the language general purpose and making the language do what people need. The complexity that comes with that is a cost and that is something that is hard for people to manage. And so there are definitely cases to be made for people who are trying to get started, being able to get started with something that's a little bit easier to get set up sometimes and that ties into the mastery component. So I saw this the other day for remembering people who are getting started in the school year and you're just getting started in your computer science class and you're doing that first assignment and you can't even get all of your Python libraries installed correctly. And does that mean I'm not going to be good at computer science? And maybe, do I need to pick a different major? Like there is this thing of trying to bring people on board and recognizing that it can be really hard for people to get through that first step. And Python sometimes does not make that easy for people. So even though it's a fantastic first language to learn and it has a learning curve, once you get started, the environment issues that people encounter getting started in Python are not insignificant. And so anything we can do to try to help people get through that first set up thing is going to be an amazing thing for everybody in terms of both building the confidence of people who are coming into the language community and also making Python something that's more easily approachable for, as it also says, professional, very experienced engineers alike. That's something that has benefits all over the place. And I love this phrase. It's from Andrew Clay schaefer. He was with Pivotal and it's minimum time to dopamine and part of it is like a yak shaving story. How much time are you spending doing fiddly tasks that kind of don't bear a relationship with what you're supposed to be working on? Or sometimes it's thinking you're doing the right activity but you're focused on the wrong thing. I think one of the things about dependency management is it can feel very shady for a lot of people and so that minimum time to dopamine is like how long does it take for you to do something cool? How long does it take for you to get started and get excited about the projects that you're working on? Is something that I think is great to keep in mind, not just for programming languages in general, but all of our projects. Like how quickly can we get people to the point where they're feeling like they're successful with the technology and that's huge. And so I think that's part of the dependency management store. But also just like a broader narrative that I think is really important to think about in tech is helping people feel successful and this ties into efficiency. Python is notoriously not the speediest language, it's memory consumption and resource footprint, especially multi threaded, are not the best. The thing about interpreted languages is that the gap between an interpreted language and a compiled language is getting smaller as hardware improves over time, it's becoming less of an issue. That said, Python was designed in a world that was not a multi threaded world and the global interpreter lock, sorry about that, it's made it hard for Python to take advantage of concurrency. And one of the things that we think is interesting I'm not sure if I have a definitive position here, but you hear a lot of chatter around the end of Moore's Law and chips are going to start reaching a point in which we're just not going to have the same speed price trade offs that we've seen in the past. And a lot of the new improvement in computing is coming around multi threading. And so what does that mean for Python? If the way that we are improving our compute power changes how the language actually interacts with the hardware underneath, I think that's definitely debatable and I'd be happy to talk with any of you after that, but I think that's definitely an area where sometimes it leads people to move to a language like Go or something like that, where it's designed to be more multi threaded. None of my photos are going to load. So this is a picture of a beautiful fashion show. One of the things that I think is interesting is we as a community tend to think of ourselves as very rational, but we are very driven by trends. And I think anyone who has heard the word Kubernetes is probably going to agree with me there. We have things that tend to take over the industry in terms of our attention span and in terms of how we think about things. And one of the things that we have seen at Red Monk is a resurgence in statically typed languages. So if we look, I'll show that chart again, but statically typed languages, especially ones go TypeScript and rust, are ones that we've seen a huge amount of community traction for. It means that the errors, dynamically typed language errors, show up at Runtime, and we're not enforcing data. Type safety can make it a little bit faster to debug and spot errors. And type safety is something that we've seen in a lot of our fastest growing language communities. So just something to be aware of. And then finally, choice. I think one of the things that's interesting, and we admittedly don't see this a lot, but we see it some, I'm sure some of you, I'm sure everybody in here knows somebody who has refused to upgrade from 2007. Is that accurate? The thing that's interesting is the lack of backwards compatibility between three and two seven, and the rise of more specialized languages means that there are some shops where they're starting to refactor apps and it's not a guaranteed upgrade path to stay on Python in that instance. So sometimes people are refactoring two different languages rather than refactoring Python 27 to Python three. So just something to be aware of. But again, it's not a common use case, but something that's out there. So I wanted to talk, given that Python is such a general use language, there are any number of languages that we could talk about in terms of alternative language choices. I just picked a handful, but we can talk about a whole bunch of them. As a user of ours, I don't feel too bad about this because this made me laugh. I think that there's a lot of choice and I think one of the things to remember about technology is technology is just trade offs. That's the discussion that should be happening anytime that we're making technology choices is that the world is not simple black and white all the time. It's nuanced and it's a discussion of trying to understand what you're trying to accomplish and what you're trading to get that this is the chart. We'll maybe come back to this a few times, but the language I want to talk about first is, for better or worse, very entrenched in the academic community. So they have a clarity of purpose and design and then a user base that's getting people started early. So I think a lot of the time if you're taking a stats class in particular, you're probably learning R rather than Python. And you can see some of this in the number of libraries that get ported between the languages. We see more coming from R into Python rather than vice versa, like the ggplot World and it's moved over to Python. We haven't seen quite a similar and significant movement backwards. And so I think it's interesting that some of the functionality of Our is something that people are desiring in other places and that's something to be aware of and something that you can do like Python is extensible. So if you want to start moving our users over, maybe it's some of those library supports that we can start building into Python TypeScript. TypeScript is one of the fastest growing languages that we've seen. So it's right there at number twelve and it has skyrocketed in the last couple of years. And some of the things that have driven that is that it has optional type checking and it's dropping compatible with JavaScript. So it gets to take advantage of the ubiquity of the JavaScript environment with some of those types of people features. And I read an article earlier this year from Guido where he was doing an interview where he expressed the desire to have some of that similar optional type safety explored in the Python world. I don't know where that project stands, but I do think that that would be something that would be interesting for language communities to explore given the current trend around static typing. And then Go is an interesting one because it's designed for a specialized use case. It's fast and it's flexible, it's kind of back end design, start to see chatter around it being a scripting language. I'm not sure where that has come from. I think of it as a backend language. It has a lot of the same library compatibility, dependency management issues that Python has. And it also has an added concern around kind of the independence of the community from the vendor, it being a Google based project. So I think that it's interesting. I think the community has its own challenges, but it does come up for us a lot in terms of people who are looking for something that is fast and flexible. So those are just a few we can definitely talk about more if people have questions. And then last thing I wanted to highlight as a potential area of concern is community. And I know I just told you all great things. I think that they're at a broader level with some concern about Guido stepping down as benevolent dictator for life, particularly the way he stepped down because of his concerns around how to drive the future of the community and some of the different competing forces that were frustrating to him. I think that there is some concern about the future direction of Python and what that means for where the language is headed. And again, I'm not going to have a picture. This is a picture of a mentos truck about to crash into a Diet Coke truck. The goal is I don't want anyone to think this is the end of the world. It's not about to have the Python community fall off the chart. You're ubiquitous, you are powerful language, you are flexible, you are general purpose, and you have a really devoted set of community and a devoted set of enterprise, which I use in a loving way. Lots of people don't use enterprise lovingly, but having an enterprise base, python is not going anywhere. This wasn't the sky is falling talked, so please don't take it that way, but I did want you to just be aware of the broader landscape and what's going on around the language community. And Python is beloved. It's going to be here for a while and we just have to think about how it is that we can move together to help drive the language forward. And I think part of that is thinking about what we can do. It can feel kind of arbitrary to talk about all these broad landscape things to a group of people who are here on a Sunday. Way to go. But what you can do is you can beat that community. I think long term, like sustainable competitive advantage rarely comes from technology. It comes from people and it comes from the community of people here. And I think that you are a big part of what makes this language usable, what makes it approachable, what makes it fun for people, is all of the work that we do to bring other people along with us. And so that would be my call to action for all of you today, is to continue being the community that people want to join. And I would be happy to take any questions. What types of customers does Redmond have and what are the most important insights that they're getting out of your reports? Yeah, so we're kind of a mix of like a research firm and a consulting firm. So we consult with technology companies. And the reason that they want to consult with us is because we do research. We have customers that range from single person technology startups all the way up to the hyper scale cloud providers. We have a wide range of people, but in general, it's trying to help everybody understand not just how to I think high level is how to talk about your technology in a way that's compelling to people. And different people need different messaging and so it's trying to help them. How do I connect with the developer community versus how do I connect with a middle manager versus how do I connect with the CTO? It's going to take different tactics to reach those. And so it's helping people kind of understand how to do that. Great talk. Thank you. My question has a long lined up. I promise to obey the Brandon Roads rule. Redmont gets to look at lots of languages and then lots of segments where languages are used. Russell Keith McGee gave a talk at Python about black swans. Lucas LONGi gave a similar talk at Pylon Dinner about Python 2020, and both of them said a number of things, including Python might not be competitive on modern web. Without getting into what that means or whether it's true, if it were true, how impactful would it be to Python's position on your graph if it became less used for web? I think one part of that is that modern web is increasingly related to mobile, and Python does not have a notoriously great footprint on mobile. And so I think having a better mobile story would be tremendously helpful for that web story. And I think that it's interesting because it's multipurpose. But I definitely do think of Python more as a back end language, more than a front end language. And so I think things like TypeScript, where you're going to have some of that interesting interplay with JavaScript, I think that you can see TypeScript having a clearer future story than something like Python in the web space. But again, I don't think Python is going anywhere overall. It's just one of those things where it might be more of a challenge. I can't believe nobody is going to call me out on CSS. Being in the language ranking is expecting to have to find out. All right, thank you all.