Video details

Unleash your inner CTO! - Rob Crowley - NDC Sydney 2021


The role of a technology leader has never been more challenging. We are in the midst of unprecedented technological change and yet keeping abreast of such advances alone will not guarantee success. Faced with such myriad options how can we be sure that we are guiding our organisation on the path towards future success?
In this talk, we will take an in-depth look at what it takes to create an adaptive technology strategy including metrics, team topologies and evolutionary architecture practices. We'll explore how to build feedback loops into your organisation’s technology decisions and ensure that its architecture is an accelerator rather than an inhibitor of business agility:
* How to identify and measure key business-aligned metrics and use these data points to pivot your technology strategy. * How to enable effective distributed technology decisions across your organisation * How to build organisational memory surrounding technology decisions that can act as inputs into new decisions.
By the end of the session, you will not only understand what is involved in creating a data-informed technology strategy but also have the knowledge to put it into practice. If you want to take to your expertise in crafting effective technology strategies to the next level, then this is the session for you. See you there!
Check out more of our featured speakers and talks at


Hello and welcome to this breakout session at NDC 2021, where we'll be looking at what it takes to think like a CTO. So many of the talks that this year's conference will focus on the myriad new technologies, frameworks and products available for us to use this talk, we're going to make it a little different. And over the course of the next hour together, we're going to learn how we derive value from this technology and use it to yield valuable business outcomes. So it's really about giving it purpose. And our agenda today is pretty ambitious. We'll start with a primer on strategy and not too gentle primer. We'll review some of the key considerations involved and cover several models that can assist with generating strategic insights. These are really your toolkit to take back to you to work on Monday. We'll also look at techniques that you can bring to your jobs and start building a reputation as someone who can think both deeply and laterally around a problem from multiple perspectives. Additionally, we'll look at technology strategy from three perspectives. First, we'll explore what it is, its purpose, and how it relates to business strategy. With this base level of understanding. We'll then explore an approach for defining, communicating and evolving a technology strategy everything you need to first build that vision and then deliver on it. And finally, we'll look at one of the most important aspects of a technology strategy, and that's the decisions and the actions that underpin it. So before we proceed, I'd like to share a little bit about myself. So I'm the head of Engineering at VIX Technology, and at VIX we build transport solutions for cities. As a point of reference, the Mikey Card in Melbourne is one of our previous solutions, and Vicks is all about connecting people and places and building a simpler customer journey. Previous to this, I was a principal consultant at Telstra Purple for a number of years, where I worked with company CIOs and CTOs to define their technical strategies. I'm also a co organizer of DD Perth W, a largest It conference. And as always, I am in awe of my fellow committee members and the NDC crew for the amazing conferences they put on for the community. And I'm also available on the Socials at Rob D Crowley. And I also have a confession to make. I absolutely love shiny tech, and there's nothing wrong with that. But my hope is by the end of the talk, I may have helped you to look at technology with a slightly different perspective and giving you some techniques to help you assess the suitability for the user or business outcome you're looking to achieve overall will give it purpose. And so I've used that term strategy a number of times already in this talk. But what exactly is it? It's often a nebulous concept that gets bandied about in multiple different contexts. So when in doubt, let's consult a dictionary and our friends at Macquarie Dictionary tell us that strategy is a plan of action designed to achieve a long term or overall aim. And from this definition, we can see two things. Firstly, we can see it's about making decisions and acting upon them. Secondly, that it's an activity that is performed to achieve a long term vision. But there, I've done it again. I've just introduced another new term, vision, and one that often gets mentioned in the same conversation as strategy. But how do they relate? So we'll look at purpose, vision and mission and the connections and the differences between them. So the purpose is why you do what you do, it's why your organization exists. And at VIX, it's about connecting people and places and it acts as our true north. At worst, a company's purpose is a line on a wall or in a document. To derive value, it needs to resonate with colleagues and act as that true north. That spine around which you then build both your strategy, but also base conversations and actions within your organization. Division is a statement of intent. It's what you want to achieve in the future, the Lighthouse, the market that your company will make on the world, the measurable impact you wish to deliver. And the mission is what you will do to build or achieve your vision. And the mission should be actionable and concrete. At VIX, our mission is to make mobility seamless. So we talk about simplicity, removing friction, both from a transit operator, but also from a rider's perspective. And you may be thinking, I'm not the person in my organization that defines the vision or the mission. Don't worry, you can still implement a brilliant strategy that serves both. And I've shared a little bit about VIX's strategy, but we can also look at other luminaries in the industry. So Microsoft's corporate vision is to help people and businesses throughout the world realize their full potential. So quite high level, but incredibly broad in its focus, which again matches the many different offerings and breadth of Microsoft as an organization. And then their mission is to empower every person and every organization on the planet to achieve more. But as we will see as we go through strategy, the strategy cannot be that high level, fluffy terms that make people feel good but don't drive change. So we'll look at how we build the extra meat, the extra detail that will then make the change and fulfill the outcomes we used to deliver. And that's the key thing. The strategy uses the mission to achieve division and that's the link between them. But the conversations can stop or start at purpose and vision, even the mission. We need to delve deeper and do the hard work to understand the challenges that the organization is facing, the potential opportunities that it can disproportionately invest in to achieve and the ability to sense and react to this is absolutely vital to effective strategy. Because strategy cannot be created in a vacuum. Without context, it is impossible to differentiate between beneficial or harmful decisions and the actions chosen to realize them. Feedback loops are absolutely key, and the ability to sense and respond to changes in the operating environment is called situational awareness, and it represents the ability to discover, communicate and learn about changes in the environment. And in this case, the environment is the operating context in which your business exists. So it's both the business itself, it's all of the competitors, it's the industry in which you've all this your customers, it's all of the system of components and interactions with which you operate. And it's again sensing where you exist in there. And is there a certain course of action that you know need to change? And likewise, just like the weather, we can't control it. New innovations are created, competitors can launch new products. Global Pandemics can rapidly change the landscape in which we operate. Even today, this is still an online conference after COVID, we have still not been able to return to what we considered normal just a few years ago. And often our best laid plans can be ruined by factors completely outside of our control. And another aspect to consider is as the environment changes, the ongoing value of previous decisions can decrease and perish. But Conversely, there can also be a boom. They can also increase in value. For example, WeWork had a massive change in the business context as a result of coverage, which dramatically impacted the occupancy of their communal spaces. However, their costs remained relatively static as they still had to service the leases they took from the landlords themselves. Conversely, the Pandemic has provided a huge financial boon to video conferencing companies like Webex or Zoom. And it's not just in that pure technology space. Coconut has a massive impact on payment approaches across the transit industry. It's been a catalyst to remove cash away from the systems, particularly in the US, and it's been a huge driver in adopting EMV so credit card payments. And this also opens the door for many opportunities, such as aligning payments via credit cards not just for travel, but potentially toll roads or parking, or even centralized government accounts. So from one particular change, a host of new options are presented to providers and we can either sense and respond to them, or our competitors will. And competition is a key aspect of strategy. Competition drives evolution or to reposition this. Without competition, stagnation will arise and not all businesses will adapt to changes in the environment. At the same time, as William Gibson said, the future is already here, it's just not very evenly distributed. So, for instance, some companies embrace cloud computing early on, and shifted from treating instances as pets to cattle, moving away from cutely named servers after Simpson or Star Wars characters to ephemeral compute layers. And that was really the first stage of I eyes. But it didn't stop there with the advent of serverless and pay as you go pricing, it's evolved from a differentiator to a pure utility, just like electricity. And how a company responds to environmental changes can vary dramatically based on their position in the market. So let's turn our attention back to the early 90s 90s for a case study that demonstrates that being first is not always a predictor for ongoing success. In the early 90s with the first browser, Warnescape began with about an 80% market share and a huge amount of goodwill from the public. As a relatively small company, most of its income came from that single product, and as such, it was financially vulnerable. And the company's total revenue never exceeded or even came close to some of the Giants in the industry. And so Microsoft sat back and they looked at what they could do with this new advent of customers who had just gained access to a potentially life changing experience, the Internet. And so Microsoft's resources allowed them to make Internet Explorer available without charge because they knew they had a 90% share of the desktop operating system. So a huge amount of leverage, and they were then able to use the revenue from Windows to fund the development and marketing around Internet Explorer and making it free to 90% of the Windows and Macintosh uses. Unlike Netscape, which was free for home and educational purposes but required a paid license for work, was game changing. In addition, we had customers that were very new to the product. They weren't savvy in terms of knowing the feature sets that really differentiated a great browsing experience from a poor one. Some of us might remember building websites where we had optimized for Netscape or optimized for Internet Explorer. There was still a very much emerging set of behavior here, but Microsoft ended up winning that first war quite easily, not because they had a better product, but because they had a stronger market position and were able to out leverage their competitor. And in many cases, we then saw wouldn't Microsoft continue in the ascendancy. But at the end of the first Frost of war, we saw a period of stagnation. They stopped investing in improving Internet Explorer at the same rate. And this ultimately led to the dominance of Chrome, which exists to this very day. And we often think of Apple as being one of the most innovative companies in the world, and I own a host of Apple products. But the Apple strategy is fundamentally based on a fast follow up policy where they look at what else is happening in the market, but then they really deep dive into what's the experience that needs to be delivered. How do we iterate? How do we improve it and then come up with a product and experience that blows away the competition? And with near full vertical integration across their supply chain, they can do things which other suppliers can daily dream of. And again, that's a massive advantage that Apple bring to all of the products that they build. We'll now look at another facet of features and customer satisfaction and we'll introduce the canoe model. And this is typically used as an approach for prioritizing features on a product roadmap and measuring the degree to which customers are likely have satisfied by them. Here I'll use it to illustrate the movement of expectations over time, how customers expectations evolve and attitudes change over time. When we're looking at building features in the top right hand Quadrant, we're really looking at the differentiating products. So the ones that we can build that will Delice the customer and then in the bottom left hand corner, the more the things that if we don't have them, your product will fail. And the trick here is that customer's expectations change over time. So what was previously a game changing feature. So face recognition when mobile phones have been able to unlock it? A number of years ago, that was the key feature that was differentiating phones. Now it's just taken as a given and it's the same way that in many cases, the decisions and the innovations that we made before become commonplace. What was new becomes a utility in time, but this in turn enables the creation of more complex innovations. The commonplace enables the novel and to go back to the example of storage and compute. The commodification of storage and compute then enabled a new wave of innovation around big data and machine learning that would never have been possible if we didn't have that previous foundation to work from. And in your organization, you're almost guaranteed to have areas of your business that are across multiple of these stages, some mature areas of the business where you're looking to build efficiency and maximize the revenue, and also other areas where maybe you're trying to find product market fit, you're trying to explore an element of disruption. And we need to be careful to choose the right method for supporting that delivery. In each one of those contexts, a one size fits all approach is suboptimal by definition. So to expand on that, if anyone ever says that an agile approach is universally better for all scenarios, that is a broad brush statement that lacks the nuance and context to enable it. So let's expand on that a little bit more and we'll do this via the three X model by Kent Beck. And during his time at Facebook, he defined this to try to find a sweet spot in terms of applying the correct methodology or approach at a point in time, and also how that maps to both organization and customer success. So on the y axis we have pay off, which is producer perspective, does this work for the business? And on the X axis we have success, which is from a customer perspective, does this work from your user's point of view? And in the beginning we're looking to find product market fit. We're building to learn. We're exploring. And our goal at this stage isn't necessarily to maximize revenue, but it's to find where the fit or the need, Where's that opportunity that we can leverage. And we're seeking to eliminate uncertainty. And in this context, agile is absolutely wonderful because that's what it's designed to do. It's built to amplify feedback loops and eliminate uncertainty. As a product matures, we then see our focus change because at this stage, as you have multiple customers coming on board, the cost of any inefficiencies gets magnified. And so we can start looking at how do we eliminate waste. And in this context, the principles as defined from Lean around the elimination of Moodle or waste are incredibly beneficial. So it's about scaling out what we have built before once we have found that product market fit. And then finally you can have probably the core offerings of your company very mature huge amount of revenue, but at the same time might not be where the market need is going three to five years from now, but at the same time, you want to extract the maximum value from it today. And in these scenarios, Lean six Sigma, which is about elimination of defect or variability, can help with that because we're no longer seeking to learn so much as to maximize our earning potential from us. And there's also a corollary to this whereby the risk appetite of the organization diminishes over time. At the start, we've invested very little. So our risk appetizer is quite high. But as we mature and the product becomes core to the revenue line, our risk appetite can drop. And this is in many cases we saw with Blockbuster, with their potential acquisition of Netflix, which they turned down all because of late fees, they were too tied to the current revenue stream to be able to look at different options. And this is a tragedy because large organizations can die from a series of reasonable decisions such as this. And in the book how to Lead a Quest by Dr. Jason Fox, he introduces a couple of really nice models, but also a metaphor for what the face that awaits companies who stick to their tried and trusted products without looking to innovate or read where the market is going. And he calls it the cracking of doom. So again, a giant, monstrous creature awaiting any company who isn't going to change and adapt as required. And one of the models introduced by Jason in this book is what he calls quest augmented strategy. And he highlights that at any point in time there will be multiple kinds of work that you are looking to progress simultaneously within your organization. The first of these is missions, so that will be the core offerings that you exist today. You have already found product market fit and you might be looking to expand or maximize the potential of that. And in this scenario, there is a relatively understood line of causation between the strategy and the decisions that you're going to make to progress it and ultimately the actions to drive progress. There is a well understood link between those two. But then there's also the more emerging of pioneering work which he refers to as quests. And here we're really looking to build a set of options hypothesis and test them as quickly as possible. So in this scenario, we start by reading the market, we look at outside influences, we consume information, we synthesize it and seek to understand what are the options are potential points of imbalance that we can leverage either from our market position or from our IP or potentially a misstep that a competitor has made. We then form a set of hypotheses which we then look to prove or disprove via experiments. And our goal here is the same as the three X model we are building to learn. And then once we have generated these business insights, they can then form part of a strategy to define how are we going to deliver or capitalize on that massive AHA moment which we generate through the options and experiments. And one of the key points with strategy is that effective or good strategy has to be actionable. There has to be a strong plan of decisions and guiding policy that then guides and builds coordination across an entire organization towards a coherent set of goals. So as John Boyd eloquently puts, decisions without actions are pointless, but actions without decisions are reckless. And the next model that I'll introduce is possibly the single most important one in today's session. And this is known as the UDA Loop. And this is really when John Boyd, who was a fighter pilot instructor, this is what he instructed or taught the fighter pilot again initially for dog fighting and it was about quickly being able to situate yourself in a scenario, absorb the information, then generate a set of options and ultimately make decisions. And his hypothesis was the fighter pilot who could do that, the quickest will ultimately emerge victorious. So it is absolutely a cycle. And in the first stage we are looking to observe. So this is sensing the outside information. So the situational awareness that I referred to earlier, once we have observed and positioned ourselves, was then about orientating. So analyzing this new information and previous experience to generate a set of options, then it's about understanding these set of options, prioritizing and grading them, assessing them, and then providing feedback to ultimately make a decision and then acting upon that decision. And that could be testing, probing or again applying all of the previous delivery experience from your organization. And the key takeaway here is that strategy is not a linear process, but an iterative cycle and in many cases an organizations ability to iterate on their strategy more quickly is far more valuable than having an optimum strategy at a point in time. Because again, the context in which we operate is not static, it is constantly evolving and evolving beyond our control. But it's very easy for strategies to become nothing but words on a page. A good strategy is not just a neatly formatted template with a purpose, vision, values and a laundry list of desires. All too often this is what companies create, which is a massive shame as it fails to express the key insight to leverage and the obstacles to overcome. So a good strategy includes a set of coherent actions. They are not mere implementation details in a strategy. The actions are the punch of the strategy, and a strategy that fails to define a variety of plausible and feasible immediate actions is missing a critical component. And those feasible actions is something that needs to be achievable, not just a sign on a Hill, but with no real understanding of what are the impediments, what are the obstacles that we would need to overcome, what are the concrete things we would need to do differently? So again, it's absolutely vital that all of the actions that are defined in the strategy should also reinforce each other. And leaders must do this deliberately and coordinate actions across departments because bad strategy is just a list of priorities and at worst, they may actually even compete with each other. And finally, an effective strategy is as much about saying no to certain courses of action as it is about saying yes to your chosen business requirements. It's about being selective and being selective around the key insight or objective that you're looking to achieve, and then how you remove the obstacles and impediments towards it. And at the same time it's then there might be a viable second option, but you choose no, I cannot spread my capacity across multiple initiatives. We're going to focus on one or set of them. But again, many companies have a lot of work put into their strategies, but they say yes to everything. And in turn, there's a tragedy of the Commons. When you say yes to everything, you deliver nothing. And so to wrap up strategy is really an informed opinion about how to win. It is subjective, there is no way to prove the single best strategy at a point in time, and it is constantly being updated based on an Iterative approach with high context feedback loop. And whether you can see it or not, competition is driving constant change across your business environment. And today's novel is destined to become a commodity. And situational awareness is the key factor in identifying this. Leveraging models such as the Udaloop and Three X to promote deliberate thought are absolutely critical and can provide a great accelerator when you start looking at these initiatives. And finally, strategic thinking is a skill and Consequently requires practice and dedication to master. And so let us now turn our attention to technology strategy. And technology strategy is a collective set of decisions created to guide the use of technology and related factors to achieve a business or end user outcome. A technology strategy exists to amplify a business outcome. Or to put it another way, if your technology strategy does not make it easier to deliver your business outcome, it is a bad technology strategy. And when we look to communicate or define a technology strategy, I strongly recommend writing it down and making it accessible. And this is vital to building alignment both for the colleagues in your organization today, but also potential new joiners over time. And I was initially somewhat dubious of this. But when I was at Telstra Purple I did an engagement with Bank West to define a technology strategy around the use of APIs. And by taking a long form approach and defining it, it was incredibly powerful to build both stronger engagement, improved feedback and ultimately a better outcome when it came to the actions and delivery side of it. And so there are multiple facets and multiple areas that you need to think about when you're defining a technology strategy. And it might be surprising, but when you start with a technology strategy, you don't start with technology. As I mentioned before, it is about amplifying the business outcomes and the technology strategy exists to deliver or better enable the business strategy. So what we're looking to set here is again the purpose, vision and alignment around it. What are the major opportunities or insights that the company is looking to capitalize on? What leverage or sources of power does the company have to which they can bring to bear across these obstacles or opportunities? And setting this context up front with a clear articulation helps build the alignment across all areas of your business. With that base established, we then look at the technology landscape and this is where we look at the challenges and the limitations. This is where we make it real. So we look at to achieve that particular outcome. Here are the challenges that are facing us today. So again, if I was to define in maybe one of the big four banks, we are going to go completely serverless with all of our work and they're going to go Rob. Our compute loads can be all serverless. What about our core banking system? So again, it's about understanding the limitations but also then saying how can you adapt to still maximize the outcomes versus what do you do to achieve or overcome the problems that are facing you and you can't ignore them? Bad strategy ignores these challenges. In lieu of a nicely worded single document that looks great to execs but doesn't actually deliver real outcome on the floor because it can't be delivered. But we also need to set what good looks like what that Lighthouse on the Hill is. And we need to set a set of guiding policies to then enable effective distributed decision making across our teams towards those goals. And that's where the tech vision and the platform principles and guardrails are absolutely key because you can centralize decision making about this. Why hire great people and then tell them what to do? It's about enabling that context to flow smoothly to the edges of your organization and then enable effective decisions to be made locally within those teams. And to make that a bit more concrete, let's look at an example that was created by Simon Wright Allen during his time as being CTO. Myob and their platform manifesto was written in a very similar style to the Agile Manifesto and here he lays out his guiding policy around the decisions and behavior that he wanted to see in his teams at that point in time. So reading some of them mediumterm decisions over shortterm thinking, nurture the platform over feature delivery, existing assets over new development. And what we quickly see here is that Simon is seeking to build alignment around really the elimination of waste in their product line. If you think back to the Tree X model, they're in expand, they already have a core product, they've already found product matter fit and now they're looking to capitalize on that and make decisions that go towards the medium term. So the long term sustainability of their product. And the inference here is that you need to understand across your organization, across your product streams whether you really want teams to use the Facebook terminal, move fast, and break things, or again, is their context, one that you really want to eliminate either waste or variability. And these will change over time as well. And the key thing from here is reading that manifesto alone, we can see that MYOB was urging their teams to make decisions and actions that are focused on the medium term view. Another aspect that I have found quite successful with teams is around the tech radar to visualize the technology, landscape, platforms, tools, techniques, languages, and also then be able to create an application lifecycle management around it as well. So what technologies are being experimented with, what one should go and adopt? Also, are there certain assets that we wish to sweat but no longer invest in or actively retire? And with a number of clients? I have used the Roll your own radar by ThoughtWorks. It has worked incredibly well with some organizations and not so well with others. A lot about the level of buying you can get from the teams, about being transparent in what they're doing. So your mileage may vary, but again, I have found it to be a wonderfully powerful option. So I do encourage you to assess it. And the last aspect is the lightweight governance around how you make decisions. And this is really if you're familiar with the architecture decision record pattern, it's really about building transparency around the technology decisions that you make. So what are you looking to achieve? Also defining the autonomy level within your teams. So how many decisions do you want to have made locally within teams versus the ones that might require a change to a contract agreement or an out of hours, or a large change to infrastructure costs or other costs as well. So then you can choose the autonomy levels about how much you want that to happen at the edge of your organization. And I found this to be a wonderfully powerful tool for building empathy around decisions. So it might be once you've defined your set of requirements, you've assessed your options and there might be a clear winner that you'd wish to choose. But for either cost or capability reasons, at that point in time, you've chosen a different, maybe possibly more pragmatic option. New team members can read these decision spaces or architecture decision records over time and understand the major decision points that the team have gone through to arrive where they are today. So wonderfully for building empathy across new joiners. But then also about building alignment around discrete thinking that has happened over time. And remember, strategy is iterative do not aim for perfection on the first pass time box. It seek feedback and repeat. So during the engagement of Bank West, I ran this as a four week initial period with feedback sessions, and then also review points after that. But the goal was get something out in the first number of weeks and then improve it as we go through. Rather than disappearing for three months and trying to create the perfect document again, it builds more buy in, and it also increases transparency. And strategy is not static. Don't treat the strategy document in a static way. Treat it as a living artifact. So review it quarterly. Ensure that any new start or new joiner who comes to the organization reviews the technical document, gets to ask questions about it. So that becomes a living, breathing part of how you deliver. And then make sure that there is a set period where it gets reviewed and updated based on the new operating context and decisions that have happened. And so technology strategy should amplify business and or user outcomes. Defining a long form technology strategy document is an excellent way of communicating the objectives and building alignment and strategy. It's an iterative process approach, defining the technology strategy document in the exact same way. And for the technology strategy to deliver value, it has to be accessible and visible and embedded in the operating rhythm of the teams. So always ensure that it's visible, accessible and embedded in the onboarding experience for new starters. Okay, we're now going to move on to possibly my favorite part of this talk, which is about metrics and situational awareness is absolutely critical to strategy, but user needs cannot be met by just becoming aware of them. That would be a think tank and single most important question you can ask is, is this work aligned to our user needs? And this is actually what an effective metric can help guide us towards. So before proceeding, let's look at another definition, and this time you won't find it in a Macquarie Dictionary. This is my own definition of a metric, and I see it as a quantifiable measure of any component or process which is used to gauge the performance of the business. So it's tied to a business outcome, and who's changed is used to inform whether an existing course of behavior should be continued or adjusted. Key takeaway a metric is only valuable if it drives behavior. If it's non actionable, it is not a good metric. Throw it away. Measuring the wrong thing is far more detrimental. And a metric can apply to an application, a certain section of your user group. So maybe a particular persona, process system or any area of your business, but they are used to guide and provide feedback on the efficacy of your current course of action. And so let's take a look at how they might be used. So if we look at an evolutionary system as an arbitrary starting point with situational awareness, we then observe our context, we orientate ourselves within that, we synthesize this information, and we generate a set of options, each option here being denoted by an arrow. And then based on our metrics, we can then define which ones are likely to or which hypotheses are likely to improve those business metrics or disapprove and potentially need to pivot. And so we can use them to select options. And in this case, we can then proceed. And once we arrive at that new point, we again iterate through that loop again, and we generate a new set of options. And again we look at our metrics. Have they improved? If they have improved, then we can continue along that course of action, or potentially run another experiment to slightly adjust to see if we can get an even better dial shift in that outcome. And we can keep proceeding with the same pattern until some arbitrary end. And we all know that in the evolutionary system, there is no end. But the key point is that the metrics themselves provide that measure for whether the experiment was a success or a failure. Did it actually move the dial versus did it feel better? And they are absolutely key to learning. And in an excellent book, poor title, but excellent read by Eric Reid and the Lean Startup. He proposes that the only way a company can win is by learning faster than everyone else. And he defines a set of principles of the Lean Startup. Mvp is one that is probably best known. But there's also the Build measure Learn loop, which is actually very similar to the UDA loop which we looked at, and also actionable metrics. So again, if you're looking at more detail around techniques to choose effective metrics, this book has a lot of valuable information, but we're also going to look at this topic ourselves over the course of the next five or so minutes. So how should good metrics be chosen. And it might seem like strange advice, but don't start with a metric. Don't take a measure and then try to build behavior around it. Make your user needs your North Star. So understand what your users need and how can you satisfy those outcomes? Potentially. Then you come and look at how would I turn the dial on a couple of those user needs? What are some concrete things that we could do to improve that? Then and only then, do you start thinking about a measure for success. The metric is arrived at once you understand the user needs and what success looks like from your customers perspective. And we have a number of options that we can use as well. So if you're looking for more guidance about how you can define an initial set of metrics, we can look at Dave McClure's Pirate Metrics for some inspiration. And here he defines your users into one of five life stages. At any point in time, we can have acquisition. So how do users find you activation once they found you? How do you ensure that they have a great first experience? Once they've used your system? Do they continue to return? Do you have that stickiness with the customer? How do you turn those users into advocates? This is about referral. So this is again creating word of mouth around your business. And then ultimately, how do you make money from those users? And in here, I should point that these are not strictly sequential steps. In this particular model, I have sequences around growth. So in this scenario, we're really looking to build a number of users and then both in terms of retention and then also expanding through referral. So in here, our business model would likely be based on scale. So a small amount of revenue, but many and large user sets depending on your business, potentially B to B revenue would come directly after activation and you'll be looking at a small number of high yield customers. But again, using these stages can help define where you're looking at for the various metrics that you need to define at that point in time. And transitioning to the use of metrics can be challenging. And in many cases, as companies look to make data, informed decisions, they realize that they don't actually have the data required to make these decisions. Because you can't create insights from nothing. You must start with data, and data must be categorized, filtered, cleanse to ultimately become information that you understand. And only once this information is synthesized can we generate valuable insights from it. And the shame is that most companies do not capture the data required to make effective, informed decisions. So an example from banking, and when I was working at Commonwealth Bank, we had an event around customer changing their name. We simply had a name changed event. But what was the reason for this? What was the intent? Was the person getting married, or was the person getting divorced? How you would respond to this purely from a data perspective is lost. But if the intent is what the customer is looking to achieve, you then have an incredibly rich set of contacts and information from them to choose more appropriate options. So again, when you're designing your systems and telemetry, don't look purely on fraud data changes, but rather the intent of the action being performed, because that will enable far better inferences to be made later on in the process. And as I mentioned a couple of times, metrics are absolutely key to driving effective behavior within your organization. And Simon Wardly, in a similar vein, has been looking at what are the principles and practices that companies should define in order to be successful. And through this research, he found a set of what he refers to as doctrine, which are universally beneficial activities such as spend control about being challenging discretionary costs within your business as a way of improving discreet decision making. And he calls these universal good practices. And so I sought out to go either universally good metrics that all companies could use irrespective of industry or size, and not for everything in the business but around delivery efficacy. There are. And in the book accelerated by Nicole Ford's Grande Humble and Gene Kim, they did a huge amount of research into continuous improvement. And the application of DevOps practices have been proven to increase the likelihood of success or predict success for the organization. Let's look at that. I see DevOps as a set of practices that works to automate and integrate the processes between software development and It teams so they can learn faster. You may see some definitions which are so they can release software more effectively. That's the action. It's not the outcome. The outcome has to be to learn quicker. And in the DevOps State of the Union work, they define four primary metrics, deployment frequency. So the number of releases your teams are doing and we want that to be as high as possible. Lead time for change. So that's how quickly you can go from idea through to production. And we want that to be as short as possible. We have mean time to recover. So if there's an impact to customer service, how quickly can we recover from it? And then we also have failed change rate. So what's the percentage of changes that fail? And we can group them into two categories. We can have deployment frequency and lead time for change which are really about speed. How do we move more quickly? And then we have meantime to recover and fail change rates which are really about stability. And if we plot them on a graph, we can put on the y axis speed of productivity and on the X axis quality or stability. And what we now see. If you could forgive the hubris, I'll introduce Trolley's non scientific cone of balanced behavior, which then shows that these sets of metrics are actually in positive tension. If your company has a measure such as number of priority one or severity one incidents in the last year, and that's what you use to measure success of your It operations, I think that is an awful measure because the only way we're going to respond to that is by releasing less often, because when we release, we inject entropy into the system. We need a counteracting measure that promotes speed. And so for your own business, you can define how quickly do you want to go versus your risk appetite for being more stable. And you yourself can then define a measure of how quickly and stable you want to be, and potentially based on individual teams or departments across It. And what we have learned from the study in Accelerate is you can absolutely have both speed and quality, but if you only focus on one due to the incorrect metrics, you will lose the other. Again, create balanced metrics that you can't optimize for one without impacting the other, and that then shows these bonds drive balanced behavior. But the path to success with metrics is fraught with failure, and it's very easy to go wrong with these as well. So let's explore some of the missteps that you can see with metrics. The first is creating metrics that are about avoiding error, such as counting the number of P ones rather than achieving excellence. Don't look to eliminate error, look to promote excellence. So that's a key point from there. And if the team doesn't quite hit a particular target, that's okay because we want teams to operate and take ownership within the risk appetite within your organization. One of my favorites is known as Campbell's Law. You may also know it as a perverse incentive or the Cobra effect. And this is when a metric is intended to have a positive outcome. But when it's used for decision making, it gets corrupted by pressures and ultimately gets distorted and yields a different outcome to the one we expected. So let's look at an example of Campbell's Law in practice. In the 1990s, New York and Pennsylvania started publishing mortality data on hospitals and surgeons who did coronary bypasses. The idea was that informed customers could steer themselves towards the surgeons with better statistics, theoretically good for patients and good for surgeons, ie. The surgeons who are doing the better work would get more customers. The reality was less ideal in those States. There's evidence that surgeons seem to be started doing operations only on healthy patients while turning away the sickest, because unfortunately, the sick of the patients, the greater the chance they will die doesn't pack in the mortality rate. The Great Hanoi Rat massacre is another famous example of Campbell's Law in action. Another misstep with metrics is vanity metrics, which are inaccessible, not audible, and not actionable. So unless your company is making revenue from admissions, then Sizing Total Page Hits is really a vanity metric not tied to the revenue model or ongoing sustainability of your business. We can also have cargo cult metrics, which is really an example of survivorship bias where we look at the metrics used by other successful companies and we seek to copy them or imitate them. And really we are substituting imitation for understanding. And while it's a beguiling shortcut, it is fraught with peril because it is almost certain that the organization's business context is not the same as yours. So you're effectively taking their roadmap. But with your own organization, there is no shortcut for the hard work of understanding the opportunities that your business can potentially capitalize on, the obstacles that are impeding you from doing so, and then the concrete decisions and actions that need to happen to achieve that. We also have uses metrics. For instance, measuring outputs, total lines of code written by a developer, total hours worked, number of PRS reviewed, meaningless, useless metrics are about outputs. We should always be seeking to measure outcomes, and we should be seeking to maximize outcomes and minimize outputs. And possibly the most pernicious or dangerous of all metric missteps which are weaponized metrics. And this is using metrics for comparison and punishment rather than for continuous improvement processes. Really using metrics for evil rather than for good. So a manager using metrics to justify consequences to apply upon the individuals or teams within the organization. And as with anything, there's a law for this and it is called Goodheart's Law. And Goodheart observes that any observed statistical regularity will collapse once pressure is placed upon us for control purposes or to generalize it. When a measure becomes a target, it ceases to be a good measure. So what can we do to confront this? Pulling it out, sharing visibility around weaponized metrics and this behavior. Removing information silos is incredibly important. Making sure that everyone in the organization has access to the same information and breaking down these silos. Also choosing metrics that optimize around pairs so you can skew for one course of behavior and ignore the competing or other side of the equation. Finding your metrics in effective pairs, Removing knowledge silos and making sure that everyone is aligned to a common set of Okis or common metrics is incredibly valuable for removing the temptation for people to act in this particular way. So metrics prioritize based on metrics rather than non actionable conversations. So user needs should be the basis for your initial set of metrics. And incentive structures can really undermine the behavior you see. So measure the wrong thing can lead to unanticipated consequences. And effective metrics often work in unison to drive balanced behavior. And finally, I've said a couple of times already, metrics only deliver value if they influence behavior. Okay, so in the last couple of minutes together, I'll provide a few tips and tricks that you can use to bring back to your own organization. And I do want to start with a Disclaimer that the transition between delivery and strategic focuses is nontrivial and requires dedication and commitment. I have no snake oil or quick wins. Strategy is a skill and like all skills can be described to the Dryfish model of skills acquisition. We will start at novices and at some time after repetition we may achieve mastery, but it will take time. There is no compression algorithm for experience. The first step is truly invest in learning your business and industry. This is vital to growing your situational awareness. Once you have this context, know and share it. Always bring conversation back to the user. Need the objective you're looking to achieve and ensure that there hasn't been a subtle shift to a proxy measure to solve a problem that is different, but on the surface looks similar. Bring it back to the user need embrace empiricism quantitative data rather than again, qualitative conversations. So based conversations on metrics rather than opinions. And this can be difficult if your organization doesn't have this data available. But it is going to be so much more powerful to persuade your leadership or teammates around a course of action. If you have the data underpinning it and you may not always have the authority to make a decision, but you can always champion a cause. So irrespective of what role you're in in your organization, if you see something that can be improved, engage around it. Build awareness. Use the data points and seek with either your leaders or peer group about building momentum around it. Build your network both inside and outside work so hallway tracks so slack for NDC Sydney please get on it, talk with your fellow speakers or attendees and start building your network. It will provide an incredible host of learning opportunities outside of the tracks themselves. If you're a leader, you should focus on growing an environment that encourages the behavior you want to see. So again, making sure that the metrics that are in place if it's around promoting learning and growth actually drives that behavior. If again, you're measuring people on outputs so lines of code written, that's what your senior engineers will do. They won't work with juniors to skill them up or share their knowledge because that will then hit their own individual KPIs. So again, make sure that you are focusing on building an environment which drives the behavior you wish to see. And as a corollary, attempting to embed behaviors that run contrary to the incentives defined by the environment is a Sisyphean and endeavor. So again, if you're looking at a course of behavior that doesn't follow what you'd like to see, chances are the environment is what's driving that change the environment be a catalyst for change. So what we've seen is strategy is often a nebulous concept, but it's really about an approach for allowing your organization to win and it is based on situational awareness that is constantly changing. And the strategy employs both a direction in terms of the strategic insight you wish to capitalize on or the problems you wish to overcome, the guiding policy that will then drive behavior and a concrete set of actions that is going to deliver measurable outcomes. A technology strategy then underpins the business strategy and it is used to amplify it. So how are the technology decisions we're making dovetailing with the business strategy to maximize the overall outcomes being delivered and we've seen that decisions and actions are the real core of any strategy. Without that it's just words and talk that is where the rubber hits the road and we have seen that with actions they are guided or the success of those actions are being guided by the result of the metrics and metrics should be anchored initially around user needs and finally we saw techniques that you can apply yourself and it starts again by understanding the context in which you operate and building a reputation for someone who can think deeply around a problem but also laterally from a business context. And so I hope I pushed your appetite with a few brain teasers or introductions around strategy. If you're interested in learning more, I have listed the set of resources or blog posts and otherwise that I have based much of the content from this talk from and I have also provided a reading list. So again all the books that I have referenced throughout this talk and so at this point I'd like to thank everyone, I will take questions on slack after this session and with that I'd like to thank you again and see you later in the conference. Bye.