The Secret to Performance Under Fire

Given that content owners, merchants and ad sales representatives are all trying to stuff 22 pounds of content into pages with a carrying capacity of only 20, business forces turn IT professionals everywhere into part time fire fighters. There is a standard method for minimizing both the frequency and impact of fire drills on your team -- the enterprise IT guys refer to it as "monitoring and alerting." Cagey veterans schooled in conversational dynamics refer to it as "getting in front of the question."

How It Feels to Get Grilled

I spoke about this topic at the premier conference for building a faster and stronger web: Velocity held in Santa Clara, California. Along with Robert Tanzola, manager of technical architecture at Autotrader and Abelardo Gonzales, director of product management at Keynote Systems, I laid out the path to get from a fire drill centric culture to an early warning, arson prevention orientation.

No matter what type of experience you work on and contribute to, you know that page or app performance is a critical factor that correlates to user engagement and business objectives. One of the reasons we all know this is because we have all been subject to the seemingly inevitable performance fire drill that happens when an executive or product owner is surprised by a performance issue on a production experience:

Executive -- This page takes six seconds to load! Nobody waits beyond 2 seconds! I thought you guys knew something about the internet! Go fix this!

Tech Resource -- Performance is a multidimensional problem that is driven by several areas. The primary one being the content on the page. And for the record, the 2 second rule applies to time to interactive (TTI) rather than page complete and the rule doesn't even apply to all page types in all industries anyway. Pages with complex, multi homed content take time to aggregate and render.

Executive -- Maybe so. But you can't deny that our time has popped up and our conversions are down! Go figure it out how it happened and fix it!

This is the discussion that kicks of a series of investigations and deep dives. Even if you are lucky enough to live in an enterprise mature enough to stop fire drills before they start (or in the worst case scenario, turn into witch hunts), you still have several days if not weeks of research, analysis and prioritization exercises for how to get your performance numbers back where they need to be. The toughest thing to accept in the complicated dance of running an online business is the idea that your business strategy will likely be designed to make your pages grow heavier every day.

A Whole New Meaning to Being in the Loop

All of the above scenario starts with a context often referred to as "being behind the question." If you've never heard of this phrase, it pretty much ensures that you've experienced its effects. If the first time you hear about a problem in your operation is from outside or above you in the chain of command, you are forced to react to the question, because you are behind it in the information processing cycle known as the OODA loop.

The OODA loop is a term with military origins developed by US Air Force Colonel, John Boyd. Boyd modeled the information processing cycle for fighter pilots in the throes of combat by breaking it down to four interrelated and overlapping processes through which one cycles continuously: 

Observation: the collection of data by means of the senses

Orientation: the analysis and synthesis of data to form one's current mental perspective

Decision: the determination of a course of action based on one's current mental perspective

Action: the physical playing-out of decisions


Since its development, the OODA loop has come to be applied as a decision making cycle for any individual or collective organism and has arisen to significant prominence in combat, business, litigation, social contexts and any interaction between multiple players.The secret to getting "in front of the question" is to increase your own OODA loop speed until it outpaces the speed of other parties, enabling your observations and actions to happen before the reactions of other parties.

When you can move through observation to action faster that the other parties, your answer to the inquiries of other parties goes something like this:

Executive -- This page takes six seconds to load! Nobody waits beyond two seconds! I thought you guys knew something about the internet! Go fix this!

Tech Resource -- We saw that uptick last night and have started working a list of 15 performance items across three places in the infrastructure thus far. I should have an update before the end of the week. I've also prepped a rollback request that requires your approval, but I think you will want to review the added content and features before you sign off on it.

Executive -- Sounds like you are on top of it. Give me a half hour to review the rollback spec.

Tech Resource -- No problem. And by the way, the tech debt reduction request is going to the review board next week as well. When those system upgrades get approved, we will be able to avoid hits like these.

There are multiple ways to move to a proactive stance on IT topics including performance, the most prominent of which combines two pillars of the DevOps movement (automation and feedback loops) into an "automated enterprise monitoring and alerting” nervous system. Automated monitoring and alerting inside an IT shop allows your organization to reduce cycle time by accelerating the observation and orientation phases of the OODA loop.

Keynote Systems and other synthetic performance testing companies have been evolving both their monitoring tools and their APIs with an aim to an indispensable part of the enterprise performance nervous system (our implementation and other tools for getting value out of Keynote Systems' API can be found on GitHub).

From One Loop to a Tapestry

We demonstrated one particular implementation on stage at Velocity. It was our aim to help the widespread IT community come to understand the broader pattern. When combined with other abstracted mechanisms for keeping tabs on key business processes with thresholds for alerting, the enterprise OODA loop gets faster and faster and faster.

The more your enterprise -- whether big or small -- can practice and embrace the enterprise religion of automated monitoring and alerting, the more your smoke alarms and sprinkler systems will get you out of fighting big fires which will allow you to proactively advance on your most important business goals.