Dont Skip the Proof Of Concept

Unfortunately, sometimes major technology purchases go unvetted and buyer’s remorse sets in hard, evident in the state of Oregon’s recent lawsuit (pdf) against Oracle for the failed deployment of its Cover Oregon healthcare exchange website.

This case reflects a dated IT evaluation process that’s all too common: businesses buying software based on RFPs and product demos, while failing to make vendors go through a proof-of-concept process. It’s a huge mistake that’s very easy to avoid.

Product Demos Aren’t Everything

Businesses generally buy software in a six-step process:

  1. ID the software requirements
  2. Create an RFP
  3. Send the RFP to vendors you want to evaluate
  4. Narrow the vendors list down based on their responses to the RFP
  5. Watch product demos from the shortlisted vendors
  6. Pick one vendor. Pretty straightforward, right?

But this widely used process fails to answer one key question: how will the software actually perform in real life?

Sure, demos are important -- they showcase the product in the best light and, if a demo goes poorly, it’s a major red flag. But many organizations overvalue them when making their software selections, often building a strong emotional connection to the product due to a flashy presentation, appealing color scheme or familiar UI. Is it any surprise that in the Cover Oregon case, the demo went wonderfully? It’s supposed to: it was designed specifically for that presentation but had not been battle-tested for real-world deployment.

Demos are examples of best-case scenarios, beautifully designed with feature-rich, highly controlled environments meant to sell you. They are NOT perfect examples of what your version of the software will look like when it’s deployed. For that you need to take the final but most important step in the buying process, adding a proof-of-concept (POC).

The Proof of Concept Phase

You’ve seen the demo and it looks great. Maybe you’re smitten and ready to buy. But wait! Before you pull the trigger, you need to see how the software will work against your specific use cases and in an environment similar to yours. You need a POC.

Asking for one isn’t as easy at you may hope. Vendors aren’t eager to develop them; it pulls in additional resources from their end, slows the sales process and increases the risk that their product won’t actually work. But when done correctly, a POC is a win-win for the vendor and customer. It helps the vendor understand if they’re a good fit for the customer because in the end, a happy customer is a happy vendor. For the customer, a successful POC helps them pick the right product based on their actual needs.

So what should go into a successful POC? First think about the goal of it and which stakeholders within your company need to evaluate the tool. You want to look at specific scenarios that allow you to assess the software’s fit for anyone who may use it. That could mean marketers, developers, designers and ops/IT. Once you’ve identified these stakeholders, you can create a real-world scenario in which the product could actually be used.

Let’s use a CMS evaluation as an example. For a POC, you may want to test a complete workflow around the creation, editing, approval and publishing of a piece of written content. Don’t just sit back and let the vendor set up a scenario for you -- take a hands-on approach. In this example you should have authors and editors creating content, developers and designers building a website, and IT architecting the solution. This would allow a number of parties (editors, designers, IT, etc.) to test and evaluate the CMS end-to-end. While the vendor should support you in a POC, they don’t know exactly what you’d be using their software for. Only you do, so create a situation to test that.

Don’t get caught in a vendor’s trap by buying their product because you fell in love with a demo, read a positive analyst report or met them at a tradeshow booth. See a demo, but be sure to ask vendors for a POC. Too much of your time and money is at stake to take a shortcut. Like buying a car, purchasing software is a big decision. Make sure you don’t end up with a lemon and take it for a test drive.

Title image by Luigi Crespo (Flickr) via a CC BY 2.0 license