Simple scorecard for evaluating software
Do you need to select software but are confused by the sheer number of potential suppliers, and don’t want to make a costly mistake?
Large organizations often have an RFP — or request for proposal — process in their technology or procurement groups for selecting software packages with an eye towards implementing one or more business capabilities. We often call this “build versus buy” as a technology group may have the option to create software themselves (build) in addition to procuring services, software, or hardware from an external supplier.
This article talks about — and provides a scorecard for — selecting the right external supplier for your business. The process of selecting COTS software is fairly well-known in the public sector space and has permeated many large organizations; smaller companies and organizations often have to construct a process which fits their culture, goals, and available resources.
We will use a scorecard (linked below) to ask the right questions to make sure a supplier’s product fits our organization along a number of key dimensions. The scorecard itself is designed to evaluate a single supplier out of the many suppliers that want your business and may answer your RFP. After your organization issues a call for proposals, you’ll get specific proposals, details on their capabilities (we’ll call this background research) and likely arrange demonstrations with certain suppliers that pique your interest. With all this information in hand, your evaluation team should be able to efficiently and concisely recommend a single solution to move forward with. We’ll do this by scoring each supplier in four areas:
- Technical evaluation criteria
- Alignment and ability to execute
- Cost criteria, and
- Contractual criteria
The technical evaluation
Technical criteria vary widely among different classes of software and should be constructed by someone knowledgeable in the domain (e.g., an ERP expert or a networking management software expert). In general, do NOT construct these statements based on the identification of features made by a single supplier — marketing may have chosen a term that sounds exciting for the brochure but “provides excellent DevOps support” may just not be a differentiator. Each criterion you come up with should be a significant scenario and is identified as required or flexible (can the supplier be close or provide an alternate implementation than what you were thinking, or do they have to exactly match or exceed the criterion?). In the last column, the team will fill in a 1–5 numerical evaluation. Occasionally, a weighting number may be useful if certain requirements are major drivers of the selection, however be cautious when using more than 10 criteria or a complex weighting scheme.
Criteria are functional (does a specific task or scenario) or based on quality attributes such as:
- Scalability — can it handle our expected workload? With our current environment?
- Performance
- Availability (possibly with a service level agreement)
- Integrations to our current systems
- Are we in a multitenant or dedicated environment (shared)?
- Release roadmap
Alignment and ability to execute
In this section, the proposal evaluation team will ask the questions related to how well the supplier’s solution fits within the organization at this point in time. We want to ensure that the solution will integrate with other solutions we already have and that we know how the solution will be implemented: this is a combination of knowing the project strategy, the business strategy, and how our users will capture value out of the solution. Specifically, we may have to spend our own time & resources to install and configure the software (maybe the supplier has consulting which can help here or a third-party may). We may have specific business objectives in mind — combining systems to reduce operational costs or develop a competitive advantage with this solution (just write it down!). And we may choose not to implement the solution for everyone in our organization — maybe it’s only a pilot effort to see if this would fit, the subsidiary in France may want to try this out.
The cost evaluation
The supplier should be able to provide the answers to most of these questions, except what offsetting savings we might have — by purchasing this solution, we may reduce costs in other areas.
Contractual, liability and security (read: risk) concerns
This section contains assessments of how to leave the service or discontinue use of the software, and liability in different areas. If the organization has regulatory compliance requirements, these should be evaluated YES/NO (binary) against the supplier’s references and evidence; most software products have legally reviewed statements of support for different compliance frameworks, especially important if the supplier will have your data and a breach occurs on their system.
A couple final notes.
This template is for a commercial software procurement in a non-governmental organization. I mention this because we often may want to use the same process for other items like services, staff augmentation (skilled labor), custom software development (see build, above), facilities, etc. Please modify as needed.
I mention non-governmental use because the federal regulations often specify additional criteria and have a much more formal process than what a small organization would need or be able to support. For example, the initial request may have to go to all registered suppliers and be published on a portal; the bids may need to separate cost from technical responses so that two independent teams can evaluate; certain bidders may receive preference, etc. Small organizations may have certain industry-specific regulations which may need to be included — for example, if you require the system to be compliant with Sarbanes-Oxley or GDPR — but these are often fewer in number than a public sector organization might need.
Two other options besides RFP are commonly used: the request for information (RFI) and the request for quote (RFQ). The RFI is used when the complete technical criteria are not well known — that is, when your organization does not know the capabilities of the software being acquired. The RFI is worded in such a way that the suppliers should suggest an ideal implementation for your organization and point out the technical capabilities in that solution. Comparing suppliers is fundamentally comparing conceptual implementations, or architectures, in the RFI process and cost information may vary quite a bit.
The RFQ is used when capabilities are well known and we are selecting mostly between commodities. For example, selecting office productivity software or a collaboration capability for our business, we might have a good scenario already established that many suppliers meet: Zoom, Microsoft Office 365, Slack, etc. Each may implement video conferencing differently but effectively we have a good solution from each. The RFQ focuses then on questions mostly related to alignment and cost.
My parting words on RFP’s is to simply focus on the steps that add value within the process and come up with an economical and efficient process for your organization. Many organizations get carried away with detailed spreadsheets to try to calculate an analytical answer (to a sparse-data problem) or process steps that require months to pass before selection. Others may follow the process “in-name-only” when it has become too burdensome for an evaluation team of 10 staff members to get together over multiple days, so they let one of the criteria owners — often cost or technical novelty — to drive the process, to the exclusion of other criteria. Both of these reduce overall confidence in the process, lead to lower buy-in to the solution selected and often having to rerun the process as the result of a poor selection process only a few months or a year later.
Please do let me know if there are enhancements or clarifications that might be helpful. My goal in this series is to write about techniques that we see in large enterprises which can be accomplished within small and mid-sized organizations. In coming up with these more efficient processes, large enterprises also can benefit through doing more with less: I call this “thinking like a startup.”