Home > TechReviews > Most product comparisons tell you jack. I propose a better way.

Most product comparisons tell you jack. I propose a better way.

Most technology evaluations you read in trade magazines, or those provided by manufacturers suffer a common and basic flaw.  They compare technologies with other similar, competitive products and rarely if ever tell the reader how good the product really is.  The evaluations list features in light of those of competitors.  In my opinion, they perpetuate mediocrity. 

I think there is a better way to look at technology products.  The method I’ve been developing for the last four years is as close to a scientific way of performing the evaluations as I can find.  My method also has another important difference: it tells the security executive (or the manufacturer) how successful the product will be at solving the end user customer’s problems, rather than telling the customer what he or she will have to settle for.

I’ll say it more plainly.  By taking a hundred, or so, criteria from end user requirements and preferences, I can score how closely any product comes to the hopes and expectations of the customer.

Here is an example of the scoring.  This product (which I will not name here) was recently awarded very high marks in a (more or less) independent product comparison with other big name access control products.  It is considered one of the best products you can buy in its category.  However, you can see from the scores, it still has some areas of improvement if it stands a chance of meeting the customer’s actual needs.

Category & Rating
Architecture and Integration, 2.7
Reliability and Scalability, 3.5
Configuration and Flexibility, 1.9
Administration and Reporting, 1.8
Overall Rating, 2.5

With a scoring range of 1 through 5, 1 would represent poor or absent support or quality.  5 would indicate satisfying, broad, flexible qualities.

Each category has several sub-categories consisting of several criteria.  Each sub-category and criterion is weighted according to its relative importance to the customer.  Therefore, the amount of database administrator time required to set up the system may be weighted more or less heavily than, say, the range of third-party databases supported by the product, depending on what customers prefer.  Similarly, the usefulness and intuitiveness of the graphical user interface or the online help tools will be weighted more heavily than the product’s support of a command line interface.

When my evaluation is completed, the CSO or product manager will see in the detailed report every major way the technology meets, exceeds, or fails to address dozens of important requirements or preferences of the end user customers.

So rather than crying and carrying on about which gizmo has more features than the next guy's, lets focus on solving the problem and meeting the end user security executive's needs. Let me know if you’d like more information on measuring the true value of technologies.  I’m happy to chat.  steve@securitydreamer.com

About these ads
Categories: TechReviews
  1. Jeff
    January 23, 2009 at 9:14 am

    What about taking the Registered Dietician approach. I get the “you’re comparing apples to oranges” all the time, but when you break down the nutritional components of apples and oranges, combined with a guide (like the Canada Food Guide or a RD) a person can make an informed choice about foods to purchase.
    What about a healthy plan for security?
    In Canada there is the Canadian Nutrient File that lists the nutrient breakdown of commonly consumed foods. How do we do that with security products? What are the nutrients of products, so users can compare with some degree of standardization?
    -Jeff

  2. January 23, 2009 at 9:59 am

    I like it, Jeff. That’s right. It’s not the product, or the fruit, but its benefits that matter.
    Erin Kinikin said the best “competitive advantage” add ons are the ones you don’t actually have to add unless the customer wants them.

  3. David Aus
    January 23, 2009 at 11:04 am

    Good idea — but have you ever noticed that most dieticians are overweight?

  4. Ross VanderKlok
    January 23, 2009 at 11:44 am

    I agree Steve. We need a “Consumer’s Reports” type magazine for security products. But who would be willing to risk alienating advertisers or opening themselves up to lawsuits? Apparently no one since it has not happened yet. I had great hopes that Anixter was headed this way when they opened their product lab some years back, but no luck. I have noticed some sites and magazines in the UK that are close to doing these types of ratings (wait for it), BUT not frequently enough or to the level it needs to be. I nominate you to start doing this work!

  5. January 23, 2009 at 6:13 pm

    I agree that this type of information is badly needed. When I was much younger (and more naive), I thought that one of the security magazines might step up to the plate and offer something like this, but as Ross says, none of the magazines want to offend the advertisers. (It seems like the best advertisers are the ones that make the poorest products, but that’s another topic…)
    Quite a few years back, one of the more renowned security consultants tried to provide a “consumer rating” service for access control systems. As I recall, he charged manufacturers some type of fee to have their products evaluated, so the objectivity of the service was called into question. Some felt it was a little like extortion – if you didn’t pay up, your product wouldn’t be evaluated, and you would be left out of the ratings entirely.
    The only challenge I see is, who would be willing to pay for a service of this type? Consumer Reports, has four million subscribers and a 20 million dollar annual budget for testing. I have a hard time seeing how there would be enough security consumers (and cheapskate consultants like me) to pay for the costs involved in providing a truly objective product evaluation service.

  6. Andrew Merric
    January 29, 2009 at 7:45 am

    I’d like to get your honest opinion on this site that deals with digital security. Is it worth it?
    http://www.justaskgemalto.com
    I hear they are about to add a blog, which I think would be a better format for the consumer.

  7. Andrew Merric
    January 29, 2009 at 7:46 am

    I’d like to get your honest opinion on this site that deals with digital security. Is it worth it?
    http://www.justaskgemalto.com
    I hear they are about to add a blog, which I think would be a better format for the consumer.

  8. Andrew Merric
    January 29, 2009 at 7:46 am

    I’d like to get your honest opinion on this site that deals with digital security. Is it worth it?
    http://www.justaskgemalto.com
    I hear they are about to add a blog, which I think would be a better format for the consumer.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: