What a Vendor Score Should Actually Measure

Short answer: A rubric that returns nearly the same answer for most of the field is not measuring anything useful, however carefully each criterion is defined. This page states plainly where ours does that.

A score is a compression, and compressions lose things

Any single number collapses several judgements into one. That is useful when the underlying judgements point the same way and misleading when they do not. A vendor strong on testing and weak on recourse is not well described by an average of the two.

So the number should always be read alongside the criteria beneath it, and a scoring site that hides the criteria is asking to be trusted rather than checked.

The criteria have to discriminate

A criterion that returns the same answer for every vendor adds length to a page and nothing to a decision. This is the most common failure in vendor scoring, and it is invisible if you only read one vendor's page.

The test is simple: look down the column rather than across the row. If every cell says the same thing, that criterion is doing no work.

Where our own rubric currently fails that test

On the three criteria that speak to credibility — lab transparency, shipping origin, payment reversibility — four of the five vendors we score currently return the same ratings. Only one separates, and it separates downward.

That means our top-of-table ordering is being decided by the remaining criteria rather than by the credibility ones, which is not what a credibility score should do. We publish that rather than let the ordering imply a distinction the evidence does not support.

Why we have not simply added more vendors

Widening an index that cannot separate the vendors already in it adds pages without adding information. The useful work is finding a criterion that discriminates — something observable that genuinely differs between sellers who all publish lab reports and all ship domestically.

Candidates we are testing: whether reports resolve on the laboratory's own portal, whether any failure has ever been published, and whether the published set covers the catalogue or a corner of it.

What a reader should take from a score

Treat it as a summary of the criteria, not as a verdict. Read the criteria that matter to your decision and ignore the total if those criteria disagree with it.

That is a strange thing for a scoring site to write. It is also the honest reading of what a five-criterion score can support.

Weighting is a judgement and should be visible

Any multi-criterion score implies a weighting even when it does not state one — an unweighted average weights everything equally, which is itself a strong claim. A reader cannot evaluate a score without knowing which criteria moved it.

Ours are equally weighted across applicable criteria, which is a defensible default and not a neutral one. Someone who cares only about lab evidence should read that column and ignore our total.

Who the score is for

A score is useful to someone choosing between options that are otherwise hard to compare. It is useless to someone who has already decided, and misleading to someone who treats it as a safety rating.

Nothing we publish establishes that any vendor's product is safe or is what it claims to be. The rubric measures how much checkable evidence a vendor publishes about itself — which is a real thing, and a narrower thing than most scoring sites imply.