How to Compare Two Vendors Fairly

Short answer: A vendor that publishes nothing scores badly on every evidence criterion, including ones where it might be excellent. Comparison measures disclosure before it measures quality, and reading a table without knowing that produces confident wrong conclusions.

Disclosure and quality are different axes

A seller with thirty published reports and a seller with none may differ in testing practice, or may differ only in what they publish. From outside these are indistinguishable, and every evidence-based comparison — including ours — rewards the first.

That is defensible, because a buyer can only act on what is checkable. It is not the same as saying the second seller is worse, and a comparison that implies it is overreaching.

Normalise for catalogue size

Thirty reports across forty products is better coverage than sixty across four hundred. Raw counts favour large sellers twice over — more products means more reports, and more reports reads as more rigour.

Compare like with like inside a criterion

Purity figures are not comparable across compound types: a short well-established peptide is easier to make cleanly than a long modified analogue, and a protein measured by fill against label is not on the same scale at all. A single ranked purity column quietly penalises difficulty.

Look down the column, not across the row

If every vendor scores the same on a criterion, that criterion is doing no work regardless of how carefully it is defined. This is the fastest way to find out whether a comparison is measuring anything, and it is almost never done.

Where ours currently fails that test

On the three criteria that speak to credibility, four of our five vendors return identical ratings. The ordering at the top of our table is therefore being decided by other criteria. We publish that rather than let the ranking imply a separation the evidence does not support.

What a fair comparison concludes

Usually something narrower than a winner. 'These two are level on everything checkable, and here is the one place they differ' is more useful, and more often true, than a ranked list.