Technology is imperfect. Anyone who鈥檚 ever used a computer knows this. You search for a file that you know to be on your hard drive and somehow the search function just cannot find it. If computers are flawed, is it any wonder that diagnostic and screening tests are too?
There is a lot of hope attached to COVID-19 serological tests. 鈥淪erological鈥� means that it relates to the serum, the liquid component of our blood that has been purged of clotting factors but that contains proteins like antibodies: it is these antibodies that are being earmarked for our salvation. Antibodies are produced by a subset of our blood cells in response to an infection. If you have antibodies in your blood against the new coronavirus, that must mean you had the virus inside your body and managed to fight it off. The presence of these antibodies infers a protection against becoming sick a second time from this coronavirus (although it is not yet guaranteed, but), and being able to tell whose body is now 鈥渟hielded鈥� from COVID-19 from those who are immunologically na茂ve could prove critical to public health authorities.
However, tests are technological in nature, and technology is imperfect. Here鈥檚 how a COVID-19 antibody test result could be wrong.
A new gold rush
The first limitation of antibody tests is that they are dependent on our immune system. You can鈥檛 detect something that isn鈥檛 there, and our blood cells do not start making antibodies against the coronavirus as soon as we become infected. In fact, it takes between two and three weeks after the start of symptoms before enough antibodies are made that they can be detected (though in some cases). Someone in their first week of infection would most likely get a negative result from an antibody test: that鈥檚 why these particular tests, unlike the diagnostic tests that use a long Q-tip to collect the virus and detect its genetic material, are best used to identify people who have had the virus but don鈥檛 anymore. It鈥檚 also important to point out that not everyone who encounters the virus will develop antibodies against it or enough antibodies that can be detected by the test.
But another big problem is incompetence. The universal need for these antibody kits has ushered in a new gold rush, and just like an inexperienced buyer may be fooled into believing pyrite is gold, unqualified companies are developing tests of a dubious accuracy to cash in on the demand. Some of the COVID-19 antibody kits on the market were manufactured by companies with (one such company usually sells Chinese herbal remedies while another sells solar panels and tires), and independent testing revealed that their accuracy was no better than flipping a coin (and a coin is after all much cheaper). Health Canada has been very cautious, only at the time of writing. The for companies wanting to sell their serological tests, telling them they did not need to go through an onerous approval process: simply let the FDA know what your kit can and cannot do and we will allow you to sell it. They have since corrected their course and voiced concerns about. Pyrite, after all, can鈥檛 do much harm but a false sense of immunity in the middle of a pandemic can be deadly.
A final issue with these antibody kits--and with any diagnostic or screening test--is that they are never 100% sensitive and 100% specific. Using the analogy of a home pregnancy test, you want the test to tell you that you are not pregnant only if you are not pregnant, and to tell you that you are pregnant only if you are. You want all of the positive test results to be true positives and all of the negative test results to be true negatives. But you may at the same time ask for world peace because these ideals simply cannot be attained in the real world. For example, a kit could tell you that you have antibodies against the coronavirus but it turns out it mistakenly detected your antibodies against a different type of coronavirus, like one of the many that cause the common cold. Health Canada asks kit manufacturers to, but these pre-approval tests are limited in scope and technology is fallible so it remains a possibility.
Even if an antibody testing kit has a very high sensitivity (meaning it will detect antibodies if they are there) and a very high specificity (meaning it will yield a negative result when antibodies are not there), there is still one more issue to tackle and it involves some eye-opening mathematics.
One number changes everything
Home pregnancy tests are not unerring but they are pretty reliable, and in part it鈥檚 because the people who use them have a higher chance of being pregnant. They often use them because they have had sex and their period hasn鈥檛 happened, so they suspect they might be pregnant. But if you gave this test to every person capable of bearing a child on the face of the Earth, the accuracy of the test would decrease. And it boils down to an important concept: how common the thing you are trying to find is in the population you are testing.
Let鈥檚 take one of the two COVID-19 antibody kits approved by Health Canada, with a sensitivity of 97.4% (it will almost always tell you you have antibodies against the coronavirus when you do) and a specificity of 98.9% (it will almost always tell you you do not have antibodies when you don鈥檛). Sounds like a near-perfect kit.
Imagine a scenario in which only 1% of the Canadian population has antibodies against the coronavirus. If every Canadian鈥檚 blood is tested for antibodies and you personally receive a positive result (meaning you are told you have antibodies and may now be immune), your result has a 47.2% chance of being accurate (a coin toss). But if 5% of the population has these antibodies and you test positive, your result has an 82.3% chance of being correct. As the percentage of the population that is exposed to the coronavirus and that develops antibodies against it grows, from 10% to 25% to 50%, the chances that your positive result is real similarly swell, from 90.8% to 96.7% to 98.9%. The more people have antibodies against the coronavirus, the likelier the odds that your positive result from this one test is accurate. A negative result鈥檚 accuracy is similarly influenced by this phenomenon.
Even with a robust kit, if the thing you are looking for is rare, the results you get will tend toward being unreliable. By that logic, being randomly tested with an antibody kit at the beginning of a pandemic will produce results that will not be as dependable as those generated toward the end of a pandemic. This is why that using antibody tests--even reliable ones--in populations where very few people are thought to have been infected with the coronavirus will yield inaccurate results.
This is also why mass screening for cancer is not as beneficial as it might sound: the more people without cancer you screen, the more false positive results you will generate, which will lead to anxiety and more invasive testing. As COVID-19 serological tests get deployed to screen the population, we need to keep in mind the many ways in which they may err. They have their place, but as imperfect technologies, they have their flaws, both in their construction and their application.
For those who want to delve deeper into 鈥淗ow to calculate the chances of a positive result from a test being accurate鈥�, click here
This will be particularly useful to science and medical students who are curious about how a test鈥檚 sensitivity and specificity do not tell the whole story. How did I calculate the percentage chance of a positive result being accurate? That percentage is known as the 鈥減ositive predictive value鈥� (or PPV), but a big caveat: you need to know how common (or prevalent) the thing being tested is in the population in which the test is being used. The positive predictive value will not tell you the odds that your positive test result is a true positive in real life; what it can do is tell you how reliable the test would be in different scenarios and thus help you decide if using this test in a specific population would be wise.
The PPV is simply the number of true positives who test positive divided by the total number of individuals who test positive, times one hundred to convert it to a percentage.
To calculate the first variable (how many true positives test positive), you need to multiply the sensitivity of the test (a 90% sensitivity would be entered as 0.90) with the fraction of the population that is truly positive (or that is known to be positive by using a gold-standard test) and with the total number of people being tested. A similar number called the 鈥渘egative predictive value鈥� (or NPV) can be calculated to learn the odds of a negative test result being accurate in a particular scenario.
You can see an example of these calculations , where the use of tables may clarify the math.
Take-home message:
- Antibody testing for COVID-19 is useful but not perfect
- Some people who are infected by the new coronavirus do not develop antibodies, and even those who do typically start producing them a few weeks into the illness
- Many antibody kit manufacturers have no experience making these types of tests and are using this medical 鈥済old rush鈥� to make a quick buck
-Every test will have a percentage of false positive and false negative results, but even when these percentages are tiny, a test鈥檚 positive results will be more and more accurate as the thing it detects is more and more common in the population
