A number of commercially available COVID-19 antibody tests, which look at a patient’s blood for signs of past infection, did not pass Mayo Clinic quality screening or meet their expectations for use, researchers from the hospital concluded in a joint investigation by the clinic and ABC News.
One rapid finger-prick test even wrongly displayed a positive result for antibodies after researchers decided to use a saline-like solution, instead of a blood sample, to see what happened. An automatic fail, doctors said.
“Clearly there had been a mistake in how that kit was constructed,” said Dr. Thomas Grys, the director of microbiology at Mayo Clinic who tested the rapid test kits, which typically rely on a finger prick and a drop of blood to return results in 15 minutes. Grys said the Mayo Clinic alerted the manufacturer of the test kit of the problem and by Thursday, they had a "reformulated" version that appeared to correct the mistake with the saline-like solution. The researchers are still testing the kits' overall accuracy.
In partnership with the ABC News Investigative Unit, Mayo Clinic doctors in Arizona and Minnesota ran over 4,500 tests on both rapid and slower, traditional lab antibody tests. Nineteen different COVID-19 antibody brands selected by the clinic were tested in under a month, a process that would normally take around a year, they said. For the review, the clinic did not provide brand names to ABC News.
“We evaluate tests and make sure they perform well before we offer them to our clinical colleagues and to providers to use,” said Dr. Elitza Theel, an associate professor of laboratory medicine and pathology at Mayo Clinic and the researcher who put the laboratory test kits to a test themselves. “And so this really underscores why laboratories continue to evaluate and validate tests thoroughly before offering them for clinical care."
MORE: COVID-19 outbreak exposes generations-old racial and economic divide in New York CityOf the 19, nine were rapid antibody tests designed to give a result in just minutes. But four out of nine of those rapid antibody tests examined failed the Mayo Clinic’s testing process for various reasons, including low accuracy and physical problems with the actual tests. Quality issues like these, the team said, are why the Mayo Clinic has not yet made a decision about using finger-prick blood tests to look for antibodies in its own clinical patients.
“Our job is every day to make sure that we’re giving the right answer on every specimen from every patient,” Grys said.
The Mayo Clinic team also saw performance issues with more sophisticated laboratory blood analysis’ in which a larger sample of the patient’s blood is drawn and taken away to be tested in a lab. They found these laboratory tests fared slightly better but still had problems.
“Out of the 10 test kits that we looked at, there were four that really had A plus ratings, there were a couple that were in the B ranges, and there were two or three that were in the F range that we definitely wouldn’t want to use in a clinical laboratory,” said Theel.
Theel said four of the kits tested had been given emergency use authorization by the Food and Drug Administration – each of which received A plus ratings in the Mayo Clinic review. The other tests are either under evaluation or had not applied for the authorization.
Theel’s team found that one laboratory test came back with a false positive rate of 17% -- meaning that 17 out of 100 people tested, who were never infected, would be falsely told they had antibodies.
“If a kit has a false positive, the concern is we think they have antibodies against the coronavirus, but really they don’t,” Grys said. “So then that person might be at risk because they think they don’t maybe need to wear a mask, or they can care for someone who’s sick with the coronavirus, and they would be exposed and not protected.”
Experts say a key metric for testing is something epidemiologists called "specificity" -- meaning how well a test identifies who does not have the novel coronavirus. While Theel said that no test is perfect, she said her team looks for tests with a specificity in the upper 90 percentile.
Calls for increased testing and companies pressured to make up lost time because of testing delays may have created a perfect storm for poorly functioning tests, the research team said.
“There’s a real pressure on manufacturers to bring these to market as quickly as they can, which means they don’t have the same opportunity to vet these things in their own manufacturing processes before they bring them out and make them available to the public,” Dr. William Morice, professor of laboratory medicine and pathology at Mayo Clinic, told ABC News. “And they’re a business. You have to look at this as a business, and there’s a business pressure to get these tests out as quickly as possible.”
With increasing consumer curiosity regarding antibody testing, the Mayo Clinic team is feeling the pressure to offer doctors, laboratories and patients more reliable information.
“We need to find out which [tests] are good,” Grys said.
From a consumer perspective, the problem is that the process for determining which tests can be trusted isn’t simple.
In March the Food and Drug Administration created a path for rapid approval of coronavirus antibody tests called an emergency use authorization (EUA) and also permitted tests to come to market without manufacturers seeking such approval.
There are currently only 12 antibody tests in the U.S. that have received FDA EUA status, including the four that received the A plus rating in the Mayo Clinic review. But, as of Wednesday, there were over 200 antibody tests available on the market that had either not sought or received such authorization.
MORE: FTC, DOJ crack down on surge of allegedly fraudulent COVID-19 'treatments'The FDA maintains a public website with the performance rates of the 12 EUA tests, showing how likely a certain test is to give false positive confirmations of the antibodies, and sensitivity, how likely the test will give a true positive result when someone has the virus. But the numbers can be hard to interpret by consumers who don’t have medical backgrounds.
So what should a consumer do if they’re interested in having an antibody test done? The Mayo Clinic team recommends patients work with a trusted medical provider instead of pursuing an antibody test on their own, and to ask that doctor if the antibody test being used is one of the 12 tests currently given the FDA's EUA.
“I would just ask the doctor if they can confidently say yes they had COVID-19 if the test is positive, or no they didn’t if the test is negative,” Morice said. “If [the doctor] can’t say that with confidence, then I would be a little concerned about the test they’re using.”
For Kelly Wrobleski, the director of infectious disease at the Association of Public Health Laboratories, the risk of inaccurate antibody test results outweighs the potential rewards.
“I think there's too many poor-performing tests on the market, and I don't think they give you really good information on how you should or shouldn't change your behavior,” Wrobleski said.
Initially, the FDA did not object to antibody tests being used even without an emergency use authorization. But with growing concern of antibody test reliability, two weeks ago, the FDA updated its antibody testing policy saying manufacturers would have to submit an EUA or go through validation at the National Cancer Institute if they wanted to sell an antibody test and were given 10 business days to do so.
“This is a top priority for the agency," an FDA spokesperson told ABC News Wednesday. In the coming days, the FDA says it will separate out tests that have not been through an EUA or other valid approval process.
“I hope they start to pull those from the market and start to hold those manufacturers accountable to not selling them,” Wrobleski said. “My hope is that by June we start to see only tests that have shown some level of quality performance out there.”
Late Thursday the FDA announced it is posting online the names of antibody tests that it said have been removed from the "notification list" of tests being offered under the department's emergency coronavirus policy, including those that were voluntarily withdrawn and those for which there is no pending EUA request or authorization. The FDA said it expects the tests that are removed from the list will "not be marketed or distributed."
FDA Commissioner Stephen Hahn called the move an "important step the agency has taken to ensure that Americans have access to trustworthy tests."
As of Wednesday night, there are more than 1.5 million diagnosed cases of COVID-19 in the U.S., with at least 93,439 deaths reported, according to data provided by Johns Hopkins University.
Editor's Note: This report has been updated to reflect new information from the Mayo Clinic.