Email: contact@AndrogenStudyGroup.org         Phone: (617) 939-1148     

Complaint to the Journal Oversight Committee at JAMA


View Complete Letter in pdf

(Click to open in new window. Right-click link and chose "Save link as.." to download and save)

August 1, 2014
Edward H. Shortliffe, MD, PhD
Chair, Journal Oversight Committee
Journal of the American Medical Association
 
Dear Dr. Shortliffe,
We are writing to you in your capacity as Chair of the Journal Oversight Committee (“The Committee”) at the Journal of the American Association (JAMA). The Androgen Study Group wishes to register with the Committee our grave concern that JAMA has violated accepted standards of medical journal ethics and editorial integrity. This concern relates to the publication of the article by Vigen et al in November, 2013, that reported increased cardiovascular risks in men receiving testosterone prescriptions (1). This article contradicted decades of research (2), created new concerns regarding risk for a frequently prescribed medication, and precipitated a brand new area of medical malpractice. Yet within a very short time, the worldwide scientific community concluded the study is meritless based on acknowledged errors and questionable methodology. Following publication JAMA failed to take appropriate action despite certain knowledge the study was invalid. In the name of medical science and the integrity of medical literature, we ask you to investigate JAMA’s actions in this case.
 
As we describe below, JAMA’s actions here violate several of its own published critical objectives. These include (numbers correspond to those provided by JAMA on its website):
 
1. “To maintain the highest standards of editorial integrity…”
2. “To publish original, important, valid, peer-reviewed articles…”
3. “To foster responsible and balanced debate on important issues that affect medicine, health, health care, and health policy”
4. “To provide physicians with continuing education in basic and clinical science to support informed clinical decisions and ongoing career development” 8. -“…JAMA has a social responsibility to improve the total human condition and to promote the integrity of science”
9. “To achieve the highest level of ethical medical journalism and to produce a publication that is timely, credible, and enjoyable to read”
 
The article was published on November 6, 2013, and reported that among a study population of 8709 men in the VA population with low serum T concentrations, men who received prescriptions for testosterone suffered a higher absolute rate of events (stroke, myocardial infarction, and death) than untreated men, at 25.7% vs 19.9%, respectively. This study was promoted heavily by JAMA and received enormous media coverage. Specifically, JAMA provided an early embargoed copy to health journalists to help them prepare their stories, and produced a video interview with the senior author that was placed on JAMA’s website and remains available for viewing as of today.
 
Two days following publication, on November 8, 2013, in a telephone call with editor-in-chief Howard Bauchner, MD, he was alerted that the reported values by Vigen et al had reversed actual percentages of individuals in each group that had suffered an adverse event (see attached email correspondence). Specifically, only 123 of 1223 (10.1%) men in the T group had an adverse event, compared with 1587 events in 7486 men (21.2%) in the untreated group. In other words, the actual percentage of events was lower, by half, in T-treated men compared with untreated men (3). This critically important result does not appear in the article, had apparently not been requested by the reviewers, and was not mentioned by the editorialist (4). Instead, the authors misreported their results as “absolute risk,” a term that indicates the raw events data supported the conclusion. Dr. Cappola’s editorial repeats this error in reporting, even providing the very numbers of events in each group that are easily totaled to show that the reported “absolute rates” are incorrect (see attached). A simple calculation from the raw data provided in the abstract was shared with Dr. Bauchner, clearly indicating that the results had been misreported. All media stories repeated the conclusion that “absolute rates” of adverse events were greater in the T group. Dr. Bauchner was advised of our position that JAMA bore responsibility for the misreporting of these distorted results.
 
Dr. Bauchner responded, “I cannot disagree with anything you’ve said.” Four days later he sent an email indicating that a revised pdf of the article was now available online (attached). The authors stood by their conclusions, but had changed “absolute rates” to “Kaplan-Meier estimated cumulative percentages with events,” a term that more accurately reflects the highly statistical approach to the data. This new version was made available on November 12, 2013, one week after publication (original and revised versions attached). Astonishingly, and troublingly, JAMA failed to acknowledge in any way that this was a revised version of the article in the public forum until the appearance two months later of a published correction, on January 15, 2014 (attached). In the meantime, the article that was made available to the public online carried no indication it was a corrected article, and the media and the public were not alerted to this fact.
 
In short, JAMA had actively promoted an article with a fundamental error that materially affected its reception in the medical and scientific community. JAMA’s editors were well aware of the enormous media attention given to this article by the time it was made aware of this critical error, yet it chose to avoid announcing the correction. The unfortunate impression is that JAMA wished to avoid bringing any attention to this correction until the media interest had died down. The decision to avoid or substantially delay public acknowledgment of the correction was an active form of deception.
 
This was a major breach of editorial ethics, and is clearly inconsistent with the goal of “maintaining the highest standard of editorial integrity.” It also had real-world ramifications. If the public had been made aware immediately that the results of this study were not as originally promoted, it is likely there would have been greater scrutiny of the article, and reevaluation of its merits. Moreover, it is difficult to understand how senior statisticians among the authors, no less JAMA’s reviewers, could have missed such a fundamental error in reporting results in the abstract and text.
 
On March 5, 2014, JAMA published our letter that challenged the authors’ exclusion of 1132 men that had suffered strokes and myocardial infarctions because they had subsequently received a testosterone prescription (attached). We argued the contribution of this group to the data was complete once they suffered an event, and therefore it was irrelevant whether or not they subsequently received testosterone. This large number of individuals who suffered an event would have increased by 71% the numbers of events in the untreated group, which in turn would have greatly influenced the results of the analysis.
 
In reply, the authors stated they had reviewed the data for these 1132 men and had made an “incorrect notation.” They then acknowledged a series of egregious data errors: 1) the number of excluded men in this group was actually 128, and not 1132, representing an error rate of 89% involving >1000 individuals, and 2) the numbers of men in a second excluded category (“abnormal coronary anatomy”) was increased from 397 to 1301, a difference of >900 individuals, representing an error rate of 69%, and 3) astonishingly, they revealed that 100 of the 1132 “men,” or 9% of the group, were discovered to be women! This information was published as a second correction on the same date (March 5, 2014) as publication of letters and authors’ reply (attached). The authors and JAMA stand by the results of this study without providing any assurance that the remainder of the dataset is not equally inaccurate.
 
The magnitude and quality of these errors destroy the credibility of the study. In order “to promote the integrity of science” JAMA was obligated to either retract the study or demand outside review of the dataset in order to confirm the results are accurate. As noted by an author of the stem cell study in Nature that was recently retracted after concerns were raised regarding a portion of the data, “In science, the integrity of data is the foundation for credible findings.” These errors add to the disturbing impression of scientific carelessness by the authors. It is mystifying why JAMA would continue its support of this article, particularly as its stated mission is to publish “important” and “valid” studies.
Twenty-nine medical societies and more than 160 of the leading testosterone researchers from 32 countries have called for retraction of the study (see list at www.androgenstudygroup.org), citing “gross data mismanagement and contamination” that rendered the study “no longer credible.” According to retraction guidelines published by the Committee on Publication Ethics, it does not matter whether errors were intentional (misconduct) or unintentional (mistakes) (5). Data that are not credible should be regarded as “misinformation,” and studies that contain such misinformation should be retracted to preserve the integrity of medical literature. Individuals calling for retraction include 8 emeritus professors, 9 current or former journal editors, and 15 society presidents. Societies calling for retraction are as follows:
 
American Society for Men’s Health (ASMH)
Brazilian Society of Endocrinology and Metabolism
Canadian Men's Sexual Health Council, an affiliate of the CUA
Canadian Society for the Study of Men’s Health (CSSAM)
European Society for the Study of the Aging Male (ESSAM)
European Society for Sexual Medicine (ESSM)
German Society for Men and Health
Indonesian Andrologist Association
International Society for Men’s Health (ISMH)
International Society for Sexual Medicine (ISSM)
International Society for the Study of the Aging Male (ISSAM)
Irish Association of Sexual Medicine
Italian Society of Andrology
Italian Society of Andrology and Sexual Medicine
Japan ASEAN Council for Men’s Health and Aging
Japanese Society for Men’s Health
Korean Society for Sexual Medicine and Andrology
Malaysian Men’s Health Initiative
Malaysian Society of Andrology and the Study of the Aging Male
Mens Health Initiative of British Columbia (Canada)
Mexican Association of Bone and Mineral Metabolism
Middle East Society for Sexual Medicine
Russian Society for Men’s Health
South Asian Society for Sexual Medicine
Sexual Medicine Society of North America
Sociedade Latinoamericana de Medicina Sexual (Latin American Society for Sexual Medicine)
The Society for Men's Health, Singapore
Society for the Study of Androgen Deficiency
Society for the Study of Andrology and Sexology, Singapore (SSASS)
 
The errors in the article by Vigen et al, particularly the inclusion of nearly 10% of an ineligible population in a dataset, have caused this article to be the object of both ridicule and outrage at scientific meetings and in the literature. Imagine a prostate cancer study that was discovered to include nearly 10% women! The authors have asserted these errors do not matter because they occurred in an excluded population, and their primary analysis remains unchanged. In an email, Dr. Bauchner justified his denial of our call for retraction based on the same argument (attached). This is a bizarre argument, since it is abundantly clear that the concern is that if one portion of the dataset is found to be so full of errors, then one must be concerned that the same degree of inaccuracy extends throughout the dataset. JAMA has failed to take any steps to assure that the primary data are accurate, and it thus violates its own objective of producing a publication that is “credible.”
 
Curiously, Dr. Bauchner’s resistance to ensuring accuracy of the data contradicts his own comments on editorial policy at JAMA regarding statistical analysis (6), published in July 2013, just 4 months prior to publication of the article by Vigen et al: “If concerns are raised in the process of editorial evaluation, we reserve the right to request the entire data set from authors to conduct our own statistical analysis…As with all manuscripts, the first priority in decisions about publication will always be the integrity of the research.” Although we note that Dr. Bauchner’s comments here pertained to the evaluation of manuscripts prior to publication, if the “integrity of the research” is “the first priority,” we see no reason why this should not equally apply to manuscripts post-publication when major errors are revealed.
 
Moreover, the authors were allowed by JAMA in their reply to letters to mischaracterize this series of errors as “an incorrect notation” in their original work. This is a gross distortion that minimizes the seriousness of the errors. An incorrect notation suggests a typographical error, or a single numerical error in the publication. In fact it was a series of errors, involving data entry, data review and oversight, data analysis, and publication. This mischaracterization of serious and multiple errors by JAMA is deceitful, and an ethical violation, falling well below the standard of transparency, clarity, and corrective actions for a journal that aims for “the highest standards of editorial integrity.”
 
These issues are more than sufficient for us to ask the Oversight Committee to investigate the actions of JAMA’s editorial board, and to support our call for retraction of this article. However, there have additionally been a series of troubling events which also raise serious ethical and editorial concerns which we wish to bring to your attention. These are as follows:
 
1. JAMA knew, or should have known, that the statistical methodology used by the authors has never been validated. The key citation in Methods, (Reference #23 in Vigen et al) by Xu et al (7), was co-authored by senior author Michael Ho. That study explored the use of inverse propensity weighting when applied to Kaplan-Meier methods in time-varying treatment, which is the same methodology applied in Vigen et al. The conclusions of this citation by Xu et al, so central to the study by Vigen et al and published only one year earlier (2012), concluded: “Clearly, assessing and confirming adequate covariate balance in IPTW time-varying models is challenging and needs further study... Further work with simulations and contrasts to other methods and other study applications would help elucidate the advantages and disadvantages of this approach." At the time of publication of Vigen et al, none of the 5 other citations of Xu et al had used the methodology to report results. This study thus relies on a new, inadequately studied statistical approach that has not yet been accepted by the statistical community, and must therefore be considered unvalidated. It is disturbing that the reviewers and JAMA’s editorial board would allow this to go unchallenged.
 
2. We wonder about the review process itself, given this and other previously stated concerns. Were the reviewers friends of the authors? We believe it would be worthwhile interviewing them to determine why they failed to note the misreporting of results, the improper exclusion of men who were treated with T prescriptions after an adverse event, and why they believed this unvetted methodology was valid.
 
3. The authors elected to not respond to all questions raised in letters, citing lack of space (see reply by authors to letters, attached). These were letters selected by JAMA’s editors, presumably because the editor(s) believed the concerns within those letters merited a response. In our own case, the editors substantially edited our letter, removing certain questions and editing others. It is troubling that JAMA would allow the authors to avoid responding to questions they had deemed worthy. This gives the impression the authors had something to hide. JAMA failed in its editorial duty to ensure that the authors responded to all questions. The end-result is a lack of transparency, compromise of the scientific process, and the appearance of allowing the authors to hide behind JAMA’s skirt.
 
4. The authors failed to provide an adequate explanation for the improper exclusion of men who received T prescriptions after suffering an MI or stroke. The FDA agreed this was improper, writing, “Finally, the exclusion of 128 patients who experienced MI or stroke before initiating testosterone was not appropriate. These patients should have been included in the analysis…” (see attached analysis by FDA).
 
5. We find it curious, and troubling, that the lead author (Rebecca Vigen) did not participate in the response to letters.
 
6. The authors performed a post-hoc analysis of their data to include the 128 men who had been improperly excluded. Even though 128 events should have been added to the untreated group with the lower rate of events, the authors claimed this resulted in a stronger hazard ratio (1.30 instead of the original 1.29), with a stronger 95% CI (1.06-1.60 instead of the original 1.04-1.58). Either the authors have made a mathematical error, or their model makes no sense.
 
7. The results are unverifiable. The authors have not published their statistical programming for their analysis on SAS or made it available for inspection anywhere else. Even if an outside party were given the original dataset, it would be impossible to reproduce the results. This violates a basic tenet of the publication of scientific research, namely that enough information regarding methods should be provided so that others may attempt to replicate and thereby verify the results. The inability to examine the methods means also that any inadvertent errors in programming would escape detection.
 
8. Over 50 variables were used for adjustment, some of which have never been associated with risk for CV events, eg, transesophageal echocardiogram and transthoracic echocardiogram. The authors also failed to include obvious variables that have been shown to be associated with CV events, such as number of medications, and baseline serum testosterone, which differed significantly between groups, and could well have influenced results. Any reviewer familiar with testosterone literature would be aware of the known inverse relationship between T concentrations and mortality.
 
9. Figure 1 (study cohort) has now been the subject of a published correction but remains available for view with the incorrect values for several items. Much of this figure is inaccurate (see annotated pdf of revised article, attached), even beyond the acknowledged errors. For example, the initial cohort lists 23,173 “men” when it has now been revealed that this value also includes women. A critical, unanswered question is how many women were inadvertently included in the final study population of 8709 individuals. The accuracy of other values for excluded individuals is also highly questionable. For example, 12 men were excluded for having PSA value >4 ng/dl, representing 0.05% of the initial study pool (N=23,173). It is not believable that only 5 men per 10,000 had an elevated PSA. In contrast, this value was 15% in the PCPT trial (8), a 300-fold difference.
 
10. Figure 2 (Kaplan-Meier curves) is misleading and nearly impossible to understand. This is the heart of the study, as the only statistically significant difference between groups was derived from these curves. There is no indication that these curves represent adjusted values, let alone for >50 variables. It is also misleading because at what appears to be end of the study (2000 days) the total percent with events in the T group is presented as greater than 30% (<70% survival) when the actual percentage of events in this group was only 10.1%. It is impossible to understand is that the authors indicate for the first time in their reply to letters that time zero for the T group began with filling of their first T prescription, which occurred at a median of 531 days following coronary angiography, whereas time zero for the untreated men was coronary angiography. Is this reflected in the X-axis? Should the legend for the X-axis of “days” be changed to “day-equivalents” or similar? See annotated copy of article (attached) for clarification.
 
11. The primary results of the revised version of the study still appear likely to be incorrect or inaccurate. The revised language in the abstract reads: “At 3 years after coronary angiography, the Kaplan-Meier estimated cumulative percentages with events were 19.9% in the no testosterone therapy group vs 25.7% in the testosterone therapy group…” Although time zero for the untreated group began with the date of coronary angiography, time zero for the T group began a median of 531d (approx 1.5y) after coronary angiography. It continues to be unclear how this time difference impacted the reporting of results. Were the values for estimated cumulative events accurately described for the time period 3y after coronary angiography? At that time approximately half the T group would have been exposed to T for less than 1.5y. It is almost certainly incorrect to describe results as stated. Also, the revised abstract still refers to “absolute risk difference of 5.8%” even though the term “absolute risk” has been replaced in the reporting of results. This appears to be incorrect and misleading, based on the definition of “absolute risk difference” in JAMAEvidence, which refers to actual percentages of events between groups.
 
12. Finally, we note the additional criticism of the FDA: “It is also unclear why the authors excluded 1,301 participants for not having coronary anatomy data (CAD status), considering the wealth of baseline information collected on medical and drug history. It is unclear how this exclusion might also have affected the risk estimate.” We understand this criticism and others are most relevant for the authors. However, it is JAMA’s editorial board that selected this paper for publication, and subsequently promoted it despite these and other serious criticisms. JAMA must bear ultimate responsibility for its publication.
 
13. Note that we have posed several of these scientific questions via email to senior author, Michael Ho, who replied that these questions should be directed to JAMA.
 
This article is a mess, and JAMA has behaved badly. Something is terribly amiss when a premier medical journal publishes such an obviously weak study that contradicts well-established literature, and in so doing, fosters fear among the public. The concern is heightened when the journal’s response to inescapable evidence that the study is meritless is to deceive, distort, stonewall, and dig in. JAMA’s behavior suggests it is more interested in sensationalism and media coverage than scientific accuracy and integrity.
 
We recognize this is a long letter with a long list of concerns. It is unfortunate that so many troubling irregularities have occurred with publication of this article and with JAMA’s actions or inactions since that time. We note the following violations of JAMA’s critical objectives as noted at the outset of this letter:
 
Objective #1- JAMA has failed to maintain the “highest standards of editorial integrity.”
 
Objective #2- By no reasonable standard can this article be considered “valid.”
 
Objective #3- There can be no “responsible and balanced debate” when JAMA publishes misreported data using unvalidated statistical methodologies, and fails to note that the actual percentages of events in two groups were reversed by this methodology.
 
Objective #4- Publication of an erroneous study undermines the goal of helping physicians make “informed clinical decisions.” In this case, JAMA is responsible for a media firestorm that has affected clinical practice, all based on a meritless study.
 
Objective #8- JAMA has failed terribly in its goal of “promoting the integrity of science.”
 
Objective #10- JAMA has violated basic rules of “ethical medical journalism” and has published a study that cannot be considered “credible.”
 
JAMA has enormous responsibility here. It has created unfounded concerns among millions of men treated appropriately for testosterone deficiency, caused patients to question the care provided by their physicians, and prompted many men to discontinue treatment even among those who had clearly benefitted. It precipitated a safety bulletin by the FDA that will result in a public review on September 17, 2014. And it has birthed a new area of medical malpractice. Although Dr. Ho declined responsibility for any such effects by citing his call for further research at the end of the article,this stance is contradicted by the final sentence of the abstract, which clearly expresses confidence in the validity of the reported results and its utility in clinical decision-making: “These findings may inform the discussion about the potential risks of testosterone therapy.”
 
We note that the global community of testosterone experts has called for retraction of this study. Even the FDA has rejected the conclusions of the study, which is particularly noteworthy since as a government agency dedicated to public health and safety, it routinely errs on the side of caution in announcing warnings about medication risks. In this case, the FDA concluded (see attachd): “Given the described limitations of the study by Vigen et al. it is difficult to attribute the reported findings to testosterone treatment.”
 
We believe that with regard to this article JAMA has failed in its review process, its selection of this article for publication, its promotion of this article, its post-publication response to important revealed errors, and in its editorial ethics and integrity. We believe these failures have damaged public health, compromised the medical literature, injured patient-physician relationships, and furthered public suspicions regarding scientific research.
We hope you will give serious consideration to the issues we have raised, and we look forward to your response.
 
Sincerely,
Abraham Morgentaler, MD (Chairman), on behalf of the Androgen Study Group
 
André Guay, MD
Mohit Khera, MD
Martin Miner, MD
Abdulmaged Traish, PhD
 
* Note about the Androgen Study Group. The Androgen Study Group is a multidisciplinary group with extensive clinical and research experience with T deficiency (hypogonadism) and its treatment, representing the disciplines of urology, endocrinology, family medicine, and basic science research. Our mission is to promote accurate reporting of results of testosterone research. Our members teach medical and doctoral students, residents, and fellows at the medical schools of Harvard, Tufts, Brown, Boston University, and Baylor College of Medicine; participate as faculty or course directors at more than two dozen CME events annually; and have served on national and international clinical guidelines and recommendations committees regarding T therapy. We are wholly dedicated to the science of testosterone in men, and to the wellbeing of our patients. The ASG receives no funding from any pharmaceutical company, and no member has any direct financial stake in any pharmaceutical company manufacturing or selling T products. However, several ASG members have received payments from pharmaceutical companies for consulting, participation in scientific advisory boards, research grants, and lecture honoraria.
 
References
 
1. Vigen R, O’Donnell CI, Baron AE, et al. Association of testosterone therapy with mortality, myocardial infarction, and stroke in men with low testosterone levels. JAMA. 2013 ;310(17):1829-1836.
2. Morgentaler A, Kacker R. Andrology: Testosterone and cardiovascular risk--deciphering the statistics. Nat Rev Urol. 2014 Mar;11(3):131-2.
3. Traish AM, Guay AT, Morgentaler A. Death by testosterone? We think not! J Sex Med. 2014 Mar;11(3):624-9.
4. Cappola AR. Testosterone therapy and risk of cardiovascular events in men. JAMA 2013; 310: 1805-6.
5. Wager E, Barbour V, Yentis S, Kleinert S; COPE Council. Retractions: guidance from the Committee on Publication Ethics (COPE). Obes Rev. 2010 Jan;11(1):64-6.
6. Bauchner H. Editorial Policies for Clinical Trials and the Continued Changes in Medical Journalism. JAMA 2013; 310:149-150.
7. Xu S, Shetterly S, Powers D, Raebel MA, Tsai TT, Ho PM, Magid D. Extension of Kaplan-Meier methods in observational studies with time-varying treatment. Value Health. 2012 Jan;15(1):167-74.
8. Thompson IM, Pauler DK, Goodman PJ, Tangen CM, Lucia MS, Parnes HL, Minasian LM, Ford LG, Lippman SM, Crawford ED, Crowley JJ, Coltman CA Jr. Prevalence of prostate cancer among men with a prostate-specific

ASG News

29 Medical Societies Support
ASG Action


ASG is an independent professional organization which does not receive any funding from pharmaceutical companies.

Get In Touch

 Address:
200 Boylston St, #A309,
Chestnut Hill, MA 02467, US


  Phone: (617) 277-5000

  Web: www.AndrogenStudyGroup.org

The Androgen Study Group