Draft synthesis: Cardiovascular benefit in primary compared… — submissions
The 10 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
10 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
A single-trial result is presented in the visual form of a pooled estimate
The respondent read the draft review of Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention… and has confined this submission to a single matter.
An outcome contributed by one trial is presented in the same forest plot format as outcomes contributed by several. The respondent states that the format carries an implication of replication that a single trial does not have.
The respondent proposes that single-trial outcomes be presented differently and labelled as unreplicated.
The secretariat accepts this submission. The form of a graphic is part of what it asserts.
Outcomes contributed by a single trial are now reported without a pooled diamond, labelled as unreplicated, and downgraded for imprecision or inconsistency according to the framework rather than presented as a synthesis.
An indirect comparison is presented without an assessment of transitivity
The respondent submits on the draft synthesis addressing Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The draft compares two interventions through a common comparator. The respondent states that the trials contributing to each side differ in background therapy and in baseline severity, and that the resulting estimate assumes a similarity the draft does not test.
The respondent proposes that transitivity be assessed explicitly and reported before any indirect estimate is presented.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 001.
The secretariat accepts this submission. An indirect estimate presented without a transitivity assessment invites a reader to treat an assumption as a result.
Transitivity is now assessed against a stated list of effect modifiers and reported before any indirect estimate, and where the assessment fails the contrast is reported as unestimable with the reason rather than estimated with a caveat.
The axis of the forest plot exaggerates a small difference
The respondent has read the draft synthesis on Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention… and makes one submission.
The plot is drawn on an axis spanning a narrow range, so an effect of no clinical consequence occupies most of the width of the figure. A reader who takes the visual impression rather than the number will overstate the finding.
The respondent proposes that a minimally important difference be marked on every forest plot where one has been established for the outcome.
The secretariat accepts the proposal where a minimally important difference exists and declines to invent one where it does not.
Forest plots now carry a marked minimally important difference wherever a published one exists for the outcome, with its source cited. Where none exists the figure says so beneath the axis.
Two included studies do not meet the registered eligibility criteria
The respondent notes that the review question — Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The respondent identifies two studies in the included set whose duration falls below the minimum stated in the protocol, and one excluded study that appears to meet every criterion.
The respondent proposes that the three be reassessed and that the outcome of the reassessment be recorded whichever way it goes.
The secretariat accepts this submission in part. On reassessment, one of the two included studies does fall below the minimum duration and has been removed. The second reports a duration that met the criterion in the protocol version registered at the time. The excluded study was excluded for a reason not clearly recorded, which was a documentation failure.
One study has been removed from the included set and the estimate recomputed, the exclusion reason for the third study has been corrected in the excluded-studies table, and the screening decisions are now recorded against the protocol version in force at the time of screening.
Results at materially different follow-up durations are pooled
The draft review of Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention… was read against its registered protocol.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The respondent has read submission 001 above and makes this submission independently of it.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
Efficacy outcomes are rated for certainty and harms are not
The respondent read Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention… in draft. The point applies to it and to the series generally.
The draft assigns certainty ratings to the efficacy outcomes and reports harms narratively without ratings. The respondent states that the asymmetry implies harms are less amenable to assessment when they are simply less well measured.
The respondent proposes that harms carry certainty ratings on the same scale, with the reasons for downgrading stated.
The secretariat accepts this submission. Rating one side of the balance and not the other produces a document that cannot be used to weigh them.
Every reported harm now carries a certainty rating on the same scale as the efficacy outcomes, with the downgrade reasons stated, and discontinuation for adverse events appears in the summary of findings rather than in an annex.
The search date is not on the face of the document
Having read the draft synthesis on Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention…, the respondent puts one point to the committee.
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
Statistical heterogeneity is treated as though it measured clinical heterogeneity
This is a submission on the draft review of Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention…, made from a statistical standpoint.
The draft reports a heterogeneity statistic and proceeds to pool where it is low. The respondent states that a low statistic in a small set of trials is uninformative, and that clinical and methodological similarity should be assessed before any statistic is consulted.
The respondent proposes that the decision to pool be justified on clinical grounds first and that the statistic be reported as a description rather than used as a threshold.
The respondent read submission 001 after drafting this one and has not altered it, the two points being distinct.
The secretariat accepts this submission in part. The decision to pool is now made on clinical and methodological grounds and stated as such. The statistic continues to be reported, because readers expect it and its absence would be read as concealment.
The decision to pool is now justified in prose against the population, intervention, comparator and outcome before any statistic is presented, and the heterogeneity statistic is reported with its confidence interval and a note of the number of contributing studies.
Overlapping primary studies across included reviews are counted more than once
The respondent’s comment on the draft review of Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention… arises from comparing the included set against the respondent’s own knowledge of the field.
The draft is an overview of reviews and several included reviews share primary studies. The respondent states that the overview reports the total number of participants across reviews as though the sets were disjoint.
The respondent proposes that overlap be assessed and reported, and that no total be given without correcting for it.
The secretariat accepts this submission. An inflated participant total overstates the evidence base by an amount the reader cannot see.
Overlap between included reviews is now assessed at primary-study level and reported in a citation matrix, and participant totals are reported for the union of primary studies rather than as a sum across reviews.
The search strategy as published cannot be re-run
This submission addresses the draft synthesis on Does the cardiovascular effect of a glucagon-like peptide-1 receptor agonist established in secondary prevention extend to a primary-prevention… from the standpoint of a reader who will use the summary and not the appendices.
The strategy is described in prose rather than reproduced as executed. The respondent, an information specialist, attempted to reconstruct it and obtained a different yield, which may reflect the reconstruction or the original.
The respondent proposes that the strategy be published line by line as run in each database, with the interface, the date and the number of records retrieved at each line.
The secretariat accepts this submission. A strategy that cannot be re-run cannot be checked, and a review whose search cannot be checked rests on an assertion.
The full line-by-line strategy for each database is now published with the interface, the run date and the yield at each line, and the total retrieved is reconciled against the screening flow.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.