Draft synthesis: incretin receptor agonists for metabolic… — submissions
The 12 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
12 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
An indirect comparison is presented without an assessment of transitivity
The respondent read the draft review of In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological… and has confined this submission to a single matter.
The draft compares two interventions through a common comparator. The respondent states that the trials contributing to each side differ in background therapy and in baseline severity, and that the resulting estimate assumes a similarity the draft does not test.
The respondent proposes that transitivity be assessed explicitly and reported before any indirect estimate is presented.
The secretariat accepts this submission. An indirect estimate presented without a transitivity assessment invites a reader to treat an assumption as a result.
Transitivity is now assessed against a stated list of effect modifiers and reported before any indirect estimate, and where the assessment fails the contrast is reported as unestimable with the reason rather than estimated with a caveat.
The same concept is given three different names in one document
The respondent’s comment on the draft review of In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological… arises from comparing the included set against the respondent’s own knowledge of the field.
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
Intention-to-treat and efficacy-estimand results are combined without distinction
The respondent has read the draft synthesis on In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological… and makes one submission.
Several contributing trials report both a treatment-policy result and an efficacy result that censors at discontinuation. The draft draws from whichever is reported first in each paper.
The respondent, a trial statistician, states that the two answer different questions, that the difference is substantial where discontinuation is common, and that a synthesis should choose one and say which.
The secretariat accepts this submission. Mixing estimands within a single pooled estimate is a defect of the synthesis rather than of the trials.
The treatment-policy estimand is used throughout as the primary analysis, the efficacy estimand is reported as a secondary analysis where available, and the estimand used is stated in every row of the summary of findings.
Risk-of-bias judgements are reported as an overall rating without the domains
The draft review of In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological… was read against its registered protocol.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
A sponsor trial meeting the eligibility criteria was excluded
The respondent has read In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological… and submits on a matter of presentation.
The submission is made on behalf of the sponsor. It identifies a completed trial of the sponsor's compound that meets the stated eligibility criteria and does not appear in the included set, and supplies the trial report and the registry record.
The sponsor asks that the trial be included and the estimate recomputed. No view is expressed on the direction the recomputation should take.
The secretariat accepts this submission in part. The trial does meet the criteria and has been included. The recomputed estimate is materially unchanged, which the response states explicitly so that the outcome of the correction is on the record.
The trial is added to the included set, the estimate and the certainty rating have been recomputed, and the screening record now states why the trial was missed, which was a database indexing gap rather than a screening judgement. The submission is identified as an industry submission.
A surrogate outcome is used as the anchor without validation evidence
The respondent notes that the review question — In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The anchor outcome in the draft is a surrogate. The respondent states that the relationship between the surrogate and the outcome a decision turns on is itself an evidential question, and that the draft assumes it.
The respondent proposes that no surrogate serve as an anchor.
The secretariat accepts this submission in part. The anchor is retained where the surrogate is the only outcome the contributing trials measured, and the validation question is addressed rather than assumed.
Where the anchor is a surrogate, the synthesis now states the evidence for the surrogate relationship, rates it separately, and downgrades the anchor rating for indirectness accordingly rather than carrying the surrogate as though it were the outcome of interest.
One trial appears to be included twice under two publications
This submission concerns the draft review of In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
Two included records report overlapping participant numbers, the same enrolment window and the same sponsor. If they are the same trial reported twice, its weight in the pooled estimate is doubled.
The respondent proposes that the review record, for each included study, the trial identifier rather than the publication, and cluster publications under it.
The secretariat accepts this submission. Counting publications rather than trials is the commonest way a review overstates its evidence base.
Included studies are now recorded against a trial identifier, with publications listed beneath it, and the included-studies table reports the number of trials and the number of reports separately.
Efficacy outcomes are rated for certainty and harms are not
The respondent submits on In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological…. The point would apply equally to any document in the series.
The draft assigns certainty ratings to the efficacy outcomes and reports harms narratively without ratings. The respondent states that the asymmetry implies harms are less amenable to assessment when they are simply less well measured.
The respondent proposes that harms carry certainty ratings on the same scale, with the reasons for downgrading stated.
The respondent notes submission 002 above and does not repeat the ground it covers.
The secretariat accepts this submission. Rating one side of the balance and not the other produces a document that cannot be used to weigh them.
Every reported harm now carries a certainty rating on the same scale as the efficacy outcomes, with the downgrade reasons stated, and discontinuation for adverse events appears in the summary of findings rather than in an annex.
Two included studies do not meet the registered eligibility criteria
Having read the draft synthesis on In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological…, the respondent puts one point to the committee.
The respondent identifies two studies in the included set whose duration falls below the minimum stated in the protocol, and one excluded study that appears to meet every criterion.
The respondent proposes that the three be reassessed and that the outcome of the reassessment be recorded whichever way it goes.
The secretariat accepts this submission in part. On reassessment, one of the two included studies does fall below the minimum duration and has been removed. The second reports a duration that met the criterion in the protocol version registered at the time. The excluded study was excluded for a reason not clearly recorded, which was a documentation failure.
One study has been removed from the included set and the estimate recomputed, the exclusion reason for the third study has been corrected in the excluded-studies table, and the screening decisions are now recorded against the protocol version in force at the time of screening.
The search date is not on the face of the document
This submission addresses the draft synthesis on In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological… from the standpoint of a reader who will use the summary and not the appendices.
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
The document should not describe uses outside the approved indication
The respondent submits on the draft synthesis addressing In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
Crossover trials are pooled with parallel-group trials without adjustment
This is a submission on the draft review of In adults with metabolic dysfunction-associated steatohepatitis, do incretin receptor agonists improve histological outcomes, and does histological…, made from a statistical standpoint.
The analysis combines parallel-group and crossover designs by taking the first-period result from the crossovers, which discards half the data, or by taking the paired result, which is not comparable with an unpaired one. The methods do not say which was done.
The respondent proposes that the handling be stated and, where crossover data are used, that a sensitivity analysis excluding them be reported.
The secretariat accepts the requirement to state the handling and declines to require a sensitivity analysis in every case.
The methods section now states the handling of every non-parallel design. A sensitivity analysis is reported where crossover trials contribute more than a fifth of the weight.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.