Draft synthesis: Gastrointestinal adverse events with… — submissions
The 12 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
12 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Baseline imbalance in a small contributing trial is not remarked on
The respondent has read the draft synthesis on What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… and makes one submission.
The respondent identifies a small contributing trial with a baseline difference in the outcome variable, and states that the difference is large enough to account for a share of the reported effect.
The respondent asks that the trial be excluded or that the imbalance be addressed.
The secretariat notes this submission. The imbalance is recorded in the risk-of-bias assessment for that trial, in the domain concerning the randomisation process, where the respondent may not have looked.
No amendment arises. The judgement and the supporting figures are already reported at domain level for that study, and a sensitivity analysis omitting it is reported in the results, which does not change the direction of the estimate.
The review question is drawn too broadly to be answerable
Having read the draft synthesis on What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with…, the respondent puts one point to the committee.
The respondent states that the population as registered spans groups in which the intervention would be expected to behave differently, and that a single estimate across them is uninformative.
The respondent proposes that the question be split and the review re-registered.
The respondent read submission 001 after drafting this one and has not altered it, the two points being distinct.
The secretariat does not accept this submission. The breadth was registered before screening, the pre-specified subgroups address the variation the respondent identifies, and re-registering after the results are known would convert a pre-specified analysis into a post hoc one.
The question stands as registered. The point is recorded in the limitations, and the assessment committee has noted it for the protocol of the next version of this review, which will be registered before the search is run. The submission remains published in full.
Analytical surveys are treated as evidence about suppliers when they establish only what was in a sample
The respondent read the draft review of What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… and has confined this submission to a single matter.
Several included surveys purchased material anonymously and analysed it. The respondent states that where the chain from a named supplier to the analysed vial cannot be documented, the result describes a sample and not a supplier.
The respondent proposes that provenance be an explicit eligibility dimension, with surveys reported separately according to whether it could be established.
The secretariat accepts this submission. Moderate certainty evidence from the included surveys supports statements about material circulating in a market; it does not support statements about any named supplier's output.
Provenance is now assessed for every included survey and reported as a study characteristic, surveys without establishable provenance are reported separately rather than pooled with those that have it, and no supplier-level inference is drawn from a sample-level result.
References should carry a persistent identifier for every cited source
The respondent’s comment on the draft review of What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… arises from comparing the included set against the respondent’s own knowledge of the field.
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
Attrition is assessed in the risk-of-bias domains and not reported as a number
The respondent submits on the draft synthesis addressing What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The risk-of-bias assessment records attrition as a judgement. The included-studies table does not carry the proportion analysed against the proportion randomised, so a reader cannot see whether a judgement of low risk was made over five per cent attrition or twenty-five.
The respondent proposes a numeric attrition column in the included-studies table.
The secretariat accepts this submission. A domain judgement is more useful when the number behind it is visible.
The included-studies table now carries randomised, analysed and attrition columns, and the risk-of-bias judgement is made against the figure the reader can see.
Regulatory assessment documents were not searched
The respondent has read What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… and submits on a matter of presentation.
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
Studies using different outcome definitions are pooled into one estimate
This submission addresses the draft synthesis on What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… from the standpoint of a reader who will use the summary and not the appendices.
The contributing studies define the outcome in at least two ways, and the draft pools them. The respondent states that the resulting estimate is an average across definitions rather than an estimate of any one quantity.
The respondent proposes that studies be pooled only within a definition, and that the definitions be reported separately with their own certainty ratings.
Submission 005 concerns the same document. The respondent’s point is a different one.
The secretariat accepts this submission. Pooling across definitions produces a figure with no referent, and the draft did it without saying so.
Studies are now pooled only within an outcome definition, each definition is reported with its own estimate and certainty rating, and the summary of findings states which definition each row concerns.
Intention-to-treat and efficacy-estimand results are combined without distinction
This submission concerns the draft review of What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
Several contributing trials report both a treatment-policy result and an efficacy result that censors at discontinuation. The draft draws from whichever is reported first in each paper.
The respondent, a trial statistician, states that the two answer different questions, that the difference is substantial where discontinuation is common, and that a synthesis should choose one and say which.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 003.
The secretariat accepts this submission. Mixing estimands within a single pooled estimate is a defect of the synthesis rather than of the trials.
The treatment-policy estimand is used throughout as the primary analysis, the efficacy estimand is reported as a secondary analysis where available, and the estimand used is stated in every row of the summary of findings.
The search strategy as published cannot be re-run
The respondent notes that the review question — What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The strategy is described in prose rather than reproduced as executed. The respondent, an information specialist, attempted to reconstruct it and obtained a different yield, which may reflect the reconstruction or the original.
The respondent proposes that the strategy be published line by line as run in each database, with the interface, the date and the number of records retrieved at each line.
The secretariat accepts this submission. A strategy that cannot be re-run cannot be checked, and a review whose search cannot be checked rests on an assertion.
The full line-by-line strategy for each database is now published with the interface, the run date and the yield at each line, and the total retrieved is reconciled against the screening flow.
The review protocol is described but its registration record is not linked
This submission concerns What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… and a convention used across the Institute’s output.
The respondent asks for the prospective registration record of the review protocol so that the registered outcomes can be compared with the reported ones.
The respondent states that without it the claim of prospective registration cannot be checked.
The secretariat accepts this submission in part. The protocol is published in full on the Institute site with its version history, which permits the comparison the respondent asks for. Where an external registration identifier is not held by the Institute, the review says so rather than supplying one.
Each synthesis now links its own protocol with the version in force at screening, and states explicitly where an external registration identifier is not held, so that the absence is a recorded fact.
Absence of evidence is presented in a form a reader will take as negative evidence
The draft review of What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with… was read against its registered protocol.
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
Studies not published in English were excluded without assessment
This is a submission on What is the incidence of gastrointestinal adverse events and of discontinuation for adverse events with incretin receptor agonists compared with….
The respondent states that a language restriction was applied at screening and that for some of the compounds in scope a substantial literature is published in other languages.
The respondent proposes that the restriction be removed and the review re-run.
The secretariat accepts this submission in part. The restriction is removed prospectively and the records already excluded on language grounds have been retrieved and screened on title and abstract in translation. The full re-run proposed is not undertaken, and the limitation is recorded rather than concealed.
Language restriction is removed from the protocol for the series, the records previously excluded on that ground are listed with their screening outcome, and the residual limitation is stated in the abstract of this review.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.