Draft synthesis: Oral compared with injectable presentations… — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Subgroup findings are reported that were not registered in the protocol
This is a submission on the draft review of Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable…, made from a statistical standpoint.
The draft reports differences between subgroups that do not appear in the registered protocol. The respondent states that unregistered subgroup analysis is hypothesis-generating and that the draft presents it in the same form as the pre-specified results.
The respondent proposes that unregistered analyses be removed.
The secretariat accepts this submission in part. The analyses are retained and relabelled rather than removed, because removing an analysis that was conducted leaves no record that it was.
Every subgroup analysis is now labelled as pre-specified or post hoc against the registered protocol, post hoc analyses are reported in a separate subsection without a certainty rating, and the protocol version against which the labelling was made is stated.
Risk-of-bias judgements are reported as an overall rating without the domains
The respondent has read Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… in draft and makes a single submission.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
The document should not describe uses outside the approved indication
The respondent read the draft review of Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… and has confined this submission to a single matter.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
A surrogate outcome is used as the anchor without validation evidence
The respondent submits on Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable….
The anchor outcome in the draft is a surrogate. The respondent states that the relationship between the surrogate and the outcome a decision turns on is itself an evidential question, and that the draft assumes it.
The respondent proposes that no surrogate serve as an anchor.
The respondent has read submission 002 above and makes this submission independently of it.
The secretariat accepts this submission in part. The anchor is retained where the surrogate is the only outcome the contributing trials measured, and the validation question is addressed rather than assumed.
Where the anchor is a surrogate, the synthesis now states the evidence for the surrogate relationship, rates it separately, and downgrades the anchor rating for indirectness accordingly rather than carrying the surrogate as though it were the outcome of interest.
Studies not published in English were excluded without assessment
This submission addresses the draft synthesis on Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… from the standpoint of a reader who will use the summary and not the appendices.
The respondent states that a language restriction was applied at screening and that for some of the compounds in scope a substantial literature is published in other languages.
The respondent proposes that the restriction be removed and the review re-run.
The secretariat accepts this submission in part. The restriction is removed prospectively and the records already excluded on language grounds have been retrieved and screened on title and abstract in translation. The full re-run proposed is not undertaken, and the limitation is recorded rather than concealed.
Language restriction is removed from the protocol for the series, the records previously excluded on that ground are listed with their screening outcome, and the residual limitation is stated in the abstract of this review.
Registered trials that never reported are not counted anywhere in the review
Having read the draft under consultation, which concerns Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable…, the respondent submits as follows.
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
Analytical surveys are treated as evidence about suppliers when they establish only what was in a sample
The respondent read Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… in draft. The point applies to it and to the series generally.
Several included surveys purchased material anonymously and analysed it. The respondent states that where the chain from a named supplier to the analysed vial cannot be documented, the result describes a sample and not a supplier.
The respondent proposes that provenance be an explicit eligibility dimension, with surveys reported separately according to whether it could be established.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 006.
The secretariat accepts this submission. Moderate certainty evidence from the included surveys supports statements about material circulating in a market; it does not support statements about any named supplier's output.
Provenance is now assessed for every included survey and reported as a study characteristic, surveys without establishable provenance are reported separately rather than pooled with those that have it, and no supplier-level inference is drawn from a sample-level result.
Studies using different outcome definitions are pooled into one estimate
This submission concerns the draft review of Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The contributing studies define the outcome in at least two ways, and the draft pools them. The respondent states that the resulting estimate is an average across definitions rather than an estimate of any one quantity.
The respondent proposes that studies be pooled only within a definition, and that the definitions be reported separately with their own certainty ratings.
Submission 006 concerns the same document. The respondent’s point is a different one.
The secretariat accepts this submission. Pooling across definitions produces a figure with no referent, and the draft did it without saying so.
Studies are now pooled only within an outcome definition, each definition is reported with its own estimate and certainty rating, and the summary of findings states which definition each row concerns.
Quantitative claims are reproduced without the method that produced them
The respondent has read the draft synthesis on Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… and makes one submission.
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
The document set should be published in translation
The respondent submits on the draft synthesis addressing Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
Point estimates are given without an interval
Having read the draft synthesis on Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable…, the respondent puts one point to the committee.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
The same concern is used to downgrade in two domains
The respondent has read Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… and submits on a matter of presentation.
The draft downgrades for both indirectness and imprecision citing the same feature of the contributing trials. The respondent states that this double-counts a single concern and produces a rating lower than the framework supports.
The respondent proposes that the rating be raised by one level.
The secretariat accepts this submission in part. The double count is real in one of the two outcomes identified and not in the other, where the two domains rest on different features.
The rating for the first outcome has been raised by one level and the domain reasoning restated, and the reasoning for the second has been rewritten to make clear that the two downgrades rest on different features of the evidence.
The search strategy as published cannot be re-run
The draft review of Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… was read against its registered protocol.
The strategy is described in prose rather than reproduced as executed. The respondent, an information specialist, attempted to reconstruct it and obtained a different yield, which may reflect the reconstruction or the original.
The respondent proposes that the strategy be published line by line as run in each database, with the interface, the date and the number of records retrieved at each line.
Submission 011 concerns the same document. The respondent’s point is a different one.
The secretariat accepts this submission. A strategy that cannot be re-run cannot be checked, and a review whose search cannot be checked rests on an assertion.
The full line-by-line strategy for each database is now published with the interface, the run date and the yield at each line, and the total retrieved is reconciled against the screening flow.
Efficacy outcomes are rated for certainty and harms are not
The respondent notes that the review question — Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
The draft assigns certainty ratings to the efficacy outcomes and reports harms narratively without ratings. The respondent states that the asymmetry implies harms are less amenable to assessment when they are simply less well measured.
The respondent proposes that harms carry certainty ratings on the same scale, with the reasons for downgrading stated.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 002.
The secretariat accepts this submission. Rating one side of the balance and not the other produces a document that cannot be used to weigh them.
Every reported harm now carries a certainty rating on the same scale as the efficacy outcomes, with the downgrade reasons stated, and discontinuation for adverse events appears in the summary of findings rather than in an annex.
A funnel plot is presented for a set too small to interpret it
The respondent’s comment on the draft review of Does an oral presentation of a glucagon-like peptide-1 receptor agonist achieve an efficacy and tolerability profile comparable with an injectable… arises from comparing the included set against the respondent’s own knowledge of the field.
The draft includes a funnel plot for an outcome contributed by fewer than ten studies. The respondent states that asymmetry cannot be assessed reliably at that number and that presenting the plot invites a conclusion the data cannot support.
The respondent proposes that the plot be removed and replaced with a statement that publication bias could not be assessed.
The respondent supports submission 013 so far as it goes and adds the matter set out here.
The secretariat accepts this submission. Presenting an uninterpretable graphic is not a neutral act.
The funnel plot is removed for every outcome contributed by fewer than ten studies, and replaced by a statement that small-study effects could not be assessed at that number, together with the count of registered trials identified without posted results.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.