Draft synthesis: incretin receptor agonists in adolescents — submissions
The 16 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
16 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Regulatory assessment documents were not searched
This submission concerns In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… and makes one point.
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
The timepoint at which each outcome was extracted was chosen after the data were seen
Having read the draft synthesis on In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and…, the respondent puts one point to the committee.
The methods state that outcomes were extracted at the latest available timepoint. Several contributing trials report the same outcome at three timepoints, and taking the latest is a choice that can be made in the knowledge of what each shows.
The respondent proposes that the extraction timepoint be prespecified in the protocol, and that where it was not, the review report the estimate at every reported timepoint so that the choice is visible.
The secretariat accepts this submission. An extraction rule applied after the data are visible is a degree of freedom and should be recorded as one.
The extraction timepoint is now prespecified in the protocol for new reviews. Where a review predates the requirement, estimates are reported at every timepoint the contributing trials report.
Registered trials that never reported are not counted anywhere in the review
This is a submission on the draft review of In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and…, made from a statistical standpoint.
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The respondent has read submission 001 and puts a further matter to the secretariat.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
The conclusion is stated more strongly than the certainty rating supports
The respondent submits on In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and….
The draft rates the anchor outcome at low certainty and then states the finding in the conclusion without qualification. The respondent states that the two sentences cannot both be true as written.
The respondent proposes that conclusion language be tied mechanically to the certainty rating, so that a low rating cannot produce an unqualified claim.
The respondent has read submission 003 and asks that this submission be considered with it.
The secretariat accepts this submission. The mismatch is the failure mode the certainty framework exists to prevent.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, so that a low certainty rating produces a statement that the evidence may suggest an effect and that the estimate is likely to change with further research.
The same concept is given three different names in one document
This submission concerns the draft review of In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and…. The respondent has conducted reviews on adjacent questions and the observation arises from that experience.
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
The search strategy as published cannot be re-run
Having read the draft under consultation, which concerns In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and…, the respondent submits as follows.
The strategy is described in prose rather than reproduced as executed. The respondent, an information specialist, attempted to reconstruct it and obtained a different yield, which may reflect the reconstruction or the original.
The respondent proposes that the strategy be published line by line as run in each database, with the interface, the date and the number of records retrieved at each line.
This submission should be read alongside submission 004, which arises on the same draft.
The secretariat accepts this submission. A strategy that cannot be re-run cannot be checked, and a review whose search cannot be checked rests on an assertion.
The full line-by-line strategy for each database is now published with the interface, the run date and the yield at each line, and the total retrieved is reconciled against the screening flow.
The comparator was given at a dose below the one in general use
The respondent notes that the review question — In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… — is answerable only if the contributing trials measured the same thing, and submits with that in view.
Two contributing trials compare the intervention against an active comparator titrated to a dose lower than that reached in ordinary practice. Pooling them with trials using a full comparator dose produces an estimate that flatters the intervention.
The respondent proposes that comparator dose be tabulated in the included-studies table and that a sensitivity analysis restricted to full-dose comparators be reported.
The respondent’s submission overlaps with submission 003 and was prepared without sight of it.
The secretariat accepts this submission. Comparator dose is a condition of an effect estimate and was not being recorded.
Comparator dose is now a column in the included-studies table, and a sensitivity analysis restricted to comparators at the dose in general use is reported wherever the trials permit it.
Attrition is assessed in the risk-of-bias domains and not reported as a number
The respondent has read the draft synthesis on In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… and makes one submission.
The risk-of-bias assessment records attrition as a judgement. The included-studies table does not carry the proportion analysed against the proportion randomised, so a reader cannot see whether a judgement of low risk was made over five per cent attrition or twenty-five.
The respondent proposes a numeric attrition column in the included-studies table.
The secretariat accepts this submission. A domain judgement is more useful when the number behind it is visible.
The included-studies table now carries randomised, analysed and attrition columns, and the risk-of-bias judgement is made against the figure the reader can see.
The document should state what a reader ought to do
The draft review of In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… was read against its registered protocol.
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The respondent notes submission 007 above and does not repeat the ground it covers.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
The review does not say when it will be updated or what would trigger an update
The respondent submits on the draft synthesis addressing In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The respondent states that a synthesis in a field where trials report frequently is out of date from the day it is published, and that a reader has no way to know whether the version in front of them is current.
The respondent proposes a stated trigger for an update as well as a scheduled review date.
The secretariat accepts this submission. A review date alone tells a reader when the document will be looked at, not whether it should already have been.
Every synthesis now records a scheduled review date and a stated update trigger, being the reporting of a trial that would materially change the contributing evidence, and the registered trials capable of triggering an update are listed by name.
Baseline imbalance in a small contributing trial is not remarked on
This submission addresses the draft synthesis on In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… from the standpoint of a reader who will use the summary and not the appendices.
The respondent identifies a small contributing trial with a baseline difference in the outcome variable, and states that the difference is large enough to account for a share of the reported effect.
The respondent asks that the trial be excluded or that the imbalance be addressed.
The respondent has read submission 004 and asks that this submission be considered with it.
The secretariat notes this submission. The imbalance is recorded in the risk-of-bias assessment for that trial, in the domain concerning the randomisation process, where the respondent may not have looked.
No amendment arises. The judgement and the supporting figures are already reported at domain level for that study, and a sensitivity analysis omitting it is reported in the results, which does not change the direction of the estimate.
Point estimates are given without an interval
The respondent read In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… in draft. The point applies to it and to the series generally.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
Risk-of-bias judgements are reported as an overall rating without the domains
The respondent has read In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… and submits on a matter of presentation.
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
Subgroup findings are reported that were not registered in the protocol
The respondent’s comment on the draft review of In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… arises from comparing the included set against the respondent’s own knowledge of the field.
The draft reports differences between subgroups that do not appear in the registered protocol. The respondent states that unregistered subgroup analysis is hypothesis-generating and that the draft presents it in the same form as the pre-specified results.
The respondent proposes that unregistered analyses be removed.
The secretariat accepts this submission in part. The analyses are retained and relabelled rather than removed, because removing an analysis that was conducted leaves no record that it was.
Every subgroup analysis is now labelled as pre-specified or post hoc against the registered protocol, post hoc analyses are reported in a separate subsection without a certainty rating, and the protocol version against which the labelling was made is stated.
Two factual descriptions of the sponsor's programme are inaccurate
The respondent has read In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… in draft and makes a single submission.
The submission is made on behalf of the marketing-authorisation holder and is confined to two matters of fact. The draft describes a trial as terminated where the sponsor closed it at a pre-specified interim analysis, and gives a dose in a unit that does not match the approved labelling.
Supporting documentation, comprising the published trial report and the current summary of product characteristics, accompanied the submission. No view is expressed on the certainty ratings, which the sponsor considers a matter for the assessment committee.
The secretariat accepts this submission. Both points are matters of fact, both were checkable against documents the Institute holds, and both were wrong in the draft.
The trial status now reads as completed at a pre-specified interim analysis, with the analysis identified, and the dose is stated in the unit used in the approved labelling. The submission is identified as an industry submission on its face in accordance with the conflicts policy.
A superseded version should remain reachable from the version that replaced it
The respondent read the draft review of In adolescents with obesity, what is the effect of an incretin receptor agonist on body-mass index, and what is known about growth, bone accrual and… and has confined this submission to a single matter.
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.