Draft scoping review: change in lean mass during… — submissions
The 7 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
7 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Patient-reported outcomes are collected by the trials and not reported by the review
The respondent has read the draft synthesis on What proportion of the mass lost during treatment with an incretin receptor agonist is lean rather than fat mass, and does it differ from that lost… and makes one submission.
The respondent states that several contributing trials measured quality of life and function and that the review reports neither, having selected outcomes on the basis of what could be pooled.
The respondent proposes that these outcomes be reported narratively where they cannot be pooled.
The secretariat accepts this submission in part. The outcomes are reported narratively. The proposal to pool them is declined because the instruments used are not comparable and pooling would produce a figure with no interpretation.
Patient-reported outcomes measured by any contributing trial are now reported narratively by instrument, with the instrument named and its minimum important difference stated where one is published, and the absence of a pooled estimate explained.
Point estimates are given without an interval
This submission addresses the draft synthesis on What proportion of the mass lost during treatment with an incretin receptor agonist is lean rather than fat mass, and does it differ from that lost… from the standpoint of a reader who will use the summary and not the appendices.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
Intention-to-treat and efficacy-estimand results are combined without distinction
Having read the draft synthesis on What proportion of the mass lost during treatment with an incretin receptor agonist is lean rather than fat mass, and does it differ from that lost…, the respondent puts one point to the committee.
Several contributing trials report both a treatment-policy result and an efficacy result that censors at discontinuation. The draft draws from whichever is reported first in each paper.
The respondent, a trial statistician, states that the two answer different questions, that the difference is substantial where discontinuation is common, and that a synthesis should choose one and say which.
The secretariat accepts this submission. Mixing estimands within a single pooled estimate is a defect of the synthesis rather than of the trials.
The treatment-policy estimand is used throughout as the primary analysis, the efficacy estimand is reported as a secondary analysis where available, and the estimand used is stated in every row of the summary of findings.
Registered trials that never reported are not counted anywhere in the review
The respondent read the draft review of What proportion of the mass lost during treatment with an incretin receptor agonist is lean rather than fat mass, and does it differ from that lost… and has confined this submission to a single matter.
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
The search was limited by publication date and the limit is not justified
The draft review of What proportion of the mass lost during treatment with an incretin receptor agonist is lean rather than fat mass, and does it differ from that lost… was read against its registered protocol.
The search strategy carries a start date some years before the review. No reason is given. If the limit reflects the date of first synthesis of the compound it is defensible and should be said; if it reflects convenience it is a source of bias.
The respondent proposes that every date limit carry its justification in the search-strategy section.
The secretariat accepts the requirement to justify and declines to remove limits that are justified.
Every date limit in the series now carries its justification in the search-strategy section. Limits that could not be justified on re-examination have been removed and the affected searches re-run.
The timepoint at which each outcome was extracted was chosen after the data were seen
The respondent submits on the draft synthesis addressing What proportion of the mass lost during treatment with an incretin receptor agonist is lean rather than fat mass, and does it differ from that lost…. This submission concerns the methods rather than the conclusion, on the view that a conclusion is only as good as the search that produced it.
The methods state that outcomes were extracted at the latest available timepoint. Several contributing trials report the same outcome at three timepoints, and taking the latest is a choice that can be made in the knowledge of what each shows.
The respondent proposes that the extraction timepoint be prespecified in the protocol, and that where it was not, the review report the estimate at every reported timepoint so that the choice is visible.
The secretariat accepts this submission. An extraction rule applied after the data are visible is a degree of freedom and should be recorded as one.
The extraction timepoint is now prespecified in the protocol for new reviews. Where a review predates the requirement, estimates are reported at every timepoint the contributing trials report.
Results at materially different follow-up durations are pooled
The respondent’s comment on the draft review of What proportion of the mass lost during treatment with an incretin receptor agonist is lean rather than fat mass, and does it differ from that lost… arises from comparing the included set against the respondent’s own knowledge of the field.
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.