Draft revision of the Institute’s certainty assessment… — disposition

§3Disposition and amendments

§3.1Disposition table

Table 1. Every submission with its disposition. Each row links to the submission in full.

§3.2Summary by disposition

Table 2. Counts by disposition, with the meaning of each.

§3.3Resulting amendments

1. Quantitative claims are reproduced without the method that produced them — arising from DRAFT-CERTAINTY-FR/001. Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
2. The framework should say when a rating must not be issued at all — arising from DRAFT-CERTAINTY-FR/002. Where no eligible study exists, no certainty rating is issued, no domain assessment is rendered, and the outcome is recorded as not assessed with the reason stated. The rule is applied retrospectively across the series.
3. The database set omits sources relevant to the compounds in scope — arising from DRAFT-CERTAINTY-FR/003. The standard database set is extended, the conditional sources are named together with the trigger for searching them, and every review states which sources were searched and which were not, with the reason.
4. Point estimates are given without an interval — arising from DRAFT-CERTAINTY-FR/004. Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
5. Sponsor concentration should be a recorded characteristic of a body of evidence — arising from DRAFT-CERTAINTY-FR/005. Sponsor concentration is now recorded for every body of evidence. It is not an automatic downgrade, since in this field a concentrated sponsor set is often the only set that exists, and a rule producing a uniform downgrade would carry no information.
6. It is not clear which provisions bind the assessment committee — arising from DRAFT-CERTAINTY-FR/007. Binding provisions are now stated in a fixed form, are listed together in an annex, and a departure from any of them must be recorded in the document it affects with the reason, while descriptive passages are marked as descriptions of practice.
7. The framework does not say when a concern warrants one level and when it warrants two — arising from DRAFT-CERTAINTY-FR/008. Worked criteria are now published for imprecision, inconsistency and indirectness. For risk of bias and publication bias the judgement is left to the assessors with a requirement to state the reason, because a criterion expressed in advance would be applied…
8. Nothing states how many assessors rate a body of evidence — arising from DRAFT-CERTAINTY-FR/009. Ratings are made independently by two assessors and adjudicated by a third where they differ. The method and the fact of any adjudication are now recorded with the rating.
9. Nothing prevents a conclusion stronger than its certainty rating — arising from DRAFT-CERTAINTY-FR/010. Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, the mapping is published in the methodology document, and a document whose conclusion does not match its rating cannot be ratified.
10. The framework does not say what new evidence would change a rating — arising from DRAFT-CERTAINTY-FR/011. Every rating now carries a statement of what would change it, and the search date against which it was made. A trial meeting the stated description triggers reassessment ahead of the ordinary cycle.
11. A composite outcome is rated as one outcome — arising from DRAFT-CERTAINTY-FR/012. A composite outcome is now reported with its components, each rated separately, and the rating of the composite states which component drove it.
12. The document is unreadable without specialist training — arising from DRAFT-CERTAINTY-FR/013. Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
13. The treatment of imprecision does not work where the event did not occur — arising from DRAFT-CERTAINTY-FR/014. The framework now states the rule for zero-event outcomes: the rating is determined by the total exposure and the frequency that exposure could have detected, and the resulting statement records that the evidence is uninformative rather than reassuring.
14. No information-size criterion is applied before an interval is judged — arising from DRAFT-CERTAINTY-FR/015. An information-size check is now applied and its result recorded. It is not applied mechanically, because a criterion computed on an assumed effect size can itself be the more arbitrary judgement, and the domain note states which consideration governed.

References cited on this page

References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.

1. International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute

Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.