Draft indication set and anchor outcomes — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The front-matter summary describes the finding without stating it
The respondent notes that indication set and anchor outcomes will govern a great many documents, and submits accordingly.
The summary says that an effect was observed and of what certainty. It does not give the estimate. A reader who reads only the summary, which is most readers, comes away with a direction and no magnitude.
The respondent proposes that the summary carry the principal estimate with its interval.
The secretariat accepts this submission.
The front-matter summary now carries the principal estimate with its interval and its certainty rating, so that the three travel together wherever the summary is quoted.
Four dates are given and their relation is not explained
The respondent has read indication set and anchor outcomes and submits on a matter of presentation.
Published, last reviewed, next review and the search date all appear. A reader has to work out which one tells them how current the assessment is, and the answer is not the largest of them.
The respondent proposes a one-line gloss on the metadata block stating which date answers that question.
The secretariat accepts this submission.
The metadata block now carries a one-line gloss stating that the search date determines how current the evidence assessment is, and that a later review date without a later search date means the document was checked and not re-searched.
A printed copy loses the information that makes it checkable
This submission concerns indication set and anchor outcomes and a convention used across the Institute’s output.
The respondent states that printing a document produces pages without the identifier, the version or the retrieval date, so that a printed copy cannot be tied back to what it was printed from.
The respondent proposes that these be printed in a fixed position on every page.
The respondent has read submission 001 above and makes this submission independently of it.
The secretariat accepts this submission. A printed assessment circulates for longer than the browser tab it came from.
The print stylesheet now carries the identifier, version, publication date and retrieval date on every page, expands link destinations into the text, and prints the full reference list rather than a link to it.
A fixed section structure will force empty sections
The respondent submits on indication set and anchor outcomes, on a matter that is not specific to this draft but is visible in it.
The respondent states that a common structure across the series will produce documents with sections that have nothing to report, and that an empty section is read as an oversight.
The respondent proposes that sections be omitted where they do not apply.
Submission 001 raises an adjacent matter. The respondent regards the two as separable and addresses only this one.
The secretariat accepts this submission in part. Sections are retained and their emptiness is made explicit, because an omitted section is indistinguishable from a section that was never considered.
A section with nothing to report now states that fact and the reason in a fixed form, rather than being omitted or left blank, so that a reader can tell the difference between an absence of evidence and an absence of effort.
A printed copy loses the document identifier
The respondent’s comment on indication set and anchor outcomes arises from consulting the series for reference rather than writing for it.
A page printed and circulated arrives without the identifier, the version or the date, so a recipient cannot tell whether it is current or find it again.
The respondent proposes that the print stylesheet carry all three in a footer on every page.
The secretariat accepts this submission.
The print stylesheet now carries the document identifier, the version, the publication date and the address in a footer on every printed page.
The identifier encodes the document type, which is a property that can change
This is a submission on indication set and anchor outcomes.
An identifier containing a type segment will have to change if a document moves between series, and an identifier that changes is not an identifier.
The respondent proposes that the type segment be dropped or that a commitment be published never to change one.
The secretariat accepts the second limb.
The identifier is retained in its present form and a standing commitment is published that no identifier will be reassigned. A document that moves between series keeps the identifier it was issued with, and the type segment is documented as historical rather than descriptive.
The document set should be published in translation
The respondent has read indication set and anchor outcomes in draft and makes one submission.
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
A superseded version should remain reachable from the version that replaced it
The respondent submits on indication set and anchor outcomes. A document series is a set of promises about where things will be found, and this submission concerns one of them.
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
This submission is made by a respondent with a different professional interest in the outcome from the one behind submission 001.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
The registry is published only as pages and not in a form that can be processed
The respondent submits on indication set and anchor outcomes. The point would apply equally to any document in the series.
The respondent, who works with the register programmatically, states that examining the document set as a whole requires extracting fields from rendered pages, and that a structured record would make the registry usable for research about the Institute itself.
The respondent proposes that the registry fields be published in a structured form alongside the pages.
The respondent has read submission 003 with interest and adds one observation the secretariat may find useful.
The secretariat accepts this submission. A body that asks others to publish checkable records should publish one.
The registry fields are now emitted in a structured form alongside the rendered pages, carrying the identifier, series, type, version, dates, certainty rating and status for every document, and the structured record is regenerated with the site so that it cannot drift from the pages.
The same quantity is presented differently in two series
This submission concerns the draft of indication set and anchor outcomes. The respondent’s interest is in retrieval: whether a document can be found again, and cited, after it has been superseded.
The respondent states that a certainty rating is displayed one way in the monograph series and another in the synthesis series, and that a reader moving between them cannot be sure the two mean the same thing.
The respondent proposes a single presentation across all series.
The secretariat accepts this submission. A difference in presentation is read as a difference in meaning.
Certainty ratings are now presented identically across every series, using the same badge, the same wording and the same accompanying formulation, and the presentation is specified in the series structure document rather than left to each series.
The default ordering of a comparison table carries an implication
This is a submission on indication set and anchor outcomes, from a respondent who reads these documents on paper.
The respondent states that whatever column a table is sorted on by default will be read as the column that matters, and that the draft sorts on a score.
The respondent proposes an alphabetical default.
This submission is made in the same spirit as submission 006 and on a different aspect of the draft.
The secretariat accepts this submission. A default is a claim whether or not it is intended as one.
The default ordering is now alphabetical, a standing statement above every comparison table records that the ordering is not a ranking, and the sort state is reset on reload so that a shared link does not carry an ordering the sender chose.
The index shows ratings without the reasons behind them
The respondent read indication set and anchor outcomes in draft and has confined this submission to a single provision.
The respondent states that a list of low ratings without their causes tells a reader that the evidence is weak but not in what respect, and that the respect is what determines whether new research would help.
The respondent proposes that the downgrade domains appear in the index row.
The secretariat accepts this submission. The reason is more actionable than the rating.
Every index row now carries the downgrade domains alongside the rating, and the index can be filtered by domain so that a reader can see, for example, every outcome downgraded for imprecision across the document set.
A section with nothing in it is emitted with a heading and a sentence saying so
The draft of indication set and anchor outcomes was read from the standpoint of someone who will navigate the series rather than write for it.
The fixed structure produces sections carrying a heading and one sentence recording that there is nothing to report. Repeated across a series this is a substantial amount of text saying nothing.
The respondent proposes that an empty section be folded into its parent with a single line, rather than given a heading of its own.
The secretariat accepts this submission.
A section with no content is now folded into its parent as a single recorded line. The fact of the absence is preserved, because an absent section and an absent statement are different, and the heading is not.
Point estimates are given without an interval
Having read the draft of indication set and anchor outcomes, the respondent puts one point to the secretariat.
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
A document is split across parts at a point that separates a claim from its support
This submission addresses indication set and anchor outcomes from the standpoint of a reader arriving from a search engine rather than from the front page.
The evidence is on one page and the certainty assessment that qualifies it on another. A reader who lands on the first from a search engine gets the claim without the qualification.
The respondent proposes that every part carry the certainty rating for the material it presents.
The secretariat accepts the proposal and applies it in the metadata rather than by restructuring.
Every part now carries the certainty rating in its own metadata header and repeats it beside any estimate it reproduces, so that no page presents an estimate without its rating.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.