Draft terms for the Institute’s calculators — submissions
The 18 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
18 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Independence is asserted and the mechanism securing it is not described
This submission concerns terms for the Institute’s calculators and makes one point.
The document states that sponsorship does not influence assessment. It does not describe what prevents it from doing so, and an assertion without a mechanism is not verifiable.
The respondent proposes that the separation between the party accepting funding and the party making assessments be described in operational terms.
The secretariat accepts this submission.
The governance document now describes the separation in operational terms: who accepts funding, who makes assessments, that the two sets do not overlap, and that assessment staff are not informed of placement decisions before publication.
What the site collects about a reader is not stated in terms a reader can check
The respondent submits on terms for the Institute’s calculators, on a matter that is not specific to this draft but is visible in it.
The respondent states that a body assessing evidence should be checkable in its own conduct, and that a general privacy statement does not permit a reader to verify what is collected.
The respondent proposes that the statement enumerate what is collected and that it be verifiable from the page itself.
The secretariat accepts this submission. An unverifiable statement about data is exactly the kind of claim the Institute declines to accept from others.
The statement now enumerates what is collected and what is not, and is written so that a reader can verify it from the page source, which is the only form of the statement that can be checked without trusting the Institute.
The search date is not on the face of the document
The respondent submits on terms for the Institute’s calculators. Governance documents are read by very few people and relied on by everyone who reads anything else the Institute publishes.
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
A party that is assessed and also sponsors should be disclosed on its own assessment
The respondent submits on terms for the Institute’s calculators.
Disclosure on the funding page is necessary and is not sufficient. The reader who most needs to know is the one reading that party’s assessment, and that reader has no reason to visit the funding page.
The respondent proposes that the disclosure appear on the assessment itself, in the same visual weight as the assessment summary.
This point is adjacent to the one made in submission 003 and the respondent puts it in a form the secretariat can act on.
The secretariat accepts this submission without qualification. It was a real defect.
The disclosure now appears as a numbered section on the dossier of any assessed party that sponsors the site, as well as on the funding page.
The terms on which the documents may be reused are not stated
This is a submission on terms for the Institute’s calculators.
The respondent states that the site carries no statement of reuse terms and that an assessment which cannot be quoted with confidence will be paraphrased instead, which is worse for accuracy.
The respondent proposes an explicit reuse statement.
The secretariat accepts this submission. An unstated position on reuse is a position that will be guessed at.
The reuse terms are now stated explicitly on the legal page and in the document footer, permitting quotation and reproduction with attribution and the document version, and requiring that any reproduction carry the certainty rating that accompanied the quoted statement.
The Institute publishes no telephone number and no messaging channel
The respondent read terms for the Institute’s calculators in draft. The point applies to it and to the series generally.
The respondent states that a body inviting submissions ordinarily provides a channel for questions about the process, and that the written address alone is a high barrier for a first-time respondent.
The respondent asks that a route for procedural questions be provided.
The secretariat notes this submission. The written route is the only route and that is a deliberate constitutional position rather than an omission.
No amendment arises. The Institute holds no telephone number and uses no messaging channel, because a submission made in writing is a document that can be published with its response and an oral exchange is not. Guidance on making a submission has been expanded so that the written route requires no prior enquiry to use.
Funding should not be accepted from any party the Institute assesses
The respondent read terms for the Institute’s calculators in draft and has confined this submission to one clause.
The respondent states that disclosure does not neutralise an incentive, and that accepting funding from an assessed party creates one whether or not it is acted on.
The respondent proposes an outright prohibition on funding from any assessed supplier or manufacturer.
The respondent supports submission 005 so far as it goes and adds the matter set out here.
The secretariat accepts this submission in part. Funding from an assessed party is not accepted for assessment work. A general prohibition covering all activity is not adopted, because it would extend to unrestricted contributions the Institute can accept without any assessment consequence.
The policy now prohibits funding from an assessed party for any activity bearing on that party's assessment, requires that any other relationship be published on the funding page and on every page it could bear on, and records the sums involved rather than the fact of a relationship alone.
No term limit applies to membership of the assessment committee
This is a submission on terms for the Institute’s calculators, from a respondent representing readers rather than practitioners.
Members are appointed and the document does not say for how long. A body whose membership does not turn over converges on a settled view and stops noticing that it has one.
The respondent proposes a fixed term, renewable once, with the appointment date published against each member.
The respondent read submission 006 after drafting this one and has not altered it, the two points being distinct.
The secretariat accepts this submission.
Members are appointed for a four-year term renewable once. Appointment and expiry dates are published in the register of members.
A reader who acted on a document is not told when it is corrected
The draft of terms for the Institute’s calculators was read for what it obliges the Institute to do rather than for what it says the Institute intends.
Corrections are published in a log. Nobody who read the original page is notified, and the corrected page carries the correction in a form that a reader arriving fresh will read past.
The respondent proposes that a corrected document carry a visible notice at the point of the correction for a stated period, in addition to the log entry.
The secretariat accepts this submission. A correction filed where only the diligent will find it is a correction the reader who needs it will miss.
A corrected passage now carries an inline notice for twelve months from the date of correction, naming the log entry, in addition to the entry itself.
A four-week window is too short for a respondent without institutional support
This submission addresses terms for the Institute’s calculators from the standpoint of an outside body that would have to rely on it.
An organisation with staff can answer a consultation in four weeks. An individual reading in their own time, particularly one who needs the document explained before they can respond to it, generally cannot.
The respondent proposes a minimum of eight weeks for any consultation on a document of general application.
The secretariat accepts the principle and adopts a shorter minimum than proposed.
The minimum window is now six weeks for a document of general application and four for a document-specific draft. A respondent who asks for more time before a window closes is given it, and that is now stated in the policy.
A submission the Institute declines to publish leaves no trace
The respondent has read terms for the Institute’s calculators in draft and makes a single submission.
The policy permits a submission to be declined where it is abusive or names an individual. Nothing records that a submission was received and declined, so the published set cannot be distinguished from the received set.
The respondent proposes a count of declined submissions with the ground for each.
The respondent notes that submission 004 has already been made and confines this submission to a matter not covered by it.
The secretariat notes this submission and has adopted part of it.
The number of submissions declined and the ground for each is now published on the disposition page of the consultation concerned. The submissions themselves are not published, for the reasons that caused them to be declined.
References should carry a persistent identifier for every cited source
The respondent’s comment on terms for the Institute’s calculators is made from the standpoint of a person who might be affected by a decision taken under it.
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The respondent endorses the general approach taken in submission 005 and asks that it be extended to the matter identified here.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
The same concept is given three different names in one document
The respondent notes that terms for the Institute’s calculators governs situations that will arise rarely and matter greatly when they do, and submits accordingly.
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
Assessment committee meetings are held in private
This submission concerns terms for the Institute’s calculators and a convention used across the Institute’s output.
Comparable bodies admit observers to at least part of a meeting. The document does not say whether the Institute has considered doing so.
The respondent proposes that part of each meeting be open, or that the reason for holding meetings in private be stated.
The secretariat notes this submission and states the position rather than changing it.
Meetings are held in private and the reason is now stated: the committee assesses draft ratings that will change before publication, and an observed deliberation on a draft rating tends to produce a published deliberation rather than a candid one. The minutes and the dissenting positions are published.
Whether external reviewers are paid is not stated
The respondent has read terms for the Institute’s calculators in draft and makes one submission.
The respondent states that payment or its absence bears on who is able to review and on what the Institute may expect of them, and that the policy does not address it.
The respondent proposes that the position be stated and that any payment be disclosed at the level of the individual document.
The secretariat accepts this submission. Payment is an interest and the policy treats interests as facts to be published.
The policy now states whether external review is remunerated, records any payment made in respect of a document in that document's front matter, and treats a payment received from the Institute as a declarable interest in the same way as any other.
There is no route to challenge a disposition
Having read the draft of terms for the Institute’s calculators, the respondent puts one point to the secretariat.
The respondent states that a submission not accepted receives a reason and nothing further, and that a respondent who considers the reason mistaken has nowhere to go.
The respondent proposes a review of disposition by a person who did not make it.
The respondent has read submission 010 with interest and adds one observation the secretariat may find useful.
The secretariat accepts this submission. A disposition that cannot be challenged is a decision without a check.
A disposition may now be referred for review to a member of the assessment committee who did not author the response, once per submission, with the referral and its outcome published alongside the original disposition whether or not it changes.
The secretariat answers submissions and is not itself named
This submission concerns the draft of terms for the Institute’s calculators. The respondent has served on committees operating comparable regimes and the observation draws on that.
The respondent states that responses are attributed to the secretariat as a body, and that a reader cannot tell who wrote a response or whether that person had a relevant interest.
The respondent proposes that responses be signed individually.
This submission is made in the same spirit as submission 004 and on a different aspect of the draft.
The secretariat accepts this submission in part. Responses are attributed to the named members of the secretariat responsible for the document rather than to individuals per response, and the interests of those members are published.
Every comment period now names the secretariat members responsible for it, with their declared interests, in the consultation front matter, so that the responses can be read against the interests of the people who wrote them.
The disclosure requirements exceed those of comparable bodies
Having read the draft under consultation, which concerns terms for the Institute’s calculators, the respondent submits as follows.
The submission is made on behalf of a manufacturer. It states that the per-page disclosure requirement is more extensive than that applied by comparable assessment bodies and that the repetition may be read as implying a relationship of greater significance than exists.
The submission does not object to disclosure and objects to its placement and frequency.
The secretariat notes this submission and records that the point about repetition is a fair one.
No amendment arises. Per-page disclosure was consulted on separately and retained, on the ground that a reader arriving at a single page from a search engine does not see a disclosure held elsewhere. The wording has been standardised so that the repeated statement is identical at every occurrence and is not read as a series of different claims. The submission is identified as an industry submission.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.