Draft corrections and versioning policy — submissions
The 19 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
19 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The threshold between a correction and a new version is drawn too high
This is a submission on corrections and versioning policy.
The respondent states that under the draft a change to a certainty rating could be applied as a correction without a version increment, and that a reader holding a citation to the earlier text would not know the rating had changed.
The respondent proposes that any change to a rating or an estimate force a new version.
The secretariat accepts this submission in part. A change to a rating, an estimate or a conclusion now forces a version increment. Typographical and reference corrections continue to be applied as corrections with a log entry.
The versioning threshold is now stated by category: a change to a certainty rating, an effect estimate or a conclusion produces a new version at a new address, and every other change is a logged correction that states the previous wording.
Where a document is generated rather than authored, the reader should be told
The respondent submits on corrections and versioning policy. Governance documents are read by very few people and relied on by everyone who reads anything else the Institute publishes.
The respondent states that some records in the document set are constructed by crossing existing records, that the prose in them is templated, and that nothing on the page distinguishes them from an authored assessment.
The respondent proposes a standing notice on every generated record.
The secretariat accepts this submission. The distinction is material to how much independent judgement a reader should assume.
Every generated record now carries a notice stating that its prose frame is templated and that its included studies, summary of findings and certainty rating are computed from the underlying records, so that the reader can see which parts were authored and which were derived.
No term limit applies to membership of the assessment committee
The respondent read corrections and versioning policy in draft. The point applies to it and to the series generally.
Members are appointed and the document does not say for how long. A body whose membership does not turn over converges on a settled view and stops noticing that it has one.
The respondent proposes a fixed term, renewable once, with the appointment date published against each member.
The secretariat accepts this submission.
Members are appointed for a four-year term renewable once. Appointment and expiry dates are published in the register of members.
The materiality threshold for a declarable holding is set too high
The respondent notes that corrections and versioning policy governs situations that will arise rarely and matter greatly when they do, and submits accordingly.
The respondent states that the threshold in the draft excludes holdings a reader would regard as relevant, and that the cost of declaring a small holding is negligible against the cost of a reader discovering an undeclared one.
The respondent proposes that all holdings be declared regardless of size.
The secretariat accepts this submission in part. The threshold is lowered. Declaration of every holding without threshold is not adopted, because an undifferentiated list obscures the holdings that matter.
The materiality threshold is lowered and is now expressed as a proportion of the declarer's holdings as well as an absolute figure, so that a holding material to the person is declarable whether or not it is large in absolute terms.
Nothing states what happens to the document set if the Institute ceases to operate
The respondent has read corrections and versioning policy in draft and makes one submission.
Documents are cited. A citation to a document that has ceased to exist is worse than no citation, and the policy is silent on what would become of the archive.
The respondent proposes a stated succession arrangement and an open licence that permits the set to be mirrored.
The secretariat accepts this submission.
The governance document now records that the document set is licensed for reuse and mirroring, and names the deposit arrangement under which it would be preserved.
Quantitative claims are reproduced without the method that produced them
This submission concerns corrections and versioning policy and makes one point.
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The respondent has read submission 005 above and makes this submission independently of it.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
Declared interests should appear on the document rather than on a separate page
This submission addresses corrections and versioning policy from the standpoint of an outside body that would have to rely on it.
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The respondent read submission 004 after drafting this one and has not altered it, the two points being distinct.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
A party that is assessed and also sponsors should be disclosed on its own assessment
This submission concerns the draft of corrections and versioning policy. The respondent has served on committees operating comparable regimes and the observation draws on that.
Disclosure on the funding page is necessary and is not sufficient. The reader who most needs to know is the one reading that party’s assessment, and that reader has no reason to visit the funding page.
The respondent proposes that the disclosure appear on the assessment itself, in the same visual weight as the assessment summary.
The secretariat accepts this submission without qualification. It was a real defect.
The disclosure now appears as a numbered section on the dossier of any assessed party that sponsors the site, as well as on the funding page.
An industry submission should be identifiable without reading the declared interest
The respondent has read corrections and versioning policy and submits on a matter of presentation.
The respondent states that a declared interest printed below a submission is read after the submission, and that a reader who has already absorbed the argument reads the declaration as an afterthought.
The respondent proposes that a submission made on behalf of a company be labelled at the head of the submission as well as in the declaration.
The secretariat accepts this submission. The order in which a reader encounters the two facts changes how the second is weighed.
A submission made on behalf of a company is now identified as an industry submission on its face, at the head of the submission and in the disposition table, in addition to the declared interest published in full beneath it.
The policy does not say who may make a submission
The respondent submits on corrections and versioning policy. The point would apply equally to any document in the series.
The document describes the form of a submission and not who is eligible to make one. A reader without a professional affiliation cannot tell whether the process is open to them.
The respondent proposes an explicit statement that anyone may submit, with the declared-interest requirement applying equally to all.
The secretariat accepts this submission. The process is open and the document did not say so.
The policy now states that any person may make a submission, that no affiliation is required, and that the declaration requirement applies identically to every respondent.
A declared interest is published without being tested
This submission concerns corrections and versioning policy and a convention used across the Institute’s output.
A respondent declares their own interest and the declaration is published as given. Nothing states what the Institute does where a declaration appears incomplete on its face.
The respondent proposes that the position be stated, whatever it is.
The secretariat accepts this submission.
The policy now states that declarations are published as made, that the Institute does not investigate them, and that where a declaration appears incomplete the respondent is asked to complete it before publication. That the Institute does not verify is now on the face of every submission page.
The standards applying to advertising placement are not published
The respondent read corrections and versioning policy in draft and has confined this submission to one clause.
The funding page records who sponsors the site. It does not state what a sponsor may and may not do, or what would cause a placement to be refused or withdrawn.
The respondent proposes publication of the placement standards themselves.
The respondent has read submission 011 and asks that this submission be considered with it.
The secretariat accepts the proposal in relation to standards and declines to publish commercial terms.
The placement standards are now published: what a unit may claim, the labelling required, the separation from assessed content, and the grounds on which a placement is refused or withdrawn. Commercial terms are not published and the reason is stated.
The terms on which the documents may be reused are not stated
Having read the draft under consultation, which concerns corrections and versioning policy, the respondent submits as follows.
The respondent states that the site carries no statement of reuse terms and that an assessment which cannot be quoted with confidence will be paraphrased instead, which is worse for accuracy.
The respondent proposes an explicit reuse statement.
The respondent endorses the general approach taken in submission 009 and asks that it be extended to the matter identified here.
The secretariat accepts this submission. An unstated position on reuse is a position that will be guessed at.
The reuse terms are now stated explicitly on the legal page and in the document footer, permitting quotation and reproduction with attribution and the document version, and requiring that any reproduction carry the certainty rating that accompanied the quoted statement.
The position of a contributor who withdraws consent is not stated
Having read the draft of corrections and versioning policy, the respondent puts one point to the secretariat.
The respondent states that consent to be named is obtained at submission and that the policy is silent on what happens if it is later withdrawn, which leaves both the Institute and the contributor without a rule.
The respondent proposes that the position be stated in advance and be the same for everyone.
The secretariat accepts this submission. A rule made at the point of the first withdrawal is a rule made about a particular person.
The policy now states that a contributor may withdraw from the register at any time and that the register entry is removed, while a signed submission already published remains published, because the consultation record is a record of what was said and by whom. The position is stated at the point consent is given.
The search date is not on the face of the document
This is a submission on corrections and versioning policy, from a respondent representing readers rather than practitioners.
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The respondent’s submission overlaps with submission 009 and was prepared without sight of it.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
Disagreement within the assessment committee is not visible
The respondent submits on corrections and versioning policy.
The respondent states that ratified documents present a single position and that a reader cannot tell whether a rating was unanimous or contested.
The respondent proposes that a dissenting view be recorded where a member asks for it to be.
This submission is made in the same spirit as submission 014 and on a different aspect of the draft.
The secretariat accepts this submission. A document that conceals a division presents more certainty about its own judgement than it holds.
A ratification record is now published with every document stating whether the assessment was unanimous, and a member may record a dissenting statement which is published with the document rather than in minutes.
The disclosure requirements exceed those of comparable bodies
The respondent’s comment on corrections and versioning policy is made from the standpoint of a person who might be affected by a decision taken under it.
The submission is made on behalf of a manufacturer. It states that the per-page disclosure requirement is more extensive than that applied by comparable assessment bodies and that the repetition may be read as implying a relationship of greater significance than exists.
The submission does not object to disclosure and objects to its placement and frequency.
The secretariat notes this submission and records that the point about repetition is a fair one.
No amendment arises. Per-page disclosure was consulted on separately and retained, on the ground that a reader arriving at a single page from a search engine does not see a disclosure held elsewhere. The wording has been standardised so that the repeated statement is identical at every occurrence and is not read as a series of different claims. The submission is identified as an industry submission.
No period is stated within which a submission will be answered
The draft of corrections and versioning policy was read for what it obliges the Institute to do rather than for what it says the Institute intends.
A respondent who has made a submission has no basis for knowing whether silence means the submission is being considered or has been lost.
The respondent proposes a stated period, with an acknowledgement on receipt and a substantive response within it.
The secretariat accepts this submission.
Receipt is acknowledged within five working days and a substantive response is published within eight weeks of the window closing. Where that period cannot be met the reason is published in its place.
A party criticised in an assessment should have a right of reply
The respondent submits on corrections and versioning policy, on a matter that is not specific to this draft but is visible in it.
The respondent states that a supplier or sponsor whose documentation is assessed adversely has no route to respond other than a comment period that may not be open.
The respondent proposes a standing right of reply published alongside the assessment.
The secretariat accepts this submission in part. A route to submit a correction of fact is opened at any time and is not dependent on a consultation window. A general right of reply published alongside the assessment is not adopted, because it would give an assessed party the last word on the Institute's own document.
A standing correction route is now published, corrections of fact are assessed and applied at any time with the change recorded in the amendment log, and a submission of substance received outside a window is held and published in the next comment period on that document.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.