Draft analytical review: surveys of unregulated peptide supply — submissions
The 14 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
14 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The document should not describe uses outside the approved indication
The respondent has read the draft synthesis on What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? and makes one submission about the analytical evidence it pools.
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
The draft moves between purity and content as though they were one quantity
This submission concerns the draft synthesis on What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers?. The respondent works with determinations of this kind and the observation arises from that.
The respondent identifies passages in which a chromatographic purity figure and a content figure are discussed as though they answered the same question.
The respondent states that purity is a relative area within what the detector saw and content is a mass fraction, that neither bounds the other, and that a review of this evidence is the last place the two should be run together.
The respondent proposes that the two be reported in separate sections with a statement of what each does and does not establish.
The secretariat accepts this submission. The conflation is the error the Institute most often records elsewhere, and it appeared here.
Purity and content are now reported in separate sections, each opening with a statement of what the quantity establishes, and no summary sentence in the review carries both without saying that they answer different questions.
Independent results are not paired with the certificate figure for the same lot
This submission addresses the draft synthesis on What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? from the standpoint of a reader who will act on the summary figure.
The draft reports independent determinations and certificate figures as two distributions rather than as pairs.
The respondent states that the question a reader has is how far a certificate figure departs from an independent determination of the same material, that this is a paired question, and that two unpaired distributions cannot answer it.
The respondent proposes that the analysis be restricted to lots for which both figures are held and reported as a paired difference.
The secretariat accepts this submission. The paired comparison is the one the review question asks for and the draft reported the marginal distributions instead.
The primary analysis is now the paired difference on lots holding both figures, the number of such lots is stated, and the unpaired results are reported separately as descriptive and are not used to support the answer to the review question.
Quantitative claims are reproduced without the method that produced them
The respondent read the draft review of What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? and has confined this submission to a single matter of measurement.
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
The chain of custody between purchase and analysis is not described
This submission concerns What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? and makes one point.
The contributing surveys describe what was analysed and do not describe how the material travelled between the seller and the instrument.
The respondent states that storage temperature and time in transit affect several of the attributes reported, so a difference between surveys may be a difference in handling rather than in supply.
The respondent proposes that transit and storage conditions be recorded for every contributing survey.
The secretariat notes this submission and records that it asked each contributing author for the information the respondent identifies.
No amendment beyond disclosure arises, because the information is not held for most of the contributing surveys. The synthesis now states, survey by survey, whether handling conditions were reported, and the limitation is carried into the certainty rating rather than left in the discussion.
Contributing results are reported in units that were not converted before pooling
Having read the draft synthesis on What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers?, the respondent puts one point about the underlying measurements to the committee.
The contributing surveys report the quantity in more than one unit, and the draft tabulates the figures as published.
The respondent states that the table therefore places figures of different magnitude in one column, and that a reader scanning it will read a unit difference as a result difference.
The respondent proposes conversion to a single unit throughout, with the figure as published retained in a second column.
The secretariat accepts this submission. A table that mixes units invites exactly the misreading the respondent describes.
Every result is now reported in a single unit stated at the head of the table, the figure as published is retained beside it, and the conversion applied is stated so that either figure can be checked against the source.
Whether the analyst knew the source of each sample is not reported
This is a submission on the draft review of What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers?, made from the standpoint of a laboratory that would have to reproduce the determinations.
The contributing surveys do not state whether the laboratory knew which source each sample came from.
The respondent states that integration decisions and the treatment of borderline results involve judgement, and that knowledge of the source is capable of influencing that judgement without any intention that it should.
The respondent proposes that blinding be recorded where it is reported and requested from contributing authors where it is not.
The secretariat notes this submission and has recorded blinding where the contributing authors reported it, which is a minority of the set.
No amendment to the estimates arises. Blinding is now recorded survey by survey, the absence of the information is stated rather than assumed either way, and the point is carried into the limitations.
Contributing surveys anonymise their sources to different degrees
The respondent submits on the draft review of What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers?. This submission concerns how the contributing results were obtained rather than what they show.
Some contributing surveys name the sources they bought from, some describe them by category, and some do not describe them at all.
The respondent states that a synthesis drawing on all three cannot report by source, and that a reader may take the category descriptions as comparable when they were not defined in common.
The respondent proposes that the level of source description be recorded for each survey and that no analysis be reported by source.
The respondent notes submission 001 above and does not repeat the ground it covers.
The secretariat notes this submission and records that the practice the respondent describes is a property of the contributing literature rather than of the draft.
No analysis by source is reported. The level of source description is now recorded survey by survey, and the review states that it draws no conclusion about any individual source and that its findings are properties of the sampled literature.
What the contributing methods could not have detected is not stated
The respondent’s comment on the draft review of What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? arises from comparing the pooled figures against the reports they were taken from.
The draft reports what the contributing surveys found and does not report what their methods were incapable of finding.
The respondent states that a reverse-phase method is blind to aggregation, that an ultraviolet detector does not see a species without a chromophore, and that a reader takes an unqualified negative as an absence.
The respondent proposes that every reported absence carry the detection scope of the method that produced it.
The secretariat accepts the disclosure and declines to reclassify the findings, which were correctly reported for the methods used.
Every reported absence now carries the detection scope of the method behind it, and the review states in its abstract that an absence in this evidence base is an absence within a stated scope rather than an absence in the material.
Results below the limit of quantitation are treated as zero
The respondent has read What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? and submits on a matter of presentation.
Several contributing results fall below the quantitation limit of the method used, and the draft carries them into the summary as zero.
The respondent states that a result below the quantitation limit means the quantity was not measured at that concentration, not that it was absent, and that treating it as zero biases every summary statistic downward.
The respondent proposes that such results be reported as below the stated limit, with the limit given, and handled by a stated convention in any pooled estimate.
The respondent’s submission overlaps with submission 003 and was prepared without sight of it.
The secretariat accepts this submission. Substituting zero for an unmeasured quantity is a silent assumption and the draft made it without stating it.
Results below the quantitation limit are now reported as such with the limit stated, the convention used in pooled estimates is stated in the methods, and a sensitivity analysis under an alternative convention is reported alongside.
Content results are reported without the standard they were assigned against
The draft review of What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? was read against the certificates and reports it draws on.
The draft reports content figures drawn from the contributing surveys and does not report what each was assigned against.
The respondent states that a content figure is meaningful only relative to a standard of known assigned content, that the contributing surveys used different standards and in some cases none, and that the figures are therefore not on one scale.
The respondent proposes that the standard be recorded for every content result and that results without one be reported separately.
The secretariat accepts this submission. Content without a stated standard is a number without a scale, and the draft placed such numbers beside figures that had one.
Every content result now records the standard against which it was assigned, results with no stated standard are reported in a separate table and excluded from pooled estimates, and the distinction is stated in the summary of findings.
The search was limited by publication date and the limit is not justified
This submission concerns What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? and a convention used across the Institute’s output.
The search strategy carries a start date some years before the review. No reason is given. If the limit reflects the date of first synthesis of the compound it is defensible and should be said; if it reflects convenience it is a source of bias.
The respondent proposes that every date limit carry its justification in the search-strategy section.
The secretariat accepts the requirement to justify and declines to remove limits that are justified.
Every date limit in the series now carries its justification in the search-strategy section. Limits that could not be justified on re-examination have been removed and the affected searches re-run.
A superseded version should remain reachable from the version that replaced it
The respondent notes that the review question — What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers? — is answerable only if the contributing determinations measured the same quantity, and submits with that in view.
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
One laboratory contributes most of the results in the pooled set
The respondent submits on What do published analytical surveys report about the identity, purity and content of peptides obtained from unregulated suppliers?.
The respondent counts the contributing results by the laboratory that produced them and finds that a majority come from one.
The respondent states that the pooled figure is therefore substantially a description of one laboratory rather than of the material, and that any systematic offset in that laboratory propagates into the finding undetected.
The respondent proposes that the contribution by laboratory be tabulated and that a leave-one-laboratory-out analysis be reported.
The respondent read submission 002 after drafting this one and has not altered it, the two points being distinct.
The secretariat accepts this submission. Concentration of the evidence base in one producer is a property of the evidence and the draft did not report it.
The contribution of each laboratory is now tabulated, a leave-one-laboratory-out analysis is reported for every pooled estimate, and the concentration is stated as a reason in the certainty rating where it bears on one.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.