Ansi Z540 [PDF]

Zitiervorschau

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/305767261

Calibration in Regulated Industries: Federal Agency Use of ISO 17025 and ANSI Z540.3 Conference Paper · July 2016

CITATIONS

READS

0

1,929

1 author: Paul Reese AbbVie 11 PUBLICATIONS   23 CITATIONS    SEE PROFILE

All content following this page was uploaded by Paul Reese on 01 August 2016. The user has requested enhancement of the downloaded file.

Calibration in Regulated Industries: Federal Agency use of ANSI Z540.3 and ISO 170251 Speaker/Author: Paul Reese Baxter Healthcare Corporation 25212 West Illinois Route 120 Mail Stop WG2-2S Round Lake, IL 60073 Phone: (224) 270-4547 Fax: (224) 270-2491 e-mail: [email protected] Abstract ANSI/NCSL Z540.3-2006 and ISO/IEC 17025:2005 are voluntary consensus standards which prescribe requirements for the calibration of measuring and test equipment and for the technical competency of the performing laboratories. Many agencies in the U.S. which are part of, or regulated by, the Federal Government are required to use instruments which have been calibrated in accordance with one or both of these standards. The National Technology and Transfer Advancement Act (NTTAA) of 1995 encourages all federal agencies to use technical standards that are developed by consensus standards bodies, in lieu of “government-unique” standards. ISO 17025 and ANSI Z540.3 have evolved over a half-century of metrological advancement, drawing upon expertise in the public and private sector. They are now supported by a mature infrastructure of third party assessment and accreditation that facilitates mutual recognition and global trade, ensuring calibrations are accepted worldwide. However, some federal agencies and regulatory bodies in the U.S. do not yet recognize these standards. Calibrations are commonly performed which may not conform to these requirements. This paper discusses the history of risk mitigation techniques applied to products and processes when declaring in-or-out of tolerance conditions. Particular focus is given to the Food and Drug Administration’s (FDA) regulation of calibration requirements in the Quality System Regulation (QSR) found in Title 21 Part 820 of the Code of Federal Regulations (CFR). Currently, a paucity of official guidance exists with respect to what constitutes an acceptable calibration program in medical device and pharmaceutical industries. Ambiguities may persist due to lack of recognition of consensus standards such as ISO 17025 and Z540.3 as guidance documents. Fundamental requirements such as traceability, measurement uncertainty, measurement decision-rules, as well as basic metrological definitions are somewhat ill-defined in the CFR. The objective of this paper is to provide relevant background information and to encourage constructive dialogue between government agencies, standards writing committees, industry partners, and third party assessment/accreditation bodies. Cooperation of this type is consistent with public law and White House policy objectives. Ultimately, such dialogue may foster recognition of these voluntary consensus standards as guidance documents for calibration in regulated industries, resulting in regulatory efficiencies, improved quality, and reduced risk to consumers or patients. Note: Throughout this paper, “emphasis” is denoted via red-highlight and/or underline for quoted references. Where applied, emphasis is the author’s and not that of the original source document. 1

th

Revised July 31, 2016. Originally submitted May 24 , and published in the July 2016 NCSLI Proceedings.

2016 NCSL International Workshop & Symposium

1. Background Many federal agencies, in diverse roles, have established minimum quality standards and regulatory requirements governing the calibration of test and measurement equipment. These agencies differ from commercial non-regulated industries in several respects. Requirements of most federal agencies are codified into law and non-compliance with such rules and regulations can result in legal action, particularly where regulatory agencies are concerned. For example, medical device and pharmaceutical manufacturers, regulated by the FDA, must comply with Title 21 of the Code of Federal Regulations (CFR), including the calibration requirements of section §820.72 (see Table 2) and other applicable sections [1]. Non-compliance can subject a manufacturer to warning letters, recalls, injunctions, seizures, and/or debarment. Federal agencies exist where legislation has been enacted to create them and where the criticality, importance, or cost of such a program is deemed too great to relegate the responsibility to private industry. USA.gov currently lists more than 600 different federal agencies, each charged with matters related to the general public well-being, with some establishing their own quality and calibration standards. Examples of Federal Agencies include:         

National Aeronautics and Space Administration (NASA) Food and Drug Administration (FDA) Environmental Protection Agency (EPA) Department of Defense (DOD) Department of Energy (DOE) Nuclear Regulatory Commission (NRC) Bureau of Alcohol Tobacco Firearms and Explosives (ATF) National Security Agency (NSA) Department of Transportation (DOT)

It has been the aspiration of government and industry alike to standardize on quality requirements for more than half a century. The largest standardization effort in quality history began with the wide spread adoption of the ISO 9000 series in 1987, the documentary roots of which began in the 1950’s. See Table 1 and Figures 1 and 2. Standardization has also been the goal with respect to the calibration requirements of test and measuring equipment used to manufacture goods or monitor critical processes in a broad range of industries. Groups developing voluntary consensus standards for calibration requirements have worked diligently to aid in this endeavor. A cursory review of Table 1 will reveal a litany of documents, developed over 65 years, governing calibration in both the public and private sector. Figure 1 provides a simplified graphical evolution of these historical standards and guidelines, with a particular focus on the FDA-regulated medical device and healthcare industry. The calibration documents in Table 1 can generally be divided into two categories:  

Government-unique standards Voluntary consensus standards

(e.g. FDA, DoE, DoD, NASA, NRC, etc.) (e.g. ISO, ASTM, ANSI, ASME, IEEE, etc.)

Full document titles for these standards are provided in Table 1 such that publications may henceforth be referenced simply by their document numbers for convenience.

2016 NCSL International Workshop & Symposium

Table 1A. Chronology of calibration system standards and guidelines YEAR 1950 1955 1958 1958 1959 1960 1960 1962 1962 1962 1962 1962 1962 1963 1964 1965 1965 1969 1969 1969 1969 1969 1970 1970 1971 1972 1973 1973

DOCUMENT NUMBER MIL-STD-120 Tech Memo 63-106 AFR 74-2 SLIM SECNAV 4355.11 BU-520 MIL-C-45662 (Ord) MIL-C-45662A SECNAV 4355.11A MIL-C-55163 NASA NPC 200-2 AR 750-25 T.O. 33-1-14 FDA 21 CFR 133.4* MIL-HDBK-52 NHB 5300.2 MIL-C-24133 RDT F3-2T NBS SP 300-1 MIL-C-24133A MIL-M-38793

SECNAV 4355.11B AEC 10 CFR 50 Appendix B, XII NATO AQAP-6 NCSL RP-4 NATO AQAP-7 SECNAV 4355.11C DEFSTAN 05-26/1

1973

DEFSTAN 05-27/1

1973 1975

NE F3-2T EIA QB4-1975

1976 1976 1976 1976

IEEE STD 498-1975 (ANSI N45.2.16) ASTM E548-76 DEFSTAN 05-26/2 NATO AQAP-6 (2nd Ed) DEFSTAN 05-32/1

1978

DEFSTAN 05-27/2

1978 1978 1978 1978

ISO Guide 25-1978 (1st Ed) ISO Guide 24:1978 NATO AQAP-7 FDA 21 CFR 820.61*

1978

FDA GMP Self Inspection Program*

1979

FDA GMP Quality Audit Manual*

1979 1979 1980 1980

BS 5781-1:1979 NCSL RP-2 MIL-STD-45662 AS 2415-1980

1980

IEEE-498-1980

1981 1981 1982 1982

NMI 5330.9A BS 5781-2:1981 ISO/IEC Guide 25:1982 (2nd Ed) FDA GMP Workshop Manual*

1983

BS 6460-1:1983

1983 1983

DEFSTAN 05-55/2 UKAS/NAMAS M10

1975

TITLE OF DOCUMENT Military Standard, Gage Inspection [Chng#1 added in 1963; MIL-STD-120 was cancelled in 1996] Navy: Factors Affecting Measurement Reliability [Jerry Hayes, Naval Ordnance Lab – Oct 24, 1955] Air Force Metrology and Calibration (AFMETCAL) Program [Established Project “Test-Shop” for PMELs] Navy: Standards Laboratory Information Manual [BuWepsRep – Pomona, CA, Feb 1958, Revised July 1959] Navy Metrology and Calibration (METCAL) Program [March 1959; Also NavOrd 4355.28 & BuWeps 4355.5] USAF Bulletin No. 520: Calibration and Certification of Measuring and Testing Equipment [May 17, 1960] Military Specification: Calibration of Standards Military Specification: Calibration System Requirements [Renamed; Supersedes MIL-C-45662 (1960)] Navy Metrology and Calibration (METCAL) Program [January 16, 1962] Calibration of Test and Measuring Equipment [Army: Cancelled in 1967] Quality Program Provisions for Space System Contractors [Chap 9 – Inspection, Measuring, & Test Equipment] Army Test, Measurement, and Diagnostic Equipment Calibration and Repair Support Program Air Force: Repair, Calibration and Certification of Precision Measurement Equipment [October 1962] Code of Federal Regulations: cGMP – Equipment, [Note: No equipment calibration requirements] Military Handbook – Evaluation of a Contractor’s Calibration System [for use with MIL-C-45662A] NASA Handbook – Apollo Metrology Requirements Manual Preparation of Calibration Procedures [Navy – Space and Naval Warfare Systems Command] Calibration System Requirements [Atomic Energy Commission, ORNL Feb 1969; See also NE F3-2T - 1973] Precision Measurement and Calibration – Statistical Concepts and Procedures [NIST] Preparation of Calibration Procedures [Navy; cancelled in 1970; superseded by MIL-M-38793] Performance Specification: Manuals, Technical: Calibration Procedures, Preparation of Navy Metrology and Calibration (METCAL) Program [July 16, 1969] Code of Federal Regulations: AEC – Control of Measuring and Test Equipment [Basic Requirement 12] NATO Measurement and Calibration System Requirements for Industry [December 1, 1970] Recommended Practice: Calibration System Specification [Withdrawn: Replaced by ANSI/NCSL Z540.1-1994] Guide for the Evaluation of a Contractor’s Calibration System for Compliance with AQAP-6 Department of the Navy Metrology and Calibration (METCAL) Program [November 7, 1973] Measurement and Calibration System Requirement for Industry Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with DEFSTAN 05-26 Calibration Program Requirements [DoE; Supersedes RDT F3-2T – 1969. NE F3-2T cancelled July 1996] Calibration System Requirements [Electronic Industries Alliance] IEEE Standard Supplementary Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations [Revised in 1980] Recommended Practice for Generic Criteria for Use in the Evaluation of Testing and Inspection Agencies. Measurement and Calibration System Requirement for Industry NATO Measurement and Calibration Systems Requirements for Industry [July 1976; Cancelled March 7, 1995] Quality Control Requirements for Test House and Laboratory Organizations [Withdrawn 1982: See BS 6460-1] Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with DEFSTAN 05-26 Guidelines for Assessing the Technical Competence of Testing Laboratories [October 1, 1978] Guidelines for the Acceptance of Testing and Inspection Agencies by Certification Bodies Guide for the Evaluation of a Contractor’s Calibration System for Compliance with AQAP-6 Code of Federal Regulations: cGMP - Measurement Equipment [Supersedes 21 CFR 133.4, 1963] Device Good Manufacturing Practices – An Industry Self-Inspection Program [pp 30 – 34, 820.61 Measurement Equipment] Device Good Manufacturing Practices – A Quality Audit Program for Industry [pp 88-91, 820.61 Measurement Equipment] Measurement and Calibration Systems. Specification for System Requirements Recommended Practice: Evaluation of Measurement Control Systems and Calibration Laboratories Military Standard: Calibration System Requirements [Superseded by MIL-STD-45662A in 1988] Calibration System Requirements [Australia – Superseded by AS 3912.1-1993] IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations [Revised in 1985] NASA Management Instruction, Metrology and Calibration Quality Assurance Requirements for Measuring Equipment. Guide to the use of BS 5781-1 General Requirements for the Technical Competence of Testing Laboratories Device Good Manufacturing Practices – A Workshop Manual [Chap. 4 Buildings, Equipment & Calibration] Accreditation of Testing Laboratories. Specification of the General Requirements for the Technical Competence of Testing Laboratories [Withdrawn 23 Nov 2007. See ISO 17025:2005] Ministry of Defence Calibration Laboratories Operation and Management General Requirements for the Competence of Testing Laboratories [See ISO Guide 25]

2016 NCSL International Workshop & Symposium

Table 1B. Chronology of calibration system standards and guidelines (continued) YEAR DOCUMENT NUMBER 1984 MIL-HDBK-52A 1984 ASTM E548-84 1984

TB 750-25

1984

FDA 85-4179 (3rd Ed)*

1985

IEEE-498-1985

1986 1986 1987 1987 1988 1988 1988 1989 1989 1989 1989 1989 1989 1990 1990 1990 1991 1991 1991

MIL-STD-1839 NCSL RP-6 (1st Ed)** FDA 87-4179 (4th Ed)* ANSI/ASQC M1-1987 MIL-STD-45662A MIL-STD-1839A DEFSTAN 05-55/3 EN 45001:1989 EN 45002:1989 BS 7501:1989 BS 7502:1989 MIL-HDBK-52B OIML G7 ISO/IEC Guide 25:1990 (3rd Ed) DOE 4330.4A NAVAIR 17-35TR-4 FDA 91-4179 (5th Ed)* ASTM E548-91 NHB 5330.9(1A)

1991

HB 18.25-1991

1991

NCSL RP-11

1991

IEEE-498-1990

1992 1992

ANSI/ASQC M2-1992 TSE – TS 10006

1992

BS 5781-1:1992

1992

ISO 10012-1:1992

1993

ISO/IEC Guide 58:1993

1993 1993

ASTM E548-93 MIL-PRF-38793A

1993

DOE-STD-1054-93

1993

HB 18.58-1993

1994

ILAC-G5:1994

1994

BS EN 30012-1:1994

1994 1994 1994

NASA RP-1342 ASTM E548-94 NMI 5330.9B

1994

DOE 4330.4B

1994 1995 1995 1995 1995 1995 1995 1995 1996 1996

ANSI/NCSL Z540.1-1994 NCSL Z540.1 Handbook ASTM E548-94(E1) MIL-STD-1839B MIL-STD-45662A (Cancelled) MIL-HDBK-52B (Cancelled) NATO AQAP-6 (Cancelled) IEEE-498-1990 (Cancelled) MIL-STD-120 (Cancelled) NE F3-2T (Cancelled)

TITLE OF DOCUMENT Military Handbook – Evaluation of a Contractor’s Calibration System [for use with MIL-STD-45662] Standard Practice General Criteria for Use in the Evaluation of Testing and Inspection Agencies Maintenance of Supplies and Equipment Army Test, Measurement, and Diagnostic Equipment (TMDE) Calibration and Repair Support (S&RS) Program Medical Device Good Manufacturing Practices Manual [Chap. 4 Equipment and Calibration] Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations [Supersedes IEEE-498-1980; Replaced by IEEE-498-1990] Military Standard – Calibration and Measurement Requirements Recommended Practice – Medical Device and Pharmaceutical Industry Calibration Control System Medical Device Good Manufacturing Practices (GMP) Manual [Chap. 5, Equipment and Calibration] American National Standard for Calibration Systems [Reaffirmed in 1996 as ANSI/ASQC M1-1996] Military Standard: Calibration System Requirements [Supersedes MIL-STD-45662 (1980); Cancelled in 1995] Military Standard – Calibration and Measurement Requirements Ministry of Defence Calibration Laboratories Operation and Management General Criteria for the Operation of Testing Laboratories [See ISO Guide 25] General Criteria for the Assessment of Testing Laboratories [See ISO Guide 25] General Criteria for the Operation of Testing Laboratories [See ISO Guide 25] General Criteria for the Assessment of Testing Laboratories [See ISO Guide 25] Military Handbook – Evaluation of a Contractor’s Calibration System [for use with MIL-STD-45662A] Guide to Calibration [International Organization for Legal Metrology] General Requirements for the Competence of Calibration and Testing Laboratories [See ISO 17025:1999] DoE Maintenance Management Program [Section 12 – Control & Calibration of Measuring & Test Equipment] Requirements for Preparation of Instrument Calibration Procedures Medical Device Good Manufacturing Practices (GMP) Manual [Chap. 5, Equipment and Calibration] Standard Guide for General Criteria Used for Evaluating Laboratory Competence [See ISO Guide-25] NASA Handbook: Metrology, Calibration & Measurement Processes Guidelines [see also NMI 5330.9] Guidelines for third-party certification and accreditation – Guide 25- General Requirements for the Competence of Calibration and Testing Laboratories [Australia: Superseded by ISO 17025:1999] Recommended Practice: Reports and Certificates of Calibration [Replaced by ANSI/NCSL Z540.1-1994] Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations [Supersedes IEEE-498-1985. Withdrawn 7/14/1995] American National Standard for the Quality Control of Measurements Military Quality Assurance Systems – Measurement and Calibration System Requirements for Industry [Turkey] Quality Assurance Requirements for Measuring Equipment – Metrological Confirmation System for Measuring Equipment [See ISO 10012-1:1992] Quality Assurance Requirements for Measuring Equipment Part 1 – Metrological confirmation system for measuring equipment [AS 3912.1-1993 in Australia, which superseded AS 2415-1980] Calibration and Testing Laboratory Accreditation Systems — General Requirements for Operation and Recognition [Superseded by ISO/IEC 17011:2004] Standard Guide for General Criteria Used for Evaluating Laboratory Competence [See ISO Guide-25] Performance Specification – Technical Manuals: Calibration Procedures – Preparation Guideline to Good Practices for Control and Calibration of Measuring and Test Equipment (M&TE) at DoE Nuclear Facilities [cancelled in 2001] Guideline for third-party certification and accreditation – Guide 58- Calibration and testing Laboratory Accreditation Systems – General Requirements for Operation and Recognition Guidelines for Calibration and Maintenance of Test and Measuring Equipment Quality Assurance Requirements for Measuring Equipment. Metrology Confirmation System for Measuring Equipment [See BS 5781-1:1992 and ISO 10012-1:1992. Superseded by ISO 10012:2003] Metrology – Calibration and Measurement Processes Guidelines [NASA Reference Publication] Standard Guide for General Criteria Used for Evaluating Laboratory Competence [See ISO Guide-25] NASA Management Instruction: Metrology Calibration and Measurement Processes [Converted to NPD 8730.1] DoE Maintenance Management Program [Section 12 – Control & Calibration of Measuring & Test Equipment. Cancelled in 1996] Calibration Laboratories and Measuring and Test Equipment – General Requirements Handbook for the Application of ANSI/NCSL Z540-1-1994 Standard Guide for General Criteria Used for Evaluating Laboratory Competence [See ISO Guide-25] DoD Standard Practice for Calibration and Measurement Requirements Note: Cancelled. Relegated requirements to ANSI Z540-1:1994 and ISO-10012-1:1992 Note: Cancelled without replacement [last revised 1989] Note: Cancelled. Superseded by ISO 10012-1:1992 Note: Cancelled 7/14/1995by IEEE Nuclear Power Engineering Committee Note: Cancelled without replacement [last revised 1963] Note: Cancelled – No longer needed per Department of Energy [last revised Feb 1973]

2016 NCSL International Workshop & Symposium

Table 1C. Chronology of calibration system standards and guidelines (continued) YEAR DOCUMENT NUMBER 1996 MIL-HDBK-1839 1996 ANSI/ASQC M1-1996 1996

SAA/SNZ HB-86.1:1996

1996

FDA 97-4179 (1st Ed)*

1997

FDA 21 CFR 820.72*

1997 1997

ISO 10012-2:1997 MIL-PRF-38793B

1997

TB 750-25

1998

NPD 8730.1

1999

ISO 17025:1999 (1st Ed)

1999 2000 2000

NCSL RP-6 (2nd Ed)** MIL-STD-1839C MIL-HDBK-1839A

2000

NUPIC Document 29 (Rev.1)

2002 2002 2002

ISPE GAMP (1st Ed)** ANSI/NCSL Z540.1-1994 (R2002) TB 43-180

2003

ISO 10012:2003

2004

ISO/IEC 17011:2004

2004 2005 2005 2005

NPD 8730.1B TB 43-180 DI-QCIC-80798B ISO 17025:2005 (2nd Ed)

2006

NUPIC Document 29 (Rev.2)

2006

DEFSTAN 05-55(PT1)/1

2006

DEFSTAN 05-55(PT2)/1

2006

DEFSTAN 05-55(PT3)/1

2006

ANSI/ASQC M1-1996 (Expired)

2006 2007 2008 2008

ANSI/NCSL Z540.3-2006 ANSI/NCSL Z540.1 (Withdrawn) NCSL RP-6 (3rd Ed)** MIL-PRF-38793C

2009

NCSL Z540.3 Handbook

2010 2010 2010 2011 2011 2012 2013 2013 2014 2015 2015 2016

MIL-STD-1839D NATO STANAG 4704 ISPE GAMP (2nd Ed)** NPD 8730.1C T.O. 00-20-14 NCSL RP-21 FDA 97-4179 (Withdrawn)* ANSI/NCSL Z540.3-2006 (R2013) AR 750-43 DIN VG-96910:2015 NCSL RP-6 (4th Ed)** ISO 17025:2017 (CD2 Draft)

TITLE OF DOCUMENT DoD Handbook – Calibration and Measurement Requirements American National Standard for Calibration Systems [Reaffirmation of ANSI/ASQC M1-1987] Handbook: A Guide to the Selection, Care, Calibration and Checking of Measuring Instruments in Industry Part 1:General Principles [Australia / New Zealand] Medical Device Quality Systems Manual: A Small Entity Compliance Guide [Chapt.7, Equipment and Calibration. Note: 1th Edition (renamed); supersedes FDA 91-4179, 5th edition 1991] Code of Federal Regulations: QSR – Inspection, Measuring, and Test Equipment [supersedes FDA 21 CFR 820.61, 1978. QSR harmonized with ISO 9001:1994 and ISO-13485:1996 CD in 1997] Quality Assurance for Measuring Equipment Part-2: Guidelines for Control of Measuring Processes Performance Specification – Technical Manuals: Calibration Procedures – Preparation Maintenance of Supplies and Equipment Army Test, Measurement, and Diagnostic Equipment (TMDE) Calibration and Repair Support (S&RS) Program NASA Policy Directive: Metrology and Calibration [Replaced NMI 5330.9B. Superseded by NPD 8730.1B] General Requirements for the Competence of Testing and Calibration Laboratories [Supersedes ISO Guide 25:1990] Recommended Practice – Calibration Control Systems for the Biomedical and Pharmaceutical Industry DoD Standard Practice for Calibration and Measurement Requirements DoD Handbook – Calibration and Measurement Requirements NRC – Nuclear Procurement Issues Committee: Commercial Grade Calibration Services Implementation Guidelines [Applicable to NUPIC Document 28, “Commercial Grade Calibration Services Checklist” – undated] Good Practice Guide: Calibration Management Calibration Laboratories and Measuring and Test Equipment – General Requirements [Reaffirmed] Calibration and Repair Requirements for the Maintenance of Army Material Measurement Management Systems – Requirements for Measurement Process and Measuring Equipment [Supersedes ISO 10012-1:1992 & ISO 10012-2:1997. AS/NZS 10012:2004 supersedes AS3912.1-1993] Conformity assessment -- General requirements for accreditation bodies accrediting conformity assessment bodies [Supersedes ISO/IEC Guide 58:1993] NASA Policy Directive: Metrology and Calibration [Superseded by NPD 8730.1C] Calibration and Repair Requirements for the Maintenance of Army Material Calibration Certificate General Requirements for the Competence of Testing and Calibration Laboratories NRC – Nuclear Procurement Issues Committee: Commercial Grade Calibration Services Implementation Guidelines [Supersedes NUPIC Document 29, Rev.1 – 2000. See also NUPIC Doc. No. 28] Measurement and Calibration System Requirement for Ministry of Defence Test & Measurement Equipment Part 1: Ministry of Defence Calibration Laboratories Operation and Management Measurement and Calibration System Requirement for Ministry of Defence Test & Measurement Equipment Part 2: Unit Level Test Measurement and Calibration System Requirement for Ministry of Defence Test & Measurement Equipment Part 3: Sub Contract of Calibration Note: First published as M1-1987, M1-1996 was not reaffirmed within 10 years of last reaffirmation; still available for reference via ASQ Catalog. Requirements for the Calibration of Measuring and Test Equipment Note: Part I replaced by ANSI/ISO/IEC 17025:2005. Part II replaced by ANSI/NCSL Z540.3-2006 Recommended Practice – Calibration Quality Systems for the Healthcare Industries Performance Specification – Technical Manuals: Calibration Procedures – Preparation Handbook for the Application of ANSI/NCSL Z540.3-2006 – Requirements for the Calibration of Measuring and Test Equipment DoD Standard Practice – Calibration and Measurement Requirements NATO Requirements for Calibration Support of Test and Measurement Equipment Good Practice Guide: A Risk-Based Approach to Calibration Management [Replaces 2002 version] NASA Policy Directive: Metrology and Calibration [Supersedes NPD 8730.1B. Expires 2016-Jun-27] USAF Air Force Metrology and Calibration Program [previous versions too numerous to list] Recommended Practice: Assessment of ANSI/NCSL Z540.3-2006 Sub-clause 5.3 [false accept risk] Note: Withdrawn without comment or replacement [Chapt. 7, Equipment and Calibration] Requirements for the Calibration of Measuring and Test Equipment [Reaffirmed] Maintenance of Supplies and Equipment: Army Test, Measurement, and Diagnostic Equipment Documentation of Calibration Services [Germany] Recommended Practice for Calibration Quality Systems for the Healthcare Industries General Requirements for the Competence of Testing and Calibration Laboratories [Committee Draft 2]

*FDA Requirements or GMP Guidance; ** Guidance via private sector healthcare industry (NCSLI or ISPE/GAMP)

2016 NCSL International Workshop & Symposium

FDA 21 CFR §133.4 (GMP) No Cal Requirements

1978 1960

1970

1973

DEFSTAN 05-26 DEFSTAN 05-27

BS 5781-1 BS 5781-2 1992

BS EN 30012-1 ISO 10012-1 ISO 10012-2 ISO 10012*

1999

ISO/IEC 17025*

1976

1997

DEFSTAN 05-32

FDA 21 CFR §820.72 (QSR) 1971

1976

1987

NCSL RP-2 NCSL RP-4 NCSL RP-11

ASTM E548

ISO 9000 Series (Quality Standards)

1979

FDA 21 CFR §820.61 (GMP)

MIL-C-45662 MIL-HDBK-52

NATO AQAP-6 NATO AQAP-7

(E36 Committee) ~1977

1987

ISO CERTICO[1] (Now CASCO)

Guidelines

ANSI/ASQC M1 ANSI/ASQC M2

1978

ISO Guide 25 ASTM E548 UKAS M10 BS 6460-1 HB 18.25 BS 7501/7502 EN 45001/45002

1994

ANSI/NCSL Z540.1 & Z540.1 Handbook

2006

ANSI/NCSL Z540.3* & Z540.3 Handbook * Asterisk designates active voluntary consensus standards for calibration. [1] CERTICO was created in 1970 as the “Committee on Certification”. In 1985, it became CASCO or “Committee on Conformity Assessment” Dates shown reflect initial publication of earliest document; see Table 1 for revision histories. Dotted lines indicate “indirect” predecessors.

Figure 1. Simplified evolution of international and national calibration standards and guidelines

2016 NCSL International Workshop & Symposium

(for calibration)

1963

Army: AR 750-25 Air Force: AFR 74-2 & BU-520 Navy: SLIM & SecNav 4355.11 Navy: Tech Memo 63-106

(for calibration)

1955

Government Standards & Guides

MIL-STD-120

Voluntary Consensus Standards & Guides

1950

MIL-STD-120 (1950) Military Standard, Gage Inspection MIL-Q-5923 (1950) General Equipment Quality Control Requirements MIL-G-14461 (1957) General Specification for Quality Control MIL-Q-9858 (1959) Quality Control System Requirements (MIL-Q-9858A , Quality Program Requirements - 1963) MIL-Q-21549A (1959) Product Quality Program Requirements for Fleet Ballistic Missile Weapon System Contractors MIL-C-45662 (1960) Calibration of Standards (MIL-C-45662A, Calibration System Requirements in 1962) (See NATO AQAP-6) MIL-I-45208 (1961) Inspection System Requirements MIL-H-110 (1960) Quality Control and Reliability Handbook NPC 200-2 (1962) Quality Program Provisions for Space System Contractors (NHB 5300.4(1B)). See also Apollo RA-001-007-1 (1966) NPC 200-3 (1962) Inspection System Provisions for Suppliers of Space Materials, Parts, Components, and Services (NHB 5300.4(1C)) MIL-C-55163 (1962) Calibration of Test and Measuring Equipment MIL-HDBK-52 (1964) Evaluation of a Contractor’s Calibration System (See NATO AQAP-7) MIL-HDBK-50 (1965) Evaluation of a Contractor’s Quality Program MIL-HDBK-51 (1967) Evaluation of a Contractor’s Inspection System (See also American Standards Association (now ANSI) Z1.1, Z1.2, Z1.3 (1941/1942) and BS 9000 (1967) ) Allied Quality Assurance Publications (NATO)

1968 - 1972

NATO AQAP-1 (1968) Quality Control System Requirements for Industry NATO AQAP-2 (1968) Guide for the Evaluation of a Contractor’s Quality Control System for Compliance with AQAP-1 NATO AQAP-4 (1970) Inspection System Requirements for Industry NATO AQAP-5 (1972) Guide for the Evaluation of a Contractor’s Inspection System for Compliance with AQAP-4 NATO AQAP-6 (1970) Measurement and Calibration Systems Requirements for Industry (See DEFSTAN 05-26 & 05-27) NATO AQAP-7 (1972) Guide for the Evaluation of a Contractor’s Calibration System for Compliance with AQAP-6 NATO AQAP-9 (1970) Basic Inspection Requirements for Industry (See also BS 4891 (1972) A Guide to Quality Assurance) British Ministry of Defence Quality Standards (MoD)

1973 - 1976

Quality System Voluntary Consensus Standards

DEFSTAN 05-21/1 (1973) Quality Control System Requirements for Industry DEFSTAN 05-22/1 (1973) Guide for the Evaluation of a Contractor’s Quality Control System for Compliance with DEFSTAN 05-21 DEFSTAN 05-24/1 (1973) Inspection System Requirements for Industry DEFSTAN 05-25/2 (1976) Guide for the Evaluation of a Contractor’s Inspection System for Compliance with DEFSTAN 05-24/2 DEFSTAN 05-26/1 (1973) Measurement and Calibration System Requirement for Industry (see BS 5781-1 and 5781-2) DEFSTAN 05-27/1 (1973) Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with DEFSTAN 05-26 DEFSTAN 05-29/1 (1973) Quality System Requirements for Final Inspection and Test DEFSTAN 05-32/1 (1976) Quality Control Requirements for Test House and Laboratory Organizations (see ISO Guide 25, BS 6460-1, BS 7501/ 7502) British Standards (BSI Group) 1974 BS 5179 (1974) Guide to the Operation and Evaluation of Quality Assurance Systems:  BS:5179-1 Part 1 – Final Inspection System  BS:5179-2 Part 2 – Comprehensive Inspection System  BS:5179-3 Part 3 – Comprehensive Quality Control System British Standards (BSI Group)

1979

BS 5750 (1979) Quality Systems. Updated in 1987 BS 5750-1 Specification for Design, Development, Production, Installation and Servicing BS 5750-2 Specification for Manufacture and Installation BS 5750-3 Specification for Final Inspection and Test (See also ANSI/ASQ Z1.15-1979 Generic Guidelines for Quality Systems) International Standards (ISO)

1987

1979 - 1982

BS 5781-1 (1979) Measurement and Calibration Systems: Specification for System Requirements

BS 5781-2 (1982) Quality Assurance Requirements for Measuring Equipment – Guide to the use of BS 5781-1

1978

ISO Guide 25 (1978) Updated in 1982 & 1990 Guidelines for Assessing the Technical Competence of Testing Laboratories (See also ASTM E548, BS 6460-1, BS 7501 / 7502 EN 45001 / 45002, HB 18.25-1991, and UKAS M10)

ISO-9000 Series (1987) Updated in 1994, 2000, 2005, 2008, & 2015 ISO 9000 Quality Management & Quality Assurance Standards – Guidelines for Selection and Use ISO 9001 Quality Systems – Model for Quality Assurance in Design/Development, Production, Installation and Servicing ISO 9002 Quality Systems – Model for Quality Assurance in Production and Installation ISO 9003 Quality Systems – Model for Quality Assurance in Final Inspection and Test ISO 9004 Quality Management and Quality System Elements – Guidelines (See also EN 29000 Series, ANSI/ASQC Q90 Series, etc.) Current International Standards (ISO) 2015 ISO-9000:2015 Quality Management Systems – Fundamentals and Vocabulary ISO 9001:2015 Quality Management Systems – Requirements ISO 9004:2009 Managing for the Sustained Success of an Organization – A Quality Management Approach (Note: ISO 9002 & 9003 are now obsolete) Medical Device International Standard ISO-13485 Quality Systems (1996) Updated in 2003, and 2016. Medical Devices – Particular Requirements for the Application of ISO 9001 (See also BS EN 46001:1997, ISO 13488:1996, EN 46002, etc.)

2016

1999

ISO 17025 (1999) Updated in 2005, CD2 2016 Draft General Requirements for the Competence of Testing and Calibration Laboratories (See also ISO 10012-1:1992, ISO 10012-2:1997, BS EN 30012:1994, and ISO 10012:2003) (See also ANSI/ASQC M1-1987, ANSI/ASQC M1-1996, ANSI/ASQC M2-1992, ANSI/NCSL Z540.1-1994 (R2002), ANSI/NCSL Z540.3-2006 (R2013))

1997 FDA Harmonized QSR in 1997 w/ ISO 9000:1994 & 13485:1996 (CD)

Figure 2. Divergence of ISO quality and calibration system standards 2016 NCSL International Workshop & Symposium

FDA 21 CFR §820 Quality System Regulation (QSR)

Government-Unique Quality & Calibration System Standards

1950 – 1967

Calibration System Voluntary Consensus Standards

Military & Aerospace Quality Standards

2. Historical Perspective Born primarily out of the defense and aerospace quality efforts of the 1950’s and 60’s, stringent quality and calibration requirements had to be met by industries wishing to supply goods and services to these government agencies; see Figure 2. One of the most important and recurring themes was how to ensure that when suppliers judged items to be In-Tolerance during calibration or inspection, they actually were In-Tolerance. A common method to achieve a reasonable level of confidence in such decisions was to require the calibration standards to be considerably more accurate than the items they were being used to test or calibrate (i.e., test accuracy ratio or TAR). Another method was to temporarily assign acceptance limits, which were tighter than the tolerance limits, during the calibration process – a practice now called guardbanding. Concepts of test accuracy (or uncertainty) ratios for calibration (e.g. 10:1 and 5:1), as well as false-accept and false-reject risk, have existed since at least 1950, as evidenced in MIL-STD-120 [2]; see Appendix B. Origins of the ubiquitous 4:1 test accuracy ratio have been documented by Castrup [3] and Mimbs [49] [A125] and attributed to Crandon and Hayes of the U.S. Navy in the 1950’s. Hayes [A5] published the first known statistical analysis of calibration quality (false accept/reject) in 1955, invoking the concept of accuracy ratios, based primarily on the 1954 seminal works of Eagle [A1] and Grubbs & Coon [A2]. Concurrently in 1953-1954, Wiesen and Clark [A3] at Sandia Corporation were independently carrying out similar analysis of how accuracy ratios and guardband limits affect incorrect accept/reject decisions, as was Mandel [A4] in the UK. They were followed shortly thereafter by Tingey & Merrill [A6] at the Atomic Energy Commission in 1957. A few years later in 1959, Wiesen & Owen [A8] at Sandia along with David et-al [A7] published additional work on the calculus of how accuracy ratios and guardbanding impact incorrect pass/fail decisions or misclassification of items judged to be inor-out of tolerance. All used the bivariate normal joint probability density model to calculate consumer and producer risk2. See Appendix A for a partial chronology of related works on measurement decision risk for calibration and testing (pass/fail) decisions. In the 1960’s, suppliers to federal agencies had to contend with a myriad of different calibration requirements. Many of these government-unique documents espoused different calibration quality mandates including different test accuracy ratios which had to be met. In most cases, minimum accuracy ratios were required, rather than detailed statistical analysis of decision-risk. See Appendix B for a history of some TAR requirements. Private industry sometimes struggled to keep up with the proliferation of these government standards and the vast array of calibration requirements they imposed, particularly those requiring specific test accuracy ratios. This was reflected in statements 50 years ago by Condon [4], Russell [5], and Fruechtenicht [6]. 2

In 1926, Shewhart addressed measurement errors as impacting ‘True Versus Observed Quality’. In 1948 , Grubbs also investigated effects of measurement errors on observed product variability and discussed isolating these variances into separate components, as did Bennett in Jan 1954. In 1962, Traver at GE, and engineers at GM, developed the Gauge R&R Study which ANOVA is now commonly used to estimate “gauge capability”, e.g. Precision to Tolerance Ratio (P/T), Signal-to-Noise Ratio (SNR), Discrimination Ratio (DR), etc. However, as stated by Montgomery, “None of these quantities really describe the capability of the gauge in any sense that is directly interpretable. The effective capability of a measuring system is best described in terms of how well it discriminates between good and bad parts… The joint probability density function… is bivariate normal… A very useful way to describe the capability of a measurement system is in terms of producer’s risk and consumer’s risk”. Appendix H.5 in ISO GUM further describes why ANOVA Gauge R&R alone is insufficient. In 1963, Eisenhart also published a seminal work, “Realistic Evaluation of the Precision and Accuracy of Instrument Calibration Systems”, but did not directly address measurement decision risk. 2016 NCSL International Workshop & Symposium

Condon: NCSL (1966): [4] “NASA’s policy on ratio-of-accuracy as stated in NPC 200-2 requires that ‘Within the state-of-the-art limitation, the standards used for calibration of inspection, measuring, and test equipment shall have a tolerance no greater than 10 % of the allowable tolerance for the equipment being calibrated.’ …many measurement requirements are becoming so sophisticated that they approached the limits of the science of metrology. In such cases, it becomes impossible to maintain the 10 to 1 ratio of accuracy in the calibration of the instrument” Russell: NCSL (1966) “…our discussion centered around the accuracy ratio of standard to instrument during measurement and calibration operations… Basically, the problem revolves around the actual or implied requirement that the accuracy of an instrument or standard used to measure a quantity, or to calibrate another instrument, shall be 10 times as accurate as the quantity or the instrument being calibrated. There is also the implication that the 10-to-1 ratio of accuracy shall exist between every level or echelon in the traceability chain for product to National Standards. This requirement could create an impossible situation… Most contractors indicated that the ratio-of-accuracy requirements imposed upon them ranged from 10:1 to 4:1, or ‘state-of-the-art’. In nearly every case, they stated that the 10-to-1 requirement was considered unrealistic from an economic as well as a practical point of view… I am of the opinion that that there are too many documents that basically state parallel requirements, the majority of which are, to a degree, unrealistic… Where 4-to-1 is maintained… the reliability of the calibrated instrument accuracy is assured…” Fruechtenicht: NCSL (1968): [6] “Specifications often referenced or required by contracts that delineate calibration requirements are:  MIL-Q-9858A  MIL-I-45208A  MIL-C-45662A  MIL-C-55163 (Sig.C)  MIL-Q-21549B (WEP)  NASA NPC-200-1A  NASA NPC-200-2  MIL-I-45607  MIL-I-8500B and  MIL-Hand-Book 50, 51, and, 52. These specifications differ as to the extent of the requirements for calibration and measurements. As examples of confusion created, some specifications require adherence to a strict 10 to 1 accuracy ratio, while others make no reference to such ratios. Due to the problems created, Government contract evaluation agencies find it difficult to enforce and administer contractual provisions as envisioned by the procuring agencies. It is important that NASA and DoD calibration system requirements be consolidated into one specification… Recommendations: That MIL-C-45662A be revised and adopted as the standard calibration specification to be referenced in all Government contracts including small business and R&D contractors where measurements are to be performed.”

Indeed, MIL-C-45662A [7] and its successors did become the ubiquitous calibration standard for critical industries for more than 30 years. This was true even outside of government agencies and their contractors. It was adopted by a multitude of industries in the U.S., particularly those which operated a quality management system, even though MIL-C-45662 had begun as a government-unique military standard. In 1982, the Food and Drug Administration [8] [9] stated: 2016 NCSL International Workshop & Symposium

FDA Device Good Manufacturing Practices – A Workshop Manual (1982): [8] (Also published in the NCSL Newsletter, Dec. 1980) [9] “The medical device GMP equipment requirements are based on MIL-C-45662 and were written to assure that production and quality assurance measurement equipment (mechanical, electronic, automated) used in the manufacture of medical devices is suitable for its intended use.”

Like the FDA calibration requirements in the Good Manufacturing Practices (GMP’s), many Federal agencies developed their own government-unique standards and requirements based, to some extent, on this venerable military standard. This included the Department of Energy (DOE), the Nuclear Regulatory Commission (NRC), and the National Aeronautics and Space Administration (NASA), among many others. General equipment calibration requirements were eventually codified into law by many government agencies via the Code of Federal Regulations (CFR). For example:   

FDA: 21 CFR §820.72 NRC: 10 CFR §50 B-XII FAA: 14 CFR §145.109

Inspection Measuring, and Test Equipment [1A] Control of Measuring and Test Equipment [10] Equipment, Materials, and Data Requirements [11]

MIL-C-45662 was first published in 1960 [12]. It had grown out of the military metrology and calibration programs established just a few years earlier in the mid 1950’s. Together, MIL-C45662A in 1962 [7] and its companion MIL-HDBK-52 in 1964 [13] formed the developmental foundation for future international/allied military calibration standards and guidelines. The North Atlantic Treaty Organization’s (NATO’s) Allied Quality Assurance Publication AQAP-6 [14] and AQAP-7 [15] for calibration were developed in the early 1970’s based to some extent these U.S. military requirements. Afterwards, the British Ministry of Defence (MoD) looked to these NATO standards when developing their own DEFSTAN 05-26 [16] and DEFSTAN 05-27 [17] calibration standards in 1973. In turn, these British MoD calibration standards were influential in the development of BS 5781-1:1979 [18] and BS 5781-2:1982 [19] which would subsequently form much of the basis for ISO 10012-1:1992 [20] and ISO 10012-2:1997 [21] (also see footnote 3 in the following section). In 2003, these two ISO documents were combined into a single document, the current ISO 10012:2003, Measurement Management Systems – Requirements for Measurement Process and Measuring Equipment [22]. See Figure 1. Concurrent to this evolution, the first edition of ISO Guide 25 had been published in 1978 [23]. Guide 25 was developed by ILAC Task Force 1, established during the first ILAC meeting of 1977 in Copenhagen [24]. Guide 25 emerged from guidelines developed earlier by the ISO Committee on Certification (CERTICO, formed 1970), now called the Committee on Conformity Assessment (CASCO, renamed in 1985). ISO Guide 25 was also derived from ASTM E548-76 (Committee E36, formed 1973) and, to some degree, DEFSTAN 05-32 in 1976. These standards addressed accreditation and quality requirements for “test house” and laboratory organizations. Guide 25 was just 4 pages long when first published in 1978. It was revised in 1982 and again in 1990. This third revision, ISO Guide 25:1990 [25] became the primary source document for the development of ISO 17025:1999 [26], with the current version being ISO 17025:2005 [27]. See Figure 1. At the time of this writing, ISO 17025 is presently under revision again by CASCO Working Group 44. A Committee Draft (CD1) was released in September 2015 for circulation and comment, with CD2 released in March 2016 [28]. A Final Draft International Standard (FDIS) is expected in late 2016 with final publication tentatively projected for September 2017. 2016 NCSL International Workshop & Symposium

3. The 1990’s: ISO Guide 25, MIL-STD-45662A, and ANSI/NCSL Z540.1 The previous background information sets the stage in the early 1990’s for the two3 competing documents that U.S. calibration programs and laboratories were being driven to comply with:  

ISO Guide 25 (1990) MIL-STD-45662A (1988)

voluntary consensus standard government-unique standard

While sharing many common objectives, there were significant differences in these documents. Maintaining compliance to multiple calibration standards was a challenge for U.S. industry, similar to the situation alluded to earlier in the late 1960’s. ISO Guide 25 was the prevailing voluntary consensus standard for calibration labs in 1990. Meanwhile, the government-unique MIL-STD-45662A [29] in 1988 had evolved substantially since its initial introduction in 1960 as MIL-C-45662. Most notably perhaps, MIL-STD-45662A was the first version to formally require the ubiquitous 4:1 Test Accuracy Ratio (TAR), although not specifically referenced by name. Section 5.2 of MIL-STD-45662A required that: MIL-STD-45662A (1988): [29] “5.2 Adequacy of Measurement Standards. Measurement standards used… for calibrating M&TE and other measurement standards shall be traceable and shall have the accuracy, stability, range and resolution required for the intended use. Unless otherwise specified in contract requirements, the collective uncertainty of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each characteristic being calibrated. The… calibration system description may include provisions for deviating for the uncertainty requirements, provided the adequacy of the calibration is not degraded. All deviations must be documented.”

A brief summary of the TAR requirements throughout the life of MIL-STD-45662A and its associated handbook is given.       

1960: 1962: 1964: 1980: 1984: 1988: 1995:

MIL-C-45662: MIL-C-45662A: MIL-HDBK-52: MIL-STD-45662: MIL-HDBK-52A: MIL-STD-45662A: Cancelled:

Required a 10:1 TAR [12] No TAR specified [7] Required 4:1 to 10:1 TAR [13] No TAR specified [30] Mentions 10:1, 4:1, 3:1, 2:1, & 1:1 TAR examples only [31] Required 4:1 TAR [29] Replaced by Z540.1-1994; retained 4:1 TAR [32]

MIL-STD-45662A and many of its predecessors had invoked test accuracy ratios as a risk mitigation technique. This was done to ensure that calibration standards were adequate for their intended use by limiting the risk of incorrect decisions during calibration. This was accomplished by ensuring that the laboratory standards were sufficiently more accurate than the instruments 3

The ANSI/ASQC M1-1987 standard had been previously published in 1987 and was reaffirmed in 1996 [74]. A history of its development was given by one of the M1/M2 authors (Schumacher) [32A] [32B]. However (arguably), the M1 standard was not as widely adopted as the later ANSI/NCSL Z540.1-1994 standard, despite M1’s comprehensive nature, a focus on “measurement assurance”, and co-authorship by Dr. Belanger of NBS/NIST. M2 was also influential in ISO-10012-2:1997. Appendix B provides a summary of M1’s treatment of accuracy ratios. In 1993, the first edition of the ISO Guide to the Expression of Uncertainty in Measurement (GUM) was also published, standardizing the way uncertainties were calculated internationally. The term Test Uncertainty Ratio (TUR) slowly began to replace Test Accuracy Ratio (TAR). A distinction is made in the Z540.3-2006 Handbook [45] between Test Uncertainty Ratio (TUR) and Test Accuracy Ratio (TAR), as elaborated by Mimbs [49]. 2016 NCSL International Workshop & Symposium

they were employed to calibrate. Incorrect pass/fail decisions can result from excessive measurement uncertainty (inaccuracy) associated with the laboratory’s standards and processes. A ≥4:1 TAR had become a quality hallmark of calibration in the U.S., the popularity of which would be difficult to overstate. See Appendix B. By contrast to MIL-STD-45662A (1988), ISO Guide 25 did not contain any requirement or mention of test accuracy ratio throughout its history from 1978 to 1990. This and several other differences between ISO Guide 25 and MIL-STD-45662A caused calibration requirements in the U.S. to become somewhat splintered into two different paradigms (ISO Guide 25 vs. MIL-STD45662A). Once again, many industries and calibration laboratories in the U.S. were struggling to resolve the apparent conflict, often having to comply with both to satisfy a broad range of interests. As reported by Held [33] in 1992 when quoting Randall [34]: T. Held – Abbott Laboratories (1992): [33] “In response to this need, the NCSL [National Conference of Standards Laboratories] formed a standing TQM committee on calibration systems and tasked it with two immediate objectives: 1. Bring together the various government bodies which audit calibration quality systems for the purpose of establishing a single American National Calibration Quality System; and 2. Promote the establishment of a nationally recognized calibration laboratory accreditation system. This committee quickly evolved, gathering representatives from  NIST  FAA  FDA (The FDA has from the onset strongly supported the first of the two objectives, but has until only recently withheld their support for the second. Currently, the FDA supports both of the TQM Committee's objectives)  NASA  U.S. Dept of Energy (DoE)  U.S Nuclear Regulatory Commission (NRC)  U.S. Naval Warfare Assmt. Ctr.  U.S. Army T.M.D.E.  U.S. Air Force  Defense Logistics Agency (DLA)  Office of the Secretary of Defense (DoD)  various private sector interests (Including Abbott Laboratories) After having spent months comparing various American Quality System requirements with the new European standards for compatibility, the committee overwhelmingly voted to adopt a proposed U.S. consensus standard consisting of ISO/IEC Guide 25 and some additional requirements to make it compatible with other European standards and satisfy some requirements important to U.S. industry and government agencies [i.e. MIL-STD-45662A]. Note that the incorporation of ISO/IEC Guide 25 will include ISO 9000 compliance and international recognition. A final draft of this document was written by Dr. Joe D. Simmons, Chief of Calibration Services at NIST. This draft was reviewed and unanimously endorsed by the TQM Committee. At this point the TQM Committee is seeking a group which will publish the standard for distribution and use.”

2016 NCSL International Workshop & Symposium

The NCSL TQM committee would eventually become the ANSI Z540 Committee, formally known as the ANSI Accredited Standards Committee on General Requirement for Calibration Laboratories and Measuring and Test Equipment. The document they developed would ultimately be published as ANSI/NCSL Z540.1-1994, “American National Standard for Calibration – Calibration Laboratories and Measuring and Test Equipment – General Requirements” [35]. Although the FDA was a member of the NCSL TQM Committee, it was not part of the final ANSI Z540 Committee for Z540.1 [35; see foreword]. However, additional comments on the two objectives of the TQM Committee and the FDA’s position on these issues were provided in 1992, also by Held [33]: T. Held – Abbott Laboratories (1992): [33] “Of interest within the U.S., is the position being taken by the FDA in their representation on the NCSL TQM Committee. The FDA from the onset strongly supported the objective of developing a single American National Calibration Control System Standard stating that they would probably use it as a supplement to existing GMP/GLP's. However, the FDA initially felt obligated to withhold their support of the concept of establishing and utilizing a nationally recognized calibration laboratory accreditation system. The FDA was apprehensive that the utilization of accredited laboratories could be perceived of as the FDA's transferring its legislative responsibility to possible private for-profit third party laboratories. At the NCSL TQM Committee meeting of July 92' the FDA announced a change in position. Their position now is that they support both of the TQM Committee's objectives:  the development of a single standard and  laboratory accreditation.”

Efforts to assimilate ISO Guide 25 and MIL-STD-45662A into a single U.S. Calibration Standard, and to develop a nationally recognized laboratory accreditation program are further documented by Cigler [36] of NIST; a history of accreditation is given by Neumann [36A]. J. Cigler – NVLAP Calibration Laboratory Program (1993): [36] “I am working closely with the NCSL TQM Committee to tailor ISO Guide 25 to the needs of U.S. Government and private industry. It should be noted that, with very minor exceptions, nothing is being tailored out of the basic ISO Guide 25 requirements. Rather, an attempt is being made to add requirements which satisfy government audit requirements (for example, MIL-STD-45662A for the Department of Defense). This is being accomplished through the NCSL TQM Committee in the form of a proposed U.S. Standard currently titled ‘General Requirements for Calibration Laboratories and Measuring and Test Equipment’ [Z540]. When completed, it will serve as this country’s interpretation and implementation of ISO Guide 25 in determining the competence of calibration laboratories as part of the NVLAP accreditation process.”

Schumacher in 1995 [32B] and Harris in 2001 [36B] would later recount the efforts of the NCSL TQM Committee (ANSI Z540 Committee) and the participation of various federal agencies. Schumacher, NCSL (1995): [32B] “…an NCSL [TQM] committee was formed by Gary Davidson with the purpose to eliminate the various similar calibration control system requirements standards issued by various agencies of the US federal government with a single standard [Z540] they could all agree upon. For that purpose, committee members were invited from numerous agencies of the federal government… Hence, the NCSL committee proceeded with writing such a standard. Its two parts are based on the ISO/IEC Guide 25 and on MIL-STD-45662A”. 2016 NCSL International Workshop & Symposium

Harris, NIST SP-986. NIST Workshop on Conformity Assessment for a Changing Government (2001): [36B] “ANSI/NCSLI Z540, Standards Writing Committee Participation - Representation on NACLA Board - Sponsorship of NACLA (Patron Member) - Stakeholder in ILAC - Annual meetings with NMI management (NIST, CENAM, NRC) Collaboration - Requested NIST establish a laboratory accreditation program for “calibration laboratories” in NVLAP - Includes government representatives on writing committees to encourage adoption of standards… - Adoption of Z540-1-1994 in NVLAP & OWM laboratory criteria at NIST - Adoption of Z540-1-1994 by DoD DoE etc for calibration laboratories. DOE adoption of ISO/IEC 17025 - FAA contacted Airline Industry Committee – regarding Z540-1 applicability to FAA regulation”.

Indeed, when Z540.1 was published in 1994, it was quickly embraced by industry and enjoyed widespread adoption in some agencies for more than a decade. With its ≥4:1 TAR requirement carried over from the military, the sanctity of this calibration quality metric in Z540.1 remained firmly entrenched in U.S. industry. It was thought to provide evidence that incorrect acceptance decisions during calibration would be rare. A 4:1 TAR instilled confidence that when a laboratory’s calibration certificate stated an item was found to be “In-Tolerance”, it actually was. Less than a year after Z540.1 was published in 1994, the Department of Defense formally cancelled MIL-STD-45662A in February of 1995, referring future requirements to Z540.1 (and ISO 10012-1) as alternatives. NATO’s AQAP-6 was also cancelled the same year, citing ISO 10012-1 as its replacement. At the time, guidance within the ISO 10012-1:1992 document alluded to both 3:1 and 10:1 TAR as guidance (see Appendix B). Eleven years later in 2003, when ISO 10012-1:1992 and 10012-2:1997 were combined into a single standard (ISO 10012:2003) the conceptual reference to test accuracy ratio was removed altogether. 4. The 2000’s: ISO 17025 and ANSI/NCSL Z540.3 During the evolution of ISO Guide 25 (3rd edition, 1990) into the 1st edition of ISO 17025 in 1999, there was much debate regarding how the accuracy (or uncertainty) of the calibration process should be taken into account during compliance decisions. As discussed earlier, taking uncertainty into account is necessary to mitigate the risk of incorrect decisions during claims of In-or-Out of tolerance. By requiring a 4:1 test accuracy ratio, MIL-STD-45662A had taken the global or program-level approach to risk management; if the accuracy of the measurement standard(s) was 4 times better than the item being tested, then, on the average, incorrect decisions would occur infrequently. The effectiveness of this simple 4:1 rule did require some rather significant assumptions regarding equipment reliability [3] [37] [38] [39], but the simplicity of the 4:1 TAR rule made it an attractive technique (see footnote 5 in Section 5 of this paper for a brief discussion of global vs. specific risk). As stated, in 1990, ISO Guide 25 contained no test accuracy ratio requirement and did not address how measurement uncertainty might result in incorrect decisions. It simply stated that appropriate methods and procedures should “…be consistent with the accuracy required”. 2016 NCSL International Workshop & Symposium

It did require that calibration certificates “…shall provide the measurement results and associated uncertainty of measurement and/or a statement of compliance with an identified metrological specification”. Early drafts, of what was to become the 1st edition of ISO 17025, reveal that the authors seriously contemplated including accuracy (or uncertainty) ratio considerations, as well as a specific decision rule for In-Tolerance declarations. Similar to the 3:1 TAR referenced in ISO 10012-1:1992, Draft V of ISO Guide 25 [40], which circulated in August 1996, proposed the following: ISO Guide 25 Draft V (1996): [40] “3.9.2.5 When, in the case of calibration certificates, parameters(s) are claimed to be within specified tolerances, the measurement values(s), extended by the estimated uncertainty of measurement, shall fall within the appropriate specification limit. Notes

1. A statement of compliance should only be made if the ratio of the uncertainty of measurement to the specified tolerance is reasonably small, e.g. 1:3.

2. If the measurement value, extended by the measurement uncertainty, exceeds the specified tolerance while the measurement value itself falls within the tolerance, neither compliance nor non-compliance can be proved. Only the measurement results and the associated uncertainty can then be given in the certificate without any statement of compliance.”

Ultimately, when ISO Guide 25 did evolve into the 1st edition ISO 17025 and was finally published in 1999, the language in the earlier draft relating to any type of accuracy or uncertainty ratio (or decision rules) to mitigate the risk of incorrect acceptance decisions had been removed. Moreover, when ISO 17025 was later revised in 2005 (2nd edition), no additional treatment of accuracy ratios or measurement decision risk was added at that opportunity in time. Today, the pertinent requirement is found in section 5.10.4.2 which simply requires that: ISO 17025:2005 (2005): [27] “5.10.4.2 When statements of compliance are made, the uncertainty of measurement shall be taken into account.”

Ten years later in September 2015, the first committee draft (CD1) of the pending 3rd edition of ISO 17025 [41] did not necessarily add any additional technical requirements to this matter. However, it did propose some much needed direction, stating: ISO 17025 CD1 Draft (2015): [41] “7.1.1 (d) when the customer requests a verification of conformity to a specification or standard for the test or calibration (e.g. pass/fail, in-tolerance/out-of-tolerance):  the specification is clearly defined in the procedure selected;  the decision rule for conformity, its level of risk and statistical assumptions is documented in the test method/procedure or is documented by the laboratory and communicated to the customer;

 the decision rule is agreed to by the customer.” 2016 NCSL International Workshop & Symposium

When the second draft (CD2) of ISO 17025 was released in March 2016 [28], it proposed: ISO 17025 CD2 Draft (2016): [28] “7.1.1.3 When the customer requests a statement of conformity to a specification or standard for the test or calibration (e.g. pass/fail, in-tolerance / out-of-tolerance) the specification and the decision rule shall be clearly defined and communicated. 7.7.1 When statement of conformity to a specification or standard for test or calibration is requested, the laboratory shall: a) Document the decision rules employed taking into account the level of risk (such as false accept and false reject and statistical assumptions) associated with the decision rule employed; b) Apply the decision rule. NOTE For further information see ISO/IEC Guide 98-4”.

The note in the CD2 draft of ISO 17025 is of great significance, as it references ISO Guide 98-4 [42], which is also known as JCGM 106:2012, “Evaluation of Measurement Data – The Role of Measurement Uncertainty in Conformity Assessment” [43]. Compliance with ISO/IEC Guide 98-4 is not specifically required by CD2 of ISO 17025. However, Guide 98-4 provides a thorough technical treatment of the statistical considerations which should be addressed any time a measured value is asserted to fall inside or outside of acceptable tolerance limits. There is always a risk that such a pass/fail decision or assertion is incorrect, due to measurement uncertainty. If the uncertainty is small by comparison with the acceptable tolerance limits (high TUR), then, on the average, the risk or probability incorrect decisions will be low. Conversely, if this uncertainty is comparable in magnitude with the tolerance limits (low TUR), the average risk of incorrect conformance decisions is elevated. This is the nature of calibration and verification. Six years prior to the publication of JCGM 106:2012, the NCSLI 174 Standards Writing Committee and ANSI had published ANSI/NCSL Z540.3-2006 [44], replacing Z540.1-1994 (R2002). For the first time in a national or international standard, Z540.3 directly addressed, in a formal manner, the rather complex issue of false accept risk during calibration. The Z540.3 standard requires that: ANSI/NCSL Z540.3 (2006): [44] “5.3 (b) Where calibration provide for verification that measurement quantities are within specified tolerances, the probability that incorrect acceptance decision (false accept) will result from calibration tests shall not exceed 2% and shall be documented. Where it is not practicable to estimate this probability, the test uncertainty ratio shall be equal to or greater than 4:1. NOTE: Achieving these requirements may involve adjustment and management of calibration system parameters such as: measurement reliability, calibration intervals, measurement uncertainty, calibration tolerances, and/or guard bands. Calibration-servicing components may be considered competent to provide calibration services when they have been accredited to meet ANSI/ISO/IEC 17025, including the requirements of this sub-clause…”

2016 NCSL International Workshop & Symposium

Additionally, NCSL International published a handbook to accompany Z540.3 [45] which contains a substantial amount of information on measurement decision risk, including six methods which can be employed to comply with the 2 % false-accept risk requirement. This guidance consumes eight pages, while Appendix A of the Z540.3 Handbook spans 45 pages. In total, more than 50 pages are dedicated to the subject of risk (probability) of incorrect decisions during in-or-out of tolerance declarations. Subsequently, additional methods have also been published [37] [38] [39] [39A]. Two NCSLI Recommended Practices, RP-18 [46] and RP-21 [47], also address the risk of incorrect decisions during calibration. Caldwell [48] has given an overview of the Z540.3 standard. A distinction is also made in the Z540.3 handbook between test uncertainty ratio (TUR) and test accuracy ratio (TAR), as elaborated by Mimbs [49]. As discussed earlier, the concept of measurement decision risk had been formalized in the 1950’s. Over the years, a proliferation of papers have been published from individuals on the subject of accuracy (or uncertainty) ratios and the risk of incorrect pass/fail decisions during calibration and testing. Appendix A provides a partial list of these papers and serves to demonstrate the considerable attention this subject has garnered, particularly in the 21st century where ISO 9001:2015 has adopted a “risk-based-thinking” model [50]. In addition to independent authors, there exists many guidelines from a variety of engaged organizations which deal directly with measurement decision risk as applied to calibration and the adequacy of measurement processes. Several prominent publications on measurement and calibration decision risk (misclassification) are listed here, many of which are available via public domain.                        

1984: 1994: 2001: 2002: 2003: 2004: 2005: 2007: 2008: 2009: 2009: 2010: 2010: 2010: 2012: 2012: 2012: 2012: 2012: 2012: 2013: 2014: 2014: 2015:

NBS (NIST) Special Publication 673 [51] NASA RP-1342 – Sections 4.10, 6.5, 6.6, Appendix C.5 & F.5 [52] ASME B89.7.3.1-2001 [53] UKAS LAB34 – Section 4 [54] ISO 10576-1:2003 [55] AFNOR FD x07-022, Section 6 [A79] ASME B89.7.4.1-2005 [56] Eurachem/CITAC Guide [57] Eurolab Cook Book Doc No. 8.0 [58] Z540.3 Handbook – Section 5.3 and Appendix A [45] ILAC-G8:03/2009 [59] NASA HDBK 8739.19-4 [60] VDA-5 [61] AIAG MSA-4 – Chapter 1, Section B & Chapter 3 Section C [62] UKAS M3003 – Appendix M [63] NCSL RP-21 [47] JCGM 106:20012 [43] ISO/IEC Guide 98-4 [42] ISO 22514-7 [64] ISO/TR 14253-6 [A140] ISO 14253-1 [65] WADA TD2014DL [66] NCSL RP-18 [46] ISO 14253-5 [A157]

2016 NCSL International Workshop & Symposium

5. Calibration in the Healthcare Industry Medical device and pharmaceutical manufacturers, regulated by the FDA, must comply with the applicable sections in Title 21 of the Code of Federal Regulations (CFR). Table 2 provides an historical summary of the prominent FDA equipment calibration requirements, including the “suitable for intended purpose” clause. These may be considered government-unique standards. The CFR calibration requirements do not specifically indicate how instruments should be assessed to ensure they are “suitable for their intended purpose” (i.e. no accuracy ratios or uncertainty considerations) nor what the qualifications or objective evidence for “traceability” are. In 1997, the FDA [67] stated that: FDA, Guide to Inspections of Medical Device Manufacturers (1997): [67] “Manufacturers must assure all inspection, measuring and test equipment (including mechanical, automated or electronic inspection and test equipment) is suitable for its intended use and is capable of producing valid results. This would normally be done through installation, operation and performance qualification [IQ/OQ/PQ] of the equipment”

The first statement above is a direct requirement of 21 CFR §820.72. However, the second statement is the FDA guidance on how that requirement would normally be achieved. From a literal perspective, it is difficult to imagine that “all” measuring and test equipment (to include micrometers, calipers, rulers, hand-held meters, etc.) would be subjected to rigorous IQ/OQ/PQ type qualification and validation activities. For medical device manufacturers, such extensive qualification activities for measuring equipment are typically limited to analytical and/or automated/computerized measuring apparatus addressed in USP [68] (e.g. “Group C”). This is contrasted with Group B instruments which may only require calibration and preventative maintenance, rather than qualification. ISPE GAMP-5 [69] further categorizes software and computerized systems for analytical instrument qualification. Efforts to harmonize these two guidance documents on qualification were published in 2014 by Vuolo-Schuessler et-al [70]. However, a certain degree of subjectivity exists when deciding which measuring instruments must be qualified via the IQ/OQ/PQ process to show suitability for the intended use, and which instruments must simply be calibrated. Thus, the type and extent of objective evidence necessary to satisfy the “suitable for intended use” clause remains somewhat nebulous. Recent standards such as ISO 22514-7:2012 [64], provide some guidance on how objective evidence to show suitability might be achieved. Similar to the ubiquitous process capability index of 1.33, ISO 22514-7 recommends maintaining a “measuring system capability index (CMS)” of ≥1.33 and Minitab® notes that a “gage capability index (Cgk)” of ≥1.33 is a common benchmark to assess repeatability and bias for a “capable gage” [64A]. ANOVA Gauge R&R techniques often espouse “gauge capability” requirements for “precision to tolerance ratio” (see footnote 2 in Section 2 of this paper). However, a thorough statistical treatment of the subject is provided in JCGM 106:2012 as a discussion of measurement capability index (Cm), mathematically equivalent to test uncertainty ratio or TUR in Z540.3. In such an approach, suitability might be determined based on the ability to provide acceptably low false-accept risk. That is, suitability is related to the probability that a measuring instrument or system may erroneously indicate that a product or process is in-tolerance. Measuring instruments exhibiting an acceptably low falseaccept risk may then be deemed suitable and “capable of providing valid results”. 2016 NCSL International Workshop & Symposium

Table 2. Review of historical FDA GMP/QSR calibration requirements FDA 21 CFR §133.4 (GMP)* Equipment (1963 – Obsolete) [71]

FDA 21 CFR §820.61 (GMP) Measuring Equipment (1978 – Obsolete) [72]

FDA 21 CFR §820.72 (QSR) Inspection, Measuring, & Test Equipment (1997 - Current) [1A]

Equipment used for the manufacture, processing, packaging, labeling, holding, or control of drugs shall be maintained in a clean and orderly manner and shall be of suitable design, size, construction, and location in relation to surroundings to facilitate maintenance and operation for its intended purpose. The equipment shall:

All production and quality assurance measurement equipment, such as mechanical, automated, or electronic equipment, shall be suitable for its intended purposes and shall be capable of producing valid results. Such equipment shall be routinely calibrated, inspected, and checked according to written procedures. Records documenting these activities shall be maintained. When computers are used as part of an automated production or quality assurance system, the computer software programs shall be validated by adequate and documented testing. All program changes shall be made by a designated individual(s) through a formal approval procedure.

a) Control of inspection, measuring, and test equipment. Each manufacturer shall ensure that all inspection, measuring, and test equipment, including mechanical, automated, or electronic inspection and test equipment, is suitable for its intended purposes and is capable of producing valid results. Each manufacturer shall establish and maintain procedures to ensure that equipment is routinely calibrated, inspected, checked, and maintained. The procedures shall include provisions for handling, preservation, and storage of equipment, so that its accuracy and fitness for use are maintained. These activities shall be documented.

(a) Calibration. Calibration procedures shall include specific directions and limits for accuracy and precision. There shall be provisions for remedial action when accuracy and precision limits are not met. Calibration shall be performed by personnel having the necessary education, training, background, and experience.

(b) Calibration. Calibration procedures shall include specific directions and limits for accuracy and precision. When accuracy and precision limits are not met, there shall be provisions for remedial action to reestablish the limits and to evaluate whether there was any adverse effect on the device's quality. These activities shall be documented.

(b) Calibration standards. Where practical, the calibration standards used for production and quality assurance measurement equipment shall be traceable to the national standards of the National Bureau of Standards, Department of Commerce. If national standards are not practical for the parameter being measured, an independent reproducible standard shall be used. If no applicable standard exists, an inhouse standard shall be developed and used.

(1) Calibration standards. Calibration standards used for inspection, measuring, and test equipment shall be traceable to national or international standards. If national or international standards are not practical or available, the manufacturer shall use an independent reproducible standard. If no applicable standard exists, the manufacturer shall establish and maintain an in-house standard.

(c) Calibration records. The calibration date, the calibrator, and the next calibration date shall be recorded and displayed, or records containing such information shall be readily available for each piece of equipment requiring calibration. A designated individual(s) shall maintain a record of calibration dates and of the individual performing each calibration.

(2) Calibration records. The equipment identification, calibration dates, the individual performing each calibration, and the next calibration date shall be documented. These records shall be displayed on or near each piece of equipment or shall be readily available to the personnel using such equipment and to the individuals responsible for calibrating the equipment.

(a) be so constructed that any surfaces that come into contact with drugs are suitable, in that they are not reactive, additive, or absorptive to an extent that significantly affects the identity, strength, quality, or purity of the drug or its components. (b) be so constructed that any substances required for the operation of the equipment, such as lubricants or cool ants, may be employed without hazard of becoming additive to drug products. (c) be constructed to facilitate adjustment, cleaning, and maintenance as necessary to assure the reliability of control procedures, to assure uniformity of production, and to assure the exclusion from drugs of contaminants, including those from previous and current manufacturing operations. (d) Be of suitable size and accuracy for use in any intended measuring, mixing, or weighing operations.

* The 1963 GMP equipment regulations (21 CFR §133.4) did not contain any calibration requirements for equipment.

2016 NCSL International Workshop & Symposium

Concepts of measuring system “suitability” or “fitness for use”, based on the ability to provide valid results with adequately low false accept risks, were recently addressed in a stimuli article in the U. S. Pharmacopeial Forum. This article is entitled, Fitness for Use: Decision Rules and Target Measurement Uncertainty [A159]. U.S. Pharmacopeial Forum Stimuli Article (2016): [A159] “There is a long history behind DRs [Decision Rules], which have been used to provide organizations with procedures for accepting and rejecting products… Some regulatory bodies in other industrial sectors are already using DRs in formulating laws and regulations for dealing with commodities, including the food and nuclear industries, environmental regulations, and highway traffic laws… The DR defines the use of the reportable result, and can provide the information, such as acceptable probabilities, needed to set the target measurement uncertainty (TMU), which is defined in the International Vocabulary of Metrology as a ‘measurement uncertainty specified as an upper limit and decided on the basis of intended use of measurement results.’ …DRs can take into account an acceptable level with regard to the probability of making a wrong decision. The wrong decision can lead to accepting an out-of-specification (OOS) reportable result which is not true or rejecting an OOS reportable result which is true, or a false failure or a missed fault…”

Target Measurement Uncertainties (TMU’s)4 are similar to minimum Test Uncertainty Ratios (TUR’s) in that each can be used to specify the maximum allowable uncertainty, based on the intended use of the measurement result. For calibration, the target measurement uncertainty might be specified as a minimum 4:1 TUR, and the intended use might be to verify that test and measurement equipment complies with established metrological specifications or allowable tolerances. For pharmacopeial assays, the TMU might be specified directly, and the intended use of the measurement result might be to verify that a drug has the required potency, within specified limits. In each case, the required/minimum Test Uncertainty Ratio (TUR) or Target Measurement Uncertainty (TMU) provides an upper limit on the magnitude of the acceptable measurement uncertainty for the intended use, i.e., deciding conformity to specifications with acceptably low risk of incorrect decisions. The Eurachem/CITAC guide on Setting and Using Target Uncertainty in Chemical Measurements [A158] (section 5.1.2, equation 1) recommends the Target Measurement Uncertainty (Utg), be 8 times smaller than the interval range (or span) of the specification limits (Qmax – Qmin) for the measurand. This Target Measurement Uncertainty is mathematically identical to a Test Uncertainty Ratio of 4:1, as defined in Z540.3. ANSI/NCSL Z540.3-2006 (R2013): [44] “3.11 Test uncertainty ratio: The ratio of the span of the tolerance of a measurement quantity subject to calibration, to twice the 95% expanded uncertainty of the measurement process used for calibration. 5.3(b) …the probability that incorrect acceptance decision (false accept) will result from calibration tests shall not exceed 2% and shall be documented. Where it is not practicable to estimate this probability, the test uncertainty ratio shall be equal to or greater than 4:1”.

4

Target Measurement Uncertainties (also called International Target Values or ITV’s) emerged from efforts in 1979 by the nuclear industry to safeguard the quantity or inventory of nuclear and/or fissile materials held in various locations, which are subject to periodic verification and accountancy via analytical measurement techniques [196]. Target Measurement Uncertainties have since also found favor with the analytical chemistry community [A158]. Although defined in the VIM, this term has not, thus far, been embraced by the calibration community at large, particularly in the U.S. As of this writing, a Google search of the term “target measurement uncertainty” together with “test uncertainty ratio” yields zero results returned (although Weitzel [173D] [173E] does relate TMU & TAR). 2016 NCSL International Workshop & Symposium

Eurachem/CITAC Guide on Setting and Using Target Measurement Uncertainty (2015): [A158] “This document discusses how to set a maximum admissible uncertainty, defined in the third edition of the International Vocabulary of Metrology as the ‘target uncertainty’, to check whether measurement quality quantified by the measurement uncertainty is fit for the intended purpose… The target uncertainty can be inferred from the compliance interval, defined by a minimum and a maximum limit (section 5.1.2), or from the quantity value, above or below a single limit, beyond which there should be a low probability of an incorrect compliance decision. If a compliance interval is defined for the measurand… the uncertainty should be small enough to distinguish max min quantities within this interval. If the compliance interval is defined by a maximum Q and a minimum Q tg quantity, the target expanded uncertainty, U , should typically be 8 times smaller than the interval range:

𝑈𝑡𝑔 =

𝑄 𝑚𝑎𝑥 − 𝑄 𝑚𝑖𝑛 8

…an important reason for setting a target uncertainty is the use of uncertainty in the assessment of compliance… The key to the assessment of compliance is the concept of ‘Decision rules’. These rules give a prescription for the acceptance or rejection of a product based on the measured quantity value, its uncertainty and the specification limit or limits, taking into account the acceptable level of the probability of making a wrong decision.”

Target Measurement Uncertainties have also been employed to identify the “end-use” uncertainty objective of the final result (or legal limit), such that, yet a sufficiently smaller measurement uncertainty should be used to determine compliance to such a limit. Section 2.1.5 of the Eurachem guide on Terminology in Analytical Measurement – An introduction to VIM 3 [202] appears to advocate a 5:1 Test Uncertainty Ratio “rule of thumb” in traceability chains: Eurachem Guide on Terminology in Analytical Measurement (2011):

[202]

“2.1.5 ...if the intention is to determine if a specific test sample complies with a legal limit… As a rule of thumb the measurement uncertainty for those steps in the measurement procedure that have a significant effect on the result should be ≤1/5 of the target measurement uncertainty for the final result. When this condition is met, the individual steps concerned will make a negligible contribution to the overall measurement uncertainty.”

An example in this Eurachem guide is also given with respect to EU legislation on maximum allowable uncertainty for the testing of surface water for lead content, where a 2:1 Test Uncertainty Ratio is apparently acceptable [202]. WELMEC uses the terms Maximum Permissible Uncertainty (MPU) and Maximum Permissible Error (MPE), recommending that “where practically possible, MPU ≤ MPE/3”, equivalent to a 3:1 TUR [A86]. Eurachem Guide on Terminology in Analytical Measurement (2011): [202] “3.1.2 The laboratory will aim to remove sources of significant uncertainty until the measurement procedure is deemed to be fit for purpose. This means that the laboratory should know the maximum measurement uncertainty that can be accepted by the customer for a specific application. This is called the target measurement uncertainty. For example, the EU legislation regarding the official control for monitoring water status state that laboratories performing measurements should use measurement procedures capable of providing results with an ‘uncertainty of measurement of 50 % or below (k=2) estimated at the level of relevant environmental quality standards.’ For example, the environmental quality standard for lead in surface waters -1 -1 is 7.2 µg L , so the target measurement uncertainty is 3.6 µg L .”

2016 NCSL International Workshop & Symposium

TUR (or TMU or MPU) can be used to calculate (or can be based on) the probability of making incorrect decisions (i.e. misclassification, missed faults, false failure, etc.), to formulate decision rules, and/or to determine acceptable guardband limits if necessary. In both cases, the acceptable measurement uncertainty is directly related to the global probability of making incorrect accept/reject decisions5. Global probabilities associated with making incorrect decisions are also dependent on the a priori probability that the device (or process) being tested complies with its specification(s) [A96]. For calibration, this is often called End of Period Reliability (EOPR) or “as-found in-tolerance probability” and can be estimated from observed historical reliably data. Mathematical boundary conditions have been shown to exist for TUR and EOPR, which can effectively limit global false-accept risk to selected levels [37] [38] [39] [39A]. Similar reliability data is often available for manufacturing processes, analytical assays, control charts, etc., where the historic ability (or confidence) to meet specified limits can be used to estimate the variability of the product or process under test. Such boundary conditions and risk mitigation techniques are equally applicable to these endeavors, not just calibration processes. Rather than requiring (sometimes arbitrary) minimum Test Uncertainty Ratios or maximum allowable Target Measurement Uncertainties, Z540.3 addresses the root intent of these attempts at risk mitigation. Rather than placing requirements on secondary parameters which are merely related to measurement decision risk, Z540.3 advocates placing allowable limits on false accept risk itself. Risk can then be mitigated by controlling Test Uncertainty Ratio, reliability of the end-item/process, and/or guardband limits. In addition to the US Pharmacopeial Forum stimuli article and the Eurachem/CITAC guides previously discussed, Target Measurement Uncertainties (and guardbanding), along with their utility in establishing “fitness for purpose or intended use”, have been addressed by several authors. These include Weitzel & Johnson [A138], Majcen, Skubic & DeBièvre [197], DeBièvre [198] [199], DaSilva & Williams [200], among many others. Accreditation bodies have also recognized TMU’s with respect to establishing uncertainty requirements in medical laboratories accredited to ISO 15189 [201]. Weitzel & Johnson - USP: (2012) [A138] “The target measurement uncertainty can be decided by following a process which involves determining…the acceptable level of risks of incorrect decisions of compliance; developing a suitable decision rule, with guard bands if appropriate; using the probability of making an incorrect decision of compliance based on the decision rule… The fitness for purpose of a quantitative method… when assessing compliance to requirements can be described by the maximum measurement uncertainty that is acceptable based on the use of the data. Decision rules, measurement uncertainty, risk and probability have been used in many scientific areas for many years. We can leverage this experience to more effectively meet cGMP requirements… The USP is using the TAR [Test Accuracy Ratio] decision rule to determine if its Certified Reference Materials are fit-foruse.” Weitzel [173E]

See also comments from the FDA [173A] and USP [173B] [173C] [173D] [173E] regarding test accuracy ratios, measurement uncertainty, decision risk, and fitness-for-use toward the end of Section 7 of this paper. 5

JCGM 106:2012 [43] invokes the term “global” consumer/producer risk when addressing probabilities of incorrect decisions, prior to obtaining any specific individual measurement result. The terms “program-level” risk [A96] and “average” risk [56] have also been used in this capacity. By contrast, JCGM 106 uses the term “specific” risk when addressing probabilities of incorrect decisions associated with a specific measurement result . The terms “bench-level” [A96] risk, “individual” [56] risk, and “local” [A85A] risk have also been used in discussing this level of risk. The distinction between global and specific risk is critically important, but is beyond the scope of this paper. 2016 NCSL International Workshop & Symposium

The FDA introduced measuring and test equipment calibration requirements into the GMP’s in 1978 [72]. Through a series of workshops, along with publication of Good Manufacturing Practice manuals from 1978 to 1997, the FDA provided guidance to the industry on how to comply with the many aspects of the GMP’s. Most of the GMP manuals dedicated an entire chapter to equipment calibration. However, none of the FDA GMP manuals addressed accuracy ratios or uncertainty considerations during calibration for in-or-out of tolerance decisions. The one exception to this appears merely as an example in a calibration procedure in the FDA guidance for mechanical measuring tools, which stated, “The calibration shall be done by a comparison to standard gage blocks… with an accuracy 3 to 10 times greater than that of the measuring tool.” See Appendix F for the 1997 version of the FDA GMP/QSR guidance on calibration. Note that this guidance was officially withdrawn by the FDA on December 12, 2013. There are currently no (known) official FDA guidelines on instrument calibration for compliance with 21 CFR §820.72. Meanwhile, private industry was publishing their own calibration guidance for medical device and pharmaceutical manufacturers. These groups were often comprised of individuals and experts with direct knowledge of metrology and calibration principles. The Association for the Advancement of Medical Instrumentation (AAMI) published the following in 1998 [73]: AAMI Quality System Compendium – GMP Requirements & Industry Practice (1998): [73] 2nd edition published in 2007; 3rd edition in 2015.

“Most manufacturers consider the necessary accuracy, precision, and resolution required for the measuring and test equipment during the establishment of their manufacturing and test procedures. Otherwise, the manufacturer may discover at a later date that the equipment is not suitable (i.e., not accurate or precise to provide reliable measurement/test results)… Generally, measuring equipment is at least four times, preferably ten times, more accurate than specified tolerances. Traceable standards are typically at least four times, preferably ten times, more accurate than the particular measuring or test equipment being calibrated. The ANSI/ASQC standard, ‘Calibration Systems’ [ANSI/ASQC M1-1987], discusses quantification of calibration errors and provides additional information on determining the degree of uncertainty of a calibration system to ensure that traceable standards are accurate enough for use in a specific calibration program”.

AAMI chose to cite the ANSI/ASQC M1-1987 standard “Calibration Systems” [74] while also referencing 4:1 and 10:1 accuracy ratios. Like AAMI, the FDA also made reference to ANSI/ASQC M1-1996 in a list of “useful standards” under the calibration section (line 1336) of the bibliography in the revised-draft guidance entitled, “Guidance for Industry: PAT – A Framework for Innovative Pharmaceutical Manufacturing and Quality Assurance” in 2003. However, this reference to the ANSI/ASQC M1 Calibration Standard was removed upon final publication of the PAT Guidance [75]. See Appendix B for treatment of accuracy ratios in ANSI/ASQC M1-1996.

2016 NCSL International Workshop & Symposium

Like the calibration guidance provided by AAMI, two other industry groups have published calibration guidance tailored specifically for the FDA-regulated pharmaceutical and medical device industry. These groups are the ISPE/GAMP (International Society of Pharmaceutical Engineers – Good Automated Manufacturing Practice) and the NCSL International Healthcare Metrology Committee. The ISPE/GAMP document is entitled, “A Risk Based Approach to Calibration Management” [76]. The NCSLI document is known as RP-6 and is entitled, “Recommended Practice: Calibration Quality Systems for the Healthcare Industries [77].” The ISPE GAMP document does not address accuracy ratios directly. However, it does provide guidance on how measurement uncertainty should be taken into account when making decisions regarding the in-or-out of tolerance status of calibration results. ISPE/GAMP Risk Based Approach to Calibration Management (2010): [76] 1st edition published in 2002

“5.5.5 Introduction to the Application of Measurement Uncertainty When measuring a critical parameter, measurement uncertainty should be understood, in order to evaluate the potential risk associated with obtaining either a false pass or a false fail… 5.5.7 Review of Third Party Calibration Certificates for Reference Standards Reference standards should be calibrated by a competent (appropriately qualified) organization… The calibration ranges, test points, and configuration should be suitable for the intended use. It also should show the errors at each test point, prior to and following any adjustment, along with a statement of the estimated measurement uncertainty. The certificate should be checked to determine whether residual errors are within agreed limits. This assessment should include consideration of the potential influence of measurement uncertainty when trying to determine pass or fail decisions (see Figure 5.3)... Given the ambiguity and inconsistency this can generate, it is considered good practice to have a policy on dealing with these various influence conditions. …the contract should state how the specification limits are selected and how the potential influence of measurement uncertainty will be treated. (Note: it is usually considered better to select larger tolerances than the manufacturer’s specification, where the application allows, as compliance with manufacturer’s specification may be difficult to achieve or demonstrate in the long term.) Measurement uncertainty can lead to the incorrect or false acceptance and rejection of calibration results. Terms which may be used by pharmaceutical organizations include: • False accept occurs when an instrument appears to pass specification limits when it actually fails: - Declaring an instrument to be within tolerance when it is out of tolerance has the potential for the instrument’s reading to cause product to be made outside the intended process range. • False reject occurs when an instrument appears to fail specifications when it actually passes: - False rejects may have a significant impact on business efficiency. For critical measurements, the calibration false accept rate should be considered.”

2016 NCSL International Workshop & Symposium

NCSLI Recommended Practice RP-6 addresses measurement decision risk and accuracy ratios as follows: NCSL International RP-6: Calibration Quality Systems for the Healthcare Industries (2015): [77] (1st edition 1986, 2nd edition 1999, 3rd edition 2008, 4th edition 2015)

“5.3 Adequacy of Measuring and Test Equipment (M&TE) Calibration tolerances should be based on the product/process requirements and limits. While the manufacturer’s specifications are often used as the assigned tolerance, a different (usually wider) tolerance is often adequate for the application. It the ratio between the product/process limits and the instrument tolerance is 4:1 or greater, the instrument tolerance is generally considered insignificant and no additional compensation for instrumental measurement uncertainty or accuracy is necessary. 5.4 Adequacy of Calibration Equipment and Standards The calibration quality system should ensure that the adequacy of calibration equipment is not compromised. This system should contain the following: Items should be calibrated with measurement systems, standards, and reference materials that have adequate accuracy, stability, and range to completely verify the performance of the calibrated item within its specified tolerance limits. Each organization should establish a program that includes measurement uncertainty, uncertainty ratios, accuracy ratios, false accepted risks, false rejected risks, or coverage factors to support the adequacy of its measurement system. 5.6.2 Measurement Uncertainty Measurement uncertainties should be calculated when… Test accuracy ratio parameters are below measurement quality thresholds There are significant uncertainty contributors that can affect the measurement result… 7. Appendix B. Adequacy of M&TE for Intended Use Decision Guideline… Depending upon the criticality of the product/process, the PAR [Process Accuracy Ratio] requirement may vary. Typically, a PAR of 4:1 or greater is considered sufficient to minimize measurement risk (e.g. false accept risk or consumer risk). However, the correct level of acceptable PAR and/or risk should be determined and documented by the organization. NOTE: Other methods of controlling measurement decision risk, such as guardbanding may also be used”.

2016 NCSL International Workshop & Symposium

An overview of the NCSLI Healthcare Metrology Committee that published RP-6 has been given by McNeely [78]. M. McNeely, NCSLI (2013): [78] “4. Goals of the 151 Healthcare Metrology Committee 4.1. Work with the US FDA on recognition of 17025 (ILAC MRA) accredited calibration laboratories. 4.1.1. This effort aspires to create a guidance document for abbreviated inspection of such laboratories, by providing greater efficiency for documentation and lab qualification, however, it does not advocate in any way the mandated use of such laboratories. It is stressed to be promoted as a voluntary guidance document. 4.1.2. If a calibration laboratory was accredited to ISO/IEC 17025 for the applicable calibration discipline by a mutually recognized accreditation body (AB), and the scope of uncertainty was considered acceptable by the contracting party to ultimately support its process, the calibration would be considered traceable to national or international standards under the scope of such accreditation, and the scope of accreditation would then be considered acceptable documentation of the unbroken chain of traceability to satisfy the requirements stated in 21 CFR 820.72(1). 4.1.3. As of May, 2013, the 151 awaits formal notification from The US FDA on the ultimate status of this proposed draft guidance document.”

As stated by McNeely, the NCSL International 151 Healthcare Metrology Committee has petitioned the FDA to create a new draft guidance document that would allow minimized FDA assessments of calibration laboratories that are accredited to ISO 17025. This would be somewhat similar to the course taken by NRC, which now readily accepts calibrations from accredited laboratories for commercial third party calibrations (see section 6.2). Such an FDA guidance document may facilitate an abbreviated inspection of calibration laboratories that voluntarily chose to become accredited. The request by the NCSLI 151 Committee was submitted to the Dockets Management Branch (HFA-305) of the FDA on October 11, 2011. See Appendix C. Such activities are consistent with 21 CFR §10.115 [79] governing FDA Good Guidance Practices and the participation in their development; see Appendix D. As of April 2016, the NCSLI 151 Committee awaits formal notification from the FDA on the status of this proposed draft guidance document. Similar advocacy regarding the recognition of national and international calibration standards was previously submitted to the FDA in 1998 by James [80] in Docket No. 97N-0477. James, Comments Submitted to FDA; Docket No. 97N-0477 (1998): [80] “All calibration systems that assure the integrity of medical measurements should employ methods similar to those set forth in national and international standards such as IS0 GUIDE 25, ANSI/NCSL Z540-1, NASA 5300.4, or U.S. DOD MIL-STD-45662A. These standards require periodic intervals and methods be established to maintain acceptable accuracy and measurement reliability. Measurement reliability [End of Period Reliability or EOPR] is defined as: the probability that the equipment under test and the measurement standard will remain in-tolerance throughout the established interval. This kind of system is designed to be both effective and efficient at addressing the needs of uncertainty growth.”

2016 NCSL International Workshop & Symposium

Another group that has published calibration guidance for the healthcare industry is The Global Harmonization Task Force (GHTF). However, as the 1997 FDA Quality System Regulation in 21 CFR had been harmonized with ISO-13485 and ISO 9001, the GHTF calibration guidance [81] largely addresses the basic calibration requirements in section 4.11 of ISO 9001:1994 and section 13 of ISO 9004-1:1994. GHTF.SG3.N99-8 Quality Systems For The Design And Manufacture Of Medical Devices (1999): [81] Published (1999) – Archived June 21, 2005. Note: The GTHF participated in ISO/TR 14969:2004(E), which effectively replaced the GHTF guidance here.

“4.11 Control of inspection, measuring and test equipment 4.11.1 General 4.11.2 Control procedure The requirements of this clause in ISO 9001 spell out in considerable detail what is to be implemented. Although the requirements pertain explicitly to inspection, measuring and test equipment, it is helpful to approach the subject from the perspective that measuring is itself a process involving raw materials, equipment and procedures. The requirements of ISO 9001 explicitly involve elements of the measurement process; elements whose collective purpose is to choose suitable measurements, suitable measuring equipment, and suitable measurement procedures. These elements are specified to provide confidence in the ability of the supplier's measuring systems to control adequately the production and inspection of the product. For both product- and process-measurement systems, statistical methods are valuable tools for achieving and demonstrating fulfilment of requirements. In particular, statistical methods are the preferred tools in fulfilling the requirement that ‘Inspection, measuring and test equipment shall be used in a manner which ensures that the measurement uncertainty is known and is consistent with the required measurement capability’. The requirements of this clause also should be applied by the supplier insofar as ‘demonstrating the conformance of product to the specified requirements’ contractually involves measurements subsequent to production and inspection of a product (e.g. during subsequent handling, storage, packaging, delivery or servicing) as may be required under other clauses of ISO 9001. Note: See also ISO 10012-1. See 13 of ISO 9004-1.”

Many entities regulated by the FDA voluntarily choose to be certified to ISO 13485:2016 [82] by a registrar. The International Accreditation Forum (IAF) has published an overview of recent enhancements to this certification process [82A]. Registration/certification to ISO 13485 is not specifically required by the FDA. However, a new Medical Device Single Audit Program (MDSAP) is currently in the final stages of a pilot program. Under this pilot program, FDA regulated entities can voluntarily choose to be audited, not by the FDA, but by a 3rd party auditing organization. Auditing organizations must be approved by the International Medical Device Regulators Forum (IMDRF), of which the FDA is a member of, and which evolved out of the previous GHTF. The MDSAP will take the place of routine FDA audits and will use ISO 13485 as its primary requirements document. The calibration requirements for control of monitoring and measuring devices in ISO 13485:2003 are given below. Note that no mention of test accuracy ratios, uncertainty, or measurement decision risk is provided.

2016 NCSL International Workshop & Symposium

ISO 13485:2003: Medical Devices – Quality Management Systems – Requirements for Regulatory Purposes (2003) [82] “7.6 Control of Monitoring and Measuring Devices The organization shall determine the monitoring and measurement to be undertaken and the monitoring and measuring devices needed to provide evidence of conformity of product to determined requirements (see 7.2.1). The organization shall establish documented procedures to ensure that monitoring and measurement can be carried out and are carried out in a manner that is consistent with the monitoring and measurement requirements. Where necessary to ensure valid results, measuring equipment shall a) Be calibrated or verified at specified intervals, or prior to use, against measurement standards traceable to international or national standards; where no such standards exist, the basis used for calibration or verification shall be recorded; b) Be adjusted or re-adjusted as necessary; c) Be identified to enable the calibration status to be determined; d) Be safeguarded from adjustments that would invalidate the measurement result; e) Be protected from damage and deterioration during handling, maintenance and storage. In addition, the organization shall assess and record the validity of the previous measuring results when the equipment is found not to conform to requirements. The organization shall take appropriate action on the equipment and any product affected. Records of the results of calibration and verification shall be maintained (see 4.2.4). When used in the monitoring and measurement of specified requirements, the ability of computer software to satisfy the intended application shall be confirmed. This shall be undertaken prior to initial use and reconfirmed as necessary. NOTE See ISO 10012 for guidance related to measurement management systems.”

As stated, the IMDRF’s Medical Device Single Audit Program will use ISO 13485 as the core framework for auditing purposes. The IMDRF recently published an MDSAP Companion Document [83], which contains guidance on auditing calibration requirements, to be used by approved auditing organizations. Although the concepts of test accuracy ratio and measurement decision risk are not directly discussed in the guidance, the primary requirement is the ubiquitous clause, “…suitable for its intended use, and capable of giving valid results.” The IMDRF guidance is provided below.

2016 NCSL International Workshop & Symposium

IMDRF Medical Device Single Audit Program (MDSAP) Companion Document (2014): [83] “13. Confirm that the organization has determined the monitoring and measuring devices needed to provide evidence of conformity to specified requirements. Verify that the monitoring and measuring equipment used in production and service control has been identified, adjusted, calibrated and maintained, and capable of producing valid results. Clause and regulation: [ISO 13485:2003: 7.5.1.1, 7.6; TG(MD)R Sch3 P1 1.4(5)(e); RDC ANVISA 16/2013: 5.1.5, 5.4; 21 CFR 820.70(g), 820.72] Additional country-specific requirements: None Assessing conformity: Maintenance and calibration While reviewing the selected production process, make note of significant pieces of process equipment and significant pieces of measuring or test equipment. Consider selecting process and test equipment that, if not properly controlled, could cause devices to not meet specified requirements; or produce inaccurate results that could lead to unrecognized nonconformities. Confirm that the production and test equipment selected for review is suitable for its intended purpose and capable of giving valid results. Review the maintenance, control, and calibration procedures (and records) for the equipment selected for review. The initial frequency with which measuring and test equipment is calibrated and maintained is usually based on the equipment manufacturer’s recommendations. As the organization gains experience with the piece of equipment, the frequency of calibration and maintenance may be adjusted, based on a documented rationale. Accuracy and precision When accuracy and precision is a factor in the validity of the result of the measuring equipment, the required accuracy and precision should be defined during the planning of product realization to ensure the equipment is suitable and capable of providing valid results Reviewing records If production equipment or test equipment is found to be outside of its maintenance or calibration requirements, verify that the organization made an assessment of the effect of the out-of-tolerance situation on in-process, finished, or released devices, based on risk. Equipment adjustment, calibration, and maintenance procedures and records may provide insight into nonconformities. Review these procedures and records to determine whether inadequate procedures or the organization’s failure to comply with adequate procedures contributed to the nonconformity. For example, determine whether the lack of specified equipment adjustment or maintenance contributed to the production of nonconforming product.”

Much like the FDA’s database on recognized consensus standards [84] [85] [86] , ISO 17025 is not currently listed as a “recognized standard” by the IMDRF N15 document, “Final Report: List of International Standards Recognized by IMDRF Management Committee Members as of March 2014” [87] [88]. As previously stated, the older 1999 GHTF SG3.N99-8 guidance document was archived in 2005 and has been essentially replaced by ISO/TR 14969:2004 [89]. While again not mentioning test accuracy ratios or measurement decision risk, the ISO Technical Report 14969 does indicate that, “Statistical methods are important in showing which monitoring and measuring devices are used in a manner which ensures that the measurement uncertainty is known and is consistent with the required measurement capability.”

2016 NCSL International Workshop & Symposium

ISO/TR 14969:2004(E) Technical Report: Medical Devices – Quality Management Systems – Guidance on the Application of ISO 13485:2003 (2004): [89] “7.6.1 The requirements refer explicitly to monitoring and measuring devices, including test software. It is helpful to approach the subject of control of monitoring and measuring devices from the perspective that measuring is itself a process involving materials, equipment and procedures. The intent of the requirements is to give the organization confidence in the monitoring and measuring devices that it uses to ensure that product meets customer and regulatory requirements. Statistical methods are important in showing which monitoring and measuring devices are used in a manner which ensures that the measurement uncertainty is known and is consistent with the required measurement capability. The requirements of this subclause are also applied by the organization when demonstrating the conformity of product to the specified requirements. This can involve measurements subsequent to production and inspection of product (e.g. during handling, storage, packaging, preservation, delivery or servicing). Documented procedures should include details of equipment type, unique identification, location, frequency of checks, check method and acceptance criteria… NOTE Additional information regarding the management of monitoring and measuring equipment is available in ISO 10012.”

The calibration requirements in ISO 13485 may not be sufficiently prescriptive to adequately minimize or control incorrect decisions due to measurement uncertainty or to provide guidance on instrument suitability. In the healthcare industry, falsely accepting a measuring device during calibration (i.e., declaring it as In-Tolerance, when it is actually Out-of-Tolerance, due to measurement uncertainty) can have serious consequences. Accuracy is of paramount importance for many instruments, not just those used during calibration and for the manufacture of medical devices, but for the actual devices themselves. Grim [90] published the following conclusions in 2002 regarding measurement error and uncertainty in blood pressure measuring instruments: Grim, Summary Report – NHBPEP / NHLBI / AHA National Institutes of Health (2002): [90] “Unless blood pressure (BP) is measured accurately, the proven benefits of diagnosing and treating an unhealthy blood pressure will not be transferred to the population. …errors as small as 2-5 mmHg can have astounding costs to the individual patient, to the health care system, to a research project, for government planning and for society. …an error of ‘only’ -5 mmHg at the 90-95 mmHg range will miss the 21 million US hypertensives in the US in this range (42 % of all with HTN). Over the next 6 years, these 21 million untreated HTN will experience 125,000 CAD [coronary artery disease] deaths of which at least 20 % would have been prevented by treatment. About the same number of fatal strokes would have also been prevented. Thus a -5 mmHg error will cause about 50,000 preventable deaths not to mention preventing perhaps twice this many non-fatal CAD and CVAs [cerebrovascular accident]. Aneroid devices out of calibration most often read too low. Measuring BP falsely high increases costs by treating those who do not truly have high BP. Thus an error of +5 mmHg would move 27 million people from 85-89 [mmHg] into the high BP range. As the estimated cost of treating one person for high BP is $1000/yr, this will cost $27 billion/yr to treat a ‘non-disease’. Even a 2 mmHg error will misclassify about 6 million persons into the 90–95 range. Current standards permit devices ±3 mm of the mercury standard. This seems too lenient.”

2016 NCSL International Workshop & Symposium

Interestingly, Lozy [91] had published a risk analysis method for the misclassification of hypertension diagnosis due to measurement error in 1982. Lozy had applied the same mathematical false accept/reject or misclassification technique (bivariate normal joint probability density function) that Hayes had applied to incorrect calibration decisions in 1955 [A5]. Other researchers such as Friedman-Gerlicz and Lilly [91A] have also employed this mathematical model to quantify misclassification rates in hypertension diagnosis due to measurement errors. Indeed, this technique is now commonly employed to model global consumer risk and producer risk anytime pass/fail decisions (or diagnoses) are based on measurements which are subject to error or uncertainty (see Appendix A). Greaves [92] when citing Forsman [93] states that 60 % to 70 % of medical decisions are based on test results. Thus, the influence of measurement uncertainty on incorrect decisions extends beyond calibration and into all areas of measurement, including manufacturing processes, analytical chemistry, diagnoses of medical conditions, etc. Kuselman et-al provided the following in 2003 [94] and Stamm in 2013 [A143A]. Kuselman et-al (2003): [94] “Traceability and uncertainty of measurement results are basic technical elements of quality systems in analytical laboratories whose competence is recognized by accreditation according to ISO/IEC 17025. However, GLP and GMP standards widely used since the 1960s for the quality system assessment in pharmaceutical industry do not include requirements for the measurement uncertainty evaluation. It is clear today that the issue discussed above should be taken into account by pharmacopoeial committees and regulatory bodies involved in drug quality assurance for harmonization of requirements to analytical results and improvement of their quality…” Stamm (2013): [A143A] …FDA regulations…require calibration of instrumentation and gauging used in the manufacture of pharmaceuticals and medical devices and also require limits for accuracy and precision of these devices... However, they do not specify a method for determining accuracy and precision estimates… While the FDA does specify calibration for measurement instrumentation… the requirement stops short of requiring an estimation of uncertainty in the measurement results…

The preceding review of the standards and guidelines employed by the healthcare industry to maintain and calibrate equipment may appear rather stringent. However, a recent article by Simpson [95] entitled, “FDA Standards on Calibrating Lab Equipment Are Not So Tough” highlights the basic FDA calibration requirements found in 21 CFR §820.72. These requirements have remained largely unchanged since the 1978 version of 21 CFR §820.61 which, as stated previously, were based to some degree on calibration requirements from MIL-C45662 published in 1960. One significant change did occur when the GMP’s were revised in 1997; a requirement was added to investigate product impact or “adverse effects” resulting from out-of-tolerance instruments (see Table 2). This same impact assessment requirement had also been added to MIL-STD-45662 in 1980, along with the addition of the 4:1 TAR requirement in MIL-STD-45662A in 1988. However, the revised 1997 GMP’s in 21 CFR §820.72 contained no accuracy or uncertainty ratio requirement. The basic FDA calibration requirements in 21 CFR §820.72 are similar to those found in ISO 9001 and ISO 13485. They are not as prescriptive as the modern requirements found in ISO 17025 or Z540.3 which many industries, both regulated and unregulated, have adopted or employ as guidance documents. Refer to Table 1 for a historical context of calibration documents and requirements and guidelines that have evolved over several decades to prescribe technical requirements for many critical calibration programs. 2016 NCSL International Workshop & Symposium

Without agreement on the use national and/or international standards (even as guidance documents), the very definitions in the text of the CFR may be ill-defined or subject to interpretation. Despite established metrological definitions in Z540.3, ISO 17025, and other related documents, there have been recent accounts where entities regulated by the FDA have received Form-483 Investigational Observations and Warning Letters arising from the failure to always adjust instruments during calibration, whether out-of-tolerance or not [96]. These incidents may be attributable to a nebulous distinction between the definitions of calibration, verification, and adjustment – which are well defined terms outside of the CFR [97] [98]. References to similar events in regulated industries have also been published where the definition of calibration has been inferred to mandate adjustment during calibration [99] [99A] [100] [101]. Without agreement on standards and guidance documents for calibration, similar events may again manifest – where the definition of other metrological terms in the CFR, such as traceability, may become the subject of future regulatory contention. Recognition of ISO 17025 and Z540.3 as guidance documents could provide a common framework for all parties to reference in such instances. An article in 2010 by Schmitt [102] provides an interview with Trautman – at the time, the FDA Associate Director of International Affairs and author of the 1997 (current) 21 CFR 820 Quality System Regulation. This article deals directly with FDA calibration expectations, particularly those required of 3rd party calibration laboratories (outsourced “test houses”). Since little exists in the public domain regarding official FDA guidance on calibration, the article by Schmitt becomes highly relevant. Owing to this, the article is reproduced in its entirety, with the author’s permission, in Appendix E of this paper. Inherent to the theme of the article is that the FDA calibration requirements are very stringent – so stringent, that care is advised when selecting outsourced third-party calibration vendors, because they may not meet rigorous FDA requirements. Cautions are given in the article that such calibration laboratories often service other “unregulated” industries – the apparent implication being that some of these other industries may not have such strict calibration requirements. 6. Calibration in Other Government/Regulated Industries The preceding inferences from the Schmitt article compare the calibration requirements of the FDA with those of other unregulated industries. However, there are a plethora of both government and non-government organizations that require calibrations to be performed in compliance with, if not accredited to, ISO 17025 and/or ANSI Z540.3. By contrast, compliance with such standards is not an FDA requirement of the calibration function of entities regulated by the FDA, nor have these documents been cited as official guidance documents by the FDA for calibration. A comparative review of some other federal government agencies is provided here.

2016 NCSL International Workshop & Symposium

6.1 NASA – National Aeronautics and Space Administration Historical NASA calibration and metrology documents are provided in Table 1. NASA’s current policy directive on metrology and calibration [103] states that: NASA Policy Directive NPD 8730.1C (2011): [103] “a. It

is NASA policy to…

(3) Control the accuracy, reliability, and use of Measuring and Test Equipment (MTE) through the use of a calibration system compliant with the requirements of American National Standards Institute/National Conference of Standards Laboratories (ANSI/NCSL) Z540.3-2006 and applicable requirements of Society of Automotive Engineers (SAE) AS9100, subject to the clarifications and modifications provided in Attachment B of this NASA Policy Directive (NPD). ATTACHMENT B: ANSI/NCSL Z540.3 Clarifications and Modifications… B.3. End-of-period-reliability (EOPR) values equal to or greater than 89 percent are considered acceptable evidence of compliance to ANSI/NCSL Z540.3 subclause 5.3.b, false acceptance requirements, and subclause 5.3.3, measurement uncertainty requirements, where such values are derived from statistically significant empirical data… B.5. Guidance concerning estimation and evaluation of measurement decision risk, including false accept risk, is provided in NASA-HDBK 8739.19-4, Estimation and Evaluation of Measurement Decision Risk, and in the NCSLI Handbook for the Application of ANSI/NCSL Z540.3-2006, Requirements for the Calibration of Measuring and Test Equipment. B.6. Guidance and methods for controlling the accuracy of measurement results are provided in the NASAHDBK 8739.19 series of measurement handbooks. The handbooks are found at http://www.hq.nasa.gov/office/codeq/doctree/hdbk873919.htm. B.7. Compliance to ANSI/NCSL Z540.3-2006 is considered to meet or exceed the requirements of ANSI/NCSL Z540.1-1994 (R2002)”.

Thus, Z540.3 is listed as a mandatory compliance document in NASA contract solicitations (Request for Proposals or RFP’s) for calibration programs. This is evidenced under the ISC contract at the Kennedy Space Center [104], the CAMS contract at Johnson Space Center [105], and the TIALS contract at Glenn Research Center [106]. The Measurement Standards and Calibration Laboratory (MSCL) at Johnson Space Center has been accredited by ACLASS (now, ANAB) to both ISO 17025 and Z540.3 [107]. Accreditation, per se, is not required by NASA policy, but compliance with the requirements of the Z540.3 standard is a requirement. Compliance is assessed via the NASA Metrology and Calibration Program Office, under the Office of Safety and Mission Assurance (OSMA).

2016 NCSL International Workshop & Symposium

6.2 DOE/NRC – Department of Energy and the Nuclear Regulatory Commission Some federal agencies may not specifically require all calibrations to be performed in strict compliance with national or international standards. However, many of these agencies still recognize, and will readily accept, calibrations which have been performed by an accredited laboratory. Department of Energy (DoE) laboratories have been accepting accredited calibrations from outside commercial suppliers for well over a decade now [108]. See Figure 3. Many of the DoE calibration laboratories have themselves, attained accreditation. In 1998, Pettit [109] published the following account: Pettit – Sandia Primary Standards Laboratory (1998): [109] “Accreditation can offer many benefits to a testing or calibration laboratory, including increased marketability of services, reduced number of outside assessments, and improved quality of services. Compared to IS0 9000 registration, the accreditation process includes a review of the entire quality system, but in addition a review of testing or calibration procedures by a technical expert and participation in proficiency testing in the areas of accreditation. Within the DOE, several facilities have recently become accredited in the area of calibration, including Sandia National Laboratories, Oak Ridge, AlliedSignal FM&T; Lockheed Martin Idaho Tech. Co., and Pacific Northwest National Lab. Recently, representatives from these and other DOE facilities throughout the U.S. formed a DOE Accreditation Committee under the sponsorship of the DOE Technical Standards Program. This committee will hold its first open meeting on September 23-24, 1998 at the National Institute of Standards and Technology in Gaithersburg, MD with the Goal of developing procedures for sharing and coordinating information within DOE on accreditation issues.”

2016 NCSL International Workshop & Symposium

Figure 3. Department of Energy approves the use of accredited calibration laboratories (used w/ permission) 2016 NCSL International Workshop & Symposium

The Nuclear Regulatory Commission (NRC) is an associated federal agency whose calibration requirements have been codified into law via the CFR for over 45 years. 10 CFR 50 Appendix B, Requirement XII. (1970 - present): [10] “Measures shall be established to assure that tools, gages, instruments, and other measuring and testing devices used in activities affecting quality are properly controlled, calibrated, and adjusted at specified periods to maintain accuracy within necessary limits.”

For many years the NRC’s calibration requirements, specified by this “Basic Requirement 12” in Appendix B of 10 CFR §50, were augmented by NQA-1 supplementary requirements [110] [111]. ANSI/ASME NQA-1 Supplement 12 S-1, Supplementary Requirements for Control of Measuring and Test Equipment (1986): [110], [111] 1. General: This Supplement provides amplified requirements for control of measuring and test equipment. It supplements the requirements of Basic Requirement 12 of this Standard and shall be used in conjunction with that Basic Requirement when and to the extent specified by the organization invoking this Standard. 2. Selection: Selection of Measuring and Test Equipment shall be controlled to assure that such items are of proper type, range, accuracy, and tolerance to accomplish the function of determining conformance to specified requirements. 3. Calibration and Control 3.1 Calibration: Measuring and test equipment shall be calibrated, adjusted and maintained at prescribed intervals or, prior to use, against certified equipment having known valid relationships to nationally recognized standards. If no nationally recognized standards exist, the basis for calibration shall be documented. 3.2

Control: The method and interval of calibration for each item shall be defined, based on the type of equipment stability characteristics, required accuracy, intended use, and other conditions affecting measurement control. When measuring and test equipment is found to be out of calibration, an evaluation shall be made and documented of the validity of previously inspected or tested. Out-of-calibration devices shall be tagged or segregated and not used until they have been recalibrated. If any measuring or test equipment is consistently found to be out of calibration, it shall be repaired or replaced. A calibration shall be performed when the accuracy of the equipment is suspect.

3.3

Commercial Devices: Calibration and control measures may not be required for rulers, tape measures, levels, and other such devices, if normal commercial equipment provides adequate accuracy.

4. Handling and Storage: Measuring and test equipment shall be properly handled and stored to maintain accuracy. 5. Records: Records shall be maintained and equipment shall be suitably marked to indicate calibration status.

2016 NCSL International Workshop & Symposium

Various revisions of IEEE STD 498 over the years (1975, 1980, 1985, 1990) [112] [113] [114] [115], also specified calibration requirements/guidance for NRC facilities, as stated in NRC Inspection Manual, Inspection Procedure 35750 [116]. NRC Inspection Manual: Procedure 35750 (1992): [116] “General Guidance: The regulatory basis for control of M&TE is found in 10 CFR 50, Appendix B, Criterion XII. The method by which the licensee will comply with this requirement is normally described in Chapter 17 of the FSAR [Final Safety Analysis Report]. IEEE 498-1980, ‘IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in Nuclear Facilities’ provides guidance for establishing a program for the control and verification of the accuracy of M&TE. Even though a regulatory guide does not currently exist to endorse IEEE Standard 498, this inspection procedure was written using the guidelines of IEEE Standard 498 because the majority of the guidance is generic in nature.”

IEEE STD 498 was withdrawn in July of 1995. See Appendix B for the requirements of IEEE STD 498, as they related to test uncertainty ratio to mitigate incorrect calibration decisions, and how these requirements evolved over two decades. For a number of years, the nuclear industry has also relied on the Nuclear Procurement Issues Committee (NUPIC) Document No. 28 and No. 29 for the procurement of 3rd party supplier “commercial-grade” calibrations. Document No. 28 is the Commercial Grade Calibration Service Checklist [117] and Document No. 29 is the Commercial Grade Calibration Services Checklist Implementation Guidelines [118]. An endeavor of the NRC to use and accept commercial calibrations from laboratories accredited to ISO 17025, in lieu of surveys/audits, was documented in NISTIR 6989 in 2003 [119]. The abstract of this report is provided here. NIST IR 6989 (2003): [119] “Representatives of the Nuclear Utilities Industry and the laboratory accreditation community met to discuss Nuclear Regulatory Commission (NRC) recognition of the use of accredited commercial calibration laboratories as suppliers of commercial grade calibration services to the regulated utilities. The purpose of the meeting was to map out a strategy for approaching the NRC concerning the issue of calibration laboratory accreditation and the National Cooperation for Laboratory Accreditation (NACLA). The intent is to gain NRC endorsement of laboratory accreditation based on ISO/IEC 17025 as a means to qualify calibration service suppliers for nuclear work. In support of this effort, a comparison between the requirements of ISO/IEC 17025: General requirements for the competence of testing and calibration laboratories and the audit requirements of the Nuclear Utilities Procurement Issues Committee (NUPIC) Commercial Grade Survey Checklist for Calibration Services was prepared.”

2016 NCSL International Workshop & Symposium

The Bellefonte Nuclear Generating Station (BLN), regulated by the NRC, states: NRC – Bellefonte (BLN) Nuclear Generating Station – Combined License (COL) – Final Safety Analysis Report (FSAR) – Chapter 17, Quality Assurance, Section 17.5.4.7 (2009): [120] …As discussed in the QAPD, the applicant also remains responsible for ensuring that the items or services are suitable for the intended application and for documenting the evaluation that supports this conclusion... • SRP Section 17.5 paragraph II.L.8 establishes provisions for the procurement of commercial-grade calibration services for safety-related applications. As an exception to these provisions, the QAPD proposes that procurement source evaluations and selection measures not be required, provided that all of the following conditions are met… -Purchase documents require reporting as-found calibration data when calibrated items are found to be out of tolerance. -A documented review of the supplier’s accreditation will be performed and will include a verification of the following: •The calibration laboratory holds a domestic accreditation by any one of the following accrediting bodies, which are recognized by the International Laboratory Accreditation Cooperation (ILAC) Mutual Recognition Arrangement (MRA): -NVLAP administered by NIST -American Association for Laboratory Accreditation (A2LA), -ACLASS Accreditation Services (ACLASS), -International Accreditation Service (IAS), -Laboratory Accreditation Bureau (L-A-B). •The accreditation encompasses ANS/ISO/IEC 17025, ‘General Requirements for the Competence of Testing and Calibration Laboratories.’ •The published scope of accreditation for the calibration laboratory covers the necessary measurement parameters, ranges, and uncertainties. NRC staff evaluated and found to be acceptable the NVLAP and A2LA accreditation programs (Ref. 17.5-6). The staff subsequently determined that the accreditation programs of ACLASS, L-A-B, and IAS are also recognized by the ILAC MRA and are therefore acceptable (Ref. 7, 8, and 9).

In addition to acceptance of NVLAP and A2LA accredited calibrations, the NRC has authored letters addressed to IAS [121], L-A-B [122], ACLASS/ANAB [123], and PJLA [124], attesting to the acceptance of calibrations from laboratories accredited by these AB’s, which are all signatories to the ILAC MRA or Mutual Recognition Arrangement [125] [126] [127] [128]. Reports in 2010 had conveyed some confusion on behalf of calibration suppliers as to the NRC’s requirements [129]. However, the most recent status of the NRC’s program of accepting accredited commercial calibrations is found in the March 2015 version of the Nuclear Energy Institute’s NEI 14-05A, “Guidelines for the Use of Accreditation in Lieu of Commercial Grade Surveys for Procurement of Laboratory Calibration and Test Services [130]”. A chronology of some historical standards and guidelines, prescribing the DOE & NRC requirements for the calibration and control of measuring and test equipment throughout history, is given in Table 3. 2016 NCSL International Workshop & Symposium

Table 3A. DOE and NRC Calibration Standards, Guidelines, and Requirements YEAR DOCUMENT NUMBER 1969 RDT F3-2T 1970 AEC 10 CFR 50 Appendix B, XII 1971 ANSI N45.2 1971 1972

AEC/NRC Regulatory Guide 1.28

1973

1976

RDT F3-2T IEEE STD 498-1975 ANSI N45.2.16 ANSI N18.7-1976 [ANS 3.2]

1977

NRC Regulatory Guide 1.33

1977

ANSI/ASME N45.2.16

1978 1979 1979 1979

ANSI N46.2-1978 ANSI/ASME NQA-1-1979 CONST-QAPP 12 Regulatory Guide 1.28 (Rev.2)

1980

IEEE 498-1980

1981 1981 1981 1981

DOE O 5700.6

1982

NUREG/CR-1369, Rev.1

1983

ANSI/ASME NQA-1-1983

1983 1985

ASME NQA-2-1983 Regulatory Guide 1.28 (Rev.3)

1985

ANSI/IEEE 498-1985

1986

DOE O 5700.6B

1986

ASME NQA-1-1986

1989 1989 1990 1990 1991 1991

ASME NQA-1-1989 ASME NQA-3-1989 NUREG-0800 (SRP 17.3) DOE 4330.4A DOE O 5700.6C NCSL RP-10 (Rev.1)

1991

IEEE 498-1990

1975

DOE O 5700.6A NUREG-0800 (SRP 17.1) NUREG-0800 (SRP 17.2)

DOCUMENT OR SECTION TITLE FOR M&TE CALIBRATION Calibration System Requirements [AEC: Atomic Energy Commission] Control of Measuring and Test Equipment [Basic Requirement 12] Section 13: Control of Measuring and Test Equipment Safety Guide 28: Quality Assurance Program Requirements (Design and Construction) [Endorsed ANSI N45.21971, including Section 13 on Control of Measuring and Test Equipment] Calibration System Requirements [AEC: Atomic Energy Commission – Feb 1973] IEEE Standard Supplementary Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations (used with ANSI N45.2) Administrative Controls and Quality Assurance for the Operational Phase of Nuclear Power Plants. Section 8: Procedures for Control of Measuring and Test Equipment for Surveillance Tests, Procedures, and Calibrations. Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and Maintenance of Nuclear Facility [Endorsed ANSI N18.7-1976/ANS-3.2] Quality Assurance Program Requirements for Post Reactor Nuclear Fuel Cycle Facilities Basic Requirement 12, Control of Measuring and Test Equipment Quality Assurance Project Plan: Control of Measuring and Test Equipment Endorsed ANSI/ASME N45.2-1977 with supplemental provisions IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations Quality Assurance [Note: No mention of calibration requirements] Quality Assurance; Attachment 1, Basic Premise 3 – calibration Standard Review Plan, section 17.1.12 Control of Measuring and Test Equipment Standard Review Plan, section 17.2.12, Activities related to Control of Measuring and Test Equipment Procedures Evaluation Checklist for Maintenance, Test and Calibration Procedures Used in Nuclear Power Plants Requirement No. 12, Control of Measuring and Test Equipment Supplement 12 S-1, Supplementary Requirements for Control of Measuring and Test Equipment Quality Assurance Requirements for Nuclear Facility Applications [Endorsed ASME NQA-1-1983 via NQA-1a-1983 addenda] IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in Nuclear Facilities [title change from 1980 version] Quality Assurance Nuclear Quality Assurance. Requirement No. 12, Control of Measuring and Test Equipment Supplement 12 S-1, Supplementary Requirements for Control of Measuring and Test Equipment Nuclear Quality Assurance. Requirement No. 12, Control of Measuring and Test Equipment Basic Requirement 12, Control of Measuring and Test Equipment Standard Review Plan, Section 17.3.B.9, Measuring and Test Equipment Control Maintenance Management Program, Control and Calibration of Measuring and Test Equipment Quality Assurance; section 9.b.2.a “Criterion 5” and section 9.b.2.d “Criterion 8” Establishment and Operation of an Electrical Utility Metrology Laboratory IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in Nuclear Facilities

2016 NCSL International Workshop & Symposium

Table 3B. DOE and NRC Calibration Standards, Guidelines, and Requirements YEAR DOCUMENT NUMBER 1992 NRC Inspection Manual 1993

DOE-STD-1054-93

1994

ASME NQA-1-1994

1994 1994 1996 1997 1998 1999 2000 2000 2001 2001 2003 2004 2004 2005 2005

DOE RW-0333P (Rev. 1) DOE 4330.4B DOE O 430.1 ASME NQA-1-1997 DOE O 414.1 DOE O 414.1A ASME NQA-1-2000 NUPIC Document 28 (Rev.1) NUPIC Document 29 (Rev.1) DOE O 433.1 DOE G 433.1-1 NISTIR 6989 ASME NQA-1-2004 NCSL RP-10, Rev.2 DOE O 414.1B DOE O 414.1C

2006

IAEA GS-R-3:2006

2006

IAEA GS-G-3.1:2006

2006

NUPIC Document 29 (Rev.2)

2007

NUREG-0800 (SRP 17.5)

2007

DOE O 433.1A

2007

IAEA DS 349:2007

Undated

2008 2010 2010 2011

ASME NQA-1-2008 w/NQA-1a2009 Addenda DOE RW-0333P (Rev.20) DOE O 433.1B Regulatory Guide 1.28 (Rev.4) DOE O 414.1D

2012

ASME NQA-1-2012

2012 2015 2015

ANSI/ANS 3.2-2012 NUREG-0800 (SRP 17.5) ASME NQA-1-2015

2015

NEI 14-05A

2008

DOCUMENT OR SECTION TITLE FOR M&TE CALIBRATION Inspection Procedure 35750, QA Program – Measuring and Test Equipment Guideline to Good Practices for Control and Calibration of M&TE at DOE Nuclear Facilities [Replaced in 2001 by DOE G 433.1-1, section 4.11, Control and Calibration of Measuring and Test Equipment] Basic Requirement 12, Control of Measuring and Test Equipment [Incorporated NQA-2 & NQA-3. Subpart 2.16 endorses IEEE 498-1985] Quality Assurance Requirements and Description. Section 12.0, Control of Measuring and Test Equipment Section 3.5.4 and Chapter 12 –Control and Calibration of Measuring and Test Equipment Cancels DOE 4330.4B – Maintenance Management Program. Basic Requirement 12, Control of Measuring and Test Equipment Quality Assurance: Section 4.b.2.a.4 “Criterion 5” and 4.b.2.d.2. “Criterion 8” [Supersedes 5700.6C-1991] Quality Assurance: Section 4.b.2.a.4 “Criterion 5” and 4.b.2.d.2. “Criterion 8” Basic Requirement 12, Control of Measuring and Test Equipment Commercial Grade Calibration Services Checklist [Publication date & revision history unknown] Commercial Grade Calibration Services Implementation Guidelines Attachment 1, section 3(k), Control and Calibration of Measuring and Test Equipment Section 4.11, Control and Calibration of Measuring and Test Equipment [Replaces DOE-STD-1054-93] Comparison of ISO/IEC 17025 with the NUPIC Audit Checklist (w/ 2007 Addendum) Establishment and Operation of an Electrical Utility Metrology Laboratory Section 4.a.3, 4.b.5.d “Criterion 5” and 4.b.8.b “Criterion 8” Section 4.a.3, 4.b.5.d “Criterion 5” and 4.b.8.b “Criterion 8” Management System for Facilities and Activities: Safety Requirements. Topic 5.15 [activities for inspection, testing, verification and validation] Application of the Management System for Facilities and Activities: Safety Guide. Topic 5 [Control of Products] Commercial Grade Calibration Services Implementation Guidelines Standard Review Plan, Section 17.5 (IIL) Control of Measuring and Test Equipment (10CFR50, Appendix B Criterion XII) Attachment 1, section 2.b.3.k, Control and Calibration of Measuring and Test Equipment Application of the Management System for Nuclear Facilities: Draft Safety Guide. Topic 5.15 to 5.19 [Control of Products] Basic Requirement 12, Control of Measuring and Test Equipment Section 12.0, Control of Measuring and Test Equipment Attachment 2, section 2 (j) Maintenance Tool and Equipment Control [Endorsed ASME NQA-1-2008 w/NQA-1a-2009 addenda] Attachment 2: Section 5.d “Criterion 5” and 8.b “Criterion 8” Quality Assurance Applications for Nuclear Facility Applications; Basic Requirement 12, Control of Measuring and Test Equipment Section 3.5.3.15, Calibration and Test Procedures; Section 3.12, Control of Measuring and Test Equipment Standard Review Plan, Section 17.5 (IIL) Control of Measuring and Test Equipment (Criterion XII) Basic Requirement 12, Control of Measuring and Test Equipment Guidelines for the Use of Accreditation in Lieu of Commercial Grade Surveys for Procurement of Laboratory Calibration and Test Services.

2016 NCSL International Workshop & Symposium

6.3 DOD: Department of Defense Several branches of the Department of Defense rely on ISO 17025 and/or Z540.3 to augment their calibration programs. The following are several examples of such use. Air Force: AFMETCAL (AFLCMC / WNM) Air Force Metrology and Calibration Traceability AFD-130625 (2013): [131] “The AFMETCAL program adheres to the general requirements of ISO 17025. Calibration Services from Non-USAF Laboratories: Some USAF owned items cannot be calibrated within the USAF calibration laboratory structure. These items represent approximately 1% of the USAF calibration workload. In order to obtain calibration support for these items, it is necessary to send them to Non-USAF calibration laboratories... 2. The Non-USAF laboratory must comply with the general requirements of ISO 17025. 3. The Non-USAF laboratory must provide information on their quality system demonstrating their compliance with the general requirements of ISO 17025. This may include information regarding accreditation or audits performed by other parties. 6. Information obtained regarding the Non-USAF laboratory's quality system, its compliance with the general requirements of ISO 17025, and a copy of the calibration report must be retained by the organization obtaining the calibration service until the calibration expiration date.” Army: AR 750-43: Army Test, Measurement, & Diagnostic Equipment (2014): [132] “6-7 Commercial Contractor Support b. All contracts with a commercial laboratory for calibration services will specify that the commercial laboratory adhere to International Organization for Standards (ISO)/IEC 17025:2005 (or later) for all measurement parameters required for the calibration. Commercial calibration laboratories that are accredited to ISO/IEC 17025 and recognized as such by the International Laboratory Accreditation Cooperation, are not subject to paragraph 6–17 of this regulation. Nonaccredited commercial laboratories will adhere to paragraph 6–17.”

Navy: NACLA Navy Partnership – Naval Surface Warfare Center, Corona Division: [133] [134] “…the Measurement Science Department of the Naval Surface Warfare Center, Corona Division, is requesting that NACLA investigate the establishment of a NACLA Recognition Program that would provide a conformity assessment process for assessment bodies (including but not limited to Accreditation Bodies) that wish to assess organizations to the requirements of ANSI/NCSL Z540.3 – 2006, Requirements for the Calibration of Measuring and Test Equipment. (Referred herein as the National Standard). This National Standard has great potential in reducing “risk” associated with calibration measurement values in the many calibration systems Suppliers employ in producing their end product. The benefits associated with reducing risk are many and varied, depending on the industry… Examples of Industries that will benefit from implementing Z540.3 include, but are not limited to: • Packaging • Machining • Transportation • Healthcare • Energy • Manufacturing • Chemistry The goal is to establish a process that will promote uniformity in assessment to the supplemental requirements of sub-clause 5.3. 2016 NCSL International Workshop & Symposium

Since the Navy published the preceding account, NACLA did indeed develop a program in 2010 for assessment of accreditation bodies that seek to provide laboratory accreditation to Z540.3 [135]. Several accreditation bodies including A2LA [136], L-A-B [137], ANAB [138], IAS [139], and PJLA [140] now offer calibration laboratory accreditation to Z540.3 as a supplement to ISO 17025 accreditation. 6.4 Automotive Industry ISO/TS 16949 [141] is an international technical specification that prescribes quality management system requirements for the automotive industry. It replaces the QS9000 standard developed earlier in 1994 by the U.S. automakers. Although there is no specific government regulatory requirement in the U.S. for automotive manufacturers or their suppliers to be certified to ISO/TS 16949, vehicle assembly plants and supply chain manufacturers have been encouraged to obtain certification to this standard; it enjoys ubiquitous implementation and has become a de facto standard for this sector. Developed by the International Automotive Task Force (IATF), it was approved and published by ISO in 2009, building upon the requirements in ISO 9001 and adding language for calibration pertaining to ISO 17025. ISO TS 16949: Quality Management Systems — Particular Requirements for the Application of ISO 9001:2008 for Automotive Production and Relevant Service Part Organizations (2009): [141] “7.6.2 Calibration/verification records Records of the calibration/verification activity for all gauges, measuring and test equipment, needed to provide evidence of conformity of product to determined requirements, including employee- and customer-owned equipment, shall include  equipment identification, including the measurement standard against which the equipment is calibrated,  revisions following engineering changes,  any out-of-specification readings as received for calibration/verification,  an assessment of the impact of out-of-specification condition,  statements of conformity to specification after calibration/verification, and notification to the customer if suspect product or material has been shipped. 7.6.3.1 Internal laboratory… NOTE Accreditation to ISO/IEC 17025 may be used to demonstrate the organization's in-house laboratory conformity to this requirement but is not mandatory. 7.6.3.2 External laboratory External/commercial/independent laboratory facilities used for inspection, test or calibration services by the organization shall have a defined laboratory scope that includes the capability to perform the required inspection, test or calibration, and either  there shall be evidence that the external laboratory is acceptable to the customer, or  the laboratory shall be accredited to ISO/IEC 17025 or national equivalent. NOTE 1 Such evidence may be demonstrated by customer assessment, for example, or by customerapproved second party assessment that the laboratory meets the intent of ISO/IEC 17025 or national equivalent. NOTE 2 When a qualified laboratory is not available for a given piece of equipment, calibration services may be performed by the equipment manufacturer. In such cases, the organization should ensure that the requirements listed in 7.6.3.1 have been met.”

Similar accreditation requirements also existed in section 4.11.2.b.1 of QS-9000 in 1998 [141A]. 2016 NCSL International Workshop & Symposium

6.5 FAA – Federal Aviation Administration FAA calibration requirements for test and inspection equipment and tools are codified in 14 CFR §145.109 [11]. The FAA requires the following: FAA: 14 CFR §145.109 – Equipment, Materials, and Data Requirements. [11] “(a) Except as otherwise prescribed by the FAA, a certificated repair station must have the equipment, tools, and materials necessary to perform the maintenance, preventive maintenance, or alterations under its repair station certificate and operations specifications in accordance with part 43. The equipment, tools, and material must be located on the premises and under the repair station's control when the work is being done. (b) A certificated repair station must ensure all test and inspection equipment and tools used to make airworthiness determinations on articles are calibrated to a standard acceptable to the FAA…”

FAA policy pertaining to this requirement in 14 CFR §145.109 is given below. FAA Order 8900.1, Vol 6, Chap 9 Part 145 Policy; Flight Standards Information System – FSIMS (2014): [142] “1)

The ASI [Aviation Safety Inspector] should verify:

a) The repair station is calibrating MTE [Measuring & Test Equipment] per intervals, procedures, and the system described in the RSM [Repair Station Manual] or QCM [Quality Control Manual]. b) All MTE are calibrated and traceable to a standard acceptable to the Federal Aviation Administration (FAA), to include those recommended by the manufacturer, and the National Institute of Standards and Technology (NIST) or other national authority. NOTE: The part 145 rule states that tooling used to make airworthiness determinations must be calibrated to a standard acceptable to the FAA. Those standards may be derived from the NIST, to a standard provided by the equipment manufacturer, or other recognized standards. The International Bureau of Weights and Measures (BIPM) is a recognized authority that maintains a global list of National Metrology Institutes (NMI). The BIPM Web site lists the NMI signatory countries that participate in the International Committee for Weights and Measures (CIPM). The CIPM Mutual Recognition Arrangement (MRA) signatories are acceptable to the FAA and can be found at http://www.bipm.org. There are many accreditation bodies that provide third-party laboratory accreditation. The International Laboratory Accreditation Cooperation (ILAC) establishes a global network for accreditation of laboratory and testing facilities. Signatories to the ILAC MRA are in full conformance with the standards of International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) 17011. ILAC MRA signatories are acceptable to the FAA and can be found at http://ww.ilac.org. Accredited laboratories have already established traceability through the assessment and accreditation process under ISO/IEC 17025. No further documentation is required once traceability is confirmed to a recognized accredited laboratory. Additionally, for foreign equipment, the standard of the country of manufacture may be used if acceptable to the Administrator.”

2016 NCSL International Workshop & Symposium

Additional FAA calibration requirements are documented in Order 6200.4G [143]. FAA Order 6200.4G – National Test Equipment Program Management (2010): [143] “11. Calibration of Test Equipment… j. All calibration service providers will minimally comply with ANSI/NCSL Z540-1, Calibration Laboratories and Measuring and Test Equipment General Requirements for the Competence of Testing and Calibration Laboratories or ISO/IEC 17025 (International Organization for Standardization / Calibration, Measurement Gages and Tests Laboratories) [sic]. These are the guides used by the FAA for calibration laboratory organization, management, quality system, personnel, environment, standards/equipment, reference materials, procedures, traceability, certification, labeling, records, and reporting.”

Similar to NASA’s practice of requiring Z540.3 compliance in solicitations or RFP’s when selecting a contractor to perform calibration work for the government, the FAA has engaged in an analogous practice of specifying calibration requirements and standards when issuing Performance Work Statements. The following is an excerpt from the solicitation for the calibration contract at the FAA Logistics Center at Mike Monroe Aeronautical Center [144]. FAA Performance Work Statement – Mike Monroney Aeronautical Center – FAA Logistic Center (2008): [144] FAA Test, Measurement & Diagnostic Equipment Calibration Services. 3. PURPOSE “1.1.1 The calibration service is the key to ensuring documented accreditation and certification for all government owned TMDE [Test Measurement & Diagnostic Equipment], and provides the government a certification of calibration attesting each item performs within specified standards of tolerance. It is not possible to separate GTMDE used for critical systems from GTMDE used for advisory systems, or to determine any FAA site as more critical than another to the safe operation of the NAS [National Air Space System]. All TMDE therefore requires certification to enable certification of critical FAA operating systems. This function is determined as essential to the safety of human life… It is the FAA’s intent to enter into a performance based, firm fixed price agreement for the services required. Offerors must develop and provide cost effective solutions, with the opportunity to propose innovative alternatives that meet the below objectives: 

Provide calibration and certification of calibration attesting each individual item of TMDE performs within specified standards of tolerance.



Ensure calibration accreditation can be tracked for each piece of TMDE used to achieve certification of critical NAS operating systems.



Ensure Calibrations are performed on time in accordance with calibration due dates.



Ensure Calibration services meet quality standards comparable to or better than best commercial, industry practices…

Calibration Documentation, Compliance, and Accreditation Report Review: FAA shall perform random audits/reviews of: The Contractors’ ISO 9001 Surveillance Reports including the nonconformance reports. If applicable, the FAA may review the Contractor’s ISO/IEC 17025 accreditation report and associated documentation. 6.3.3 WORKING STANDARDS ACCURACY 6.3.3.1 All Working Standards utilized by the Contractor in the calibration process shall be more accurate than the TMDE being calibrated. Accuracy ratios (Working Standards to TMDE being calibrated) shall be maintained at 4:1 as a minimum except in rare cases where the Contracting Officer’s Technical Representative (COTR) approves less than a 4:1 ratio”.

2016 NCSL International Workshop & Symposium

Many aircraft rely on accurate measurement data from a variety of sensors being fed to flight computers. Autopilots and fly-by-wire systems employ these measurements to make real-time adjustments to the flight control system. Inaccurate calibration of these sensors can result in significant consequences. For example, on February 25th, 2008, a B-2 bomber suffered a serious crash at Andersen Air Force Base due to mis-calibrated Port Transducer Units (PTU’s) in the Air Data System. The crash resulted in injuries to the co-pilot and the total loss of a $1.4 billion aircraft [145]. 7. The National Technology Transfer and Advancement Act of 1995 (NTTAA) For many decades, there has been a movement within federal agencies to incorporate voluntary consensus standards into their programs and regulatory affairs, in lieu of government-unique standards. For calibration requirements, this has been evidenced by NASA, the Department of Energy, the Nuclear Regulatory Commission, the Department of Defense, the Federal Aviation Administration, and many others. Agencies have come to rely on Z540.3 and/or ISO 17025 in support of their calibration and metrology programs. For some, this was due in part to the National Technology Transfer and Advancement Act of 1995 (NTTAA) [146]. For the Department of Defense, this was also due to the issuance of what has become known as the “Perry Memo” in 1994 [147]. The Perry Memo was largely responsible for the cancellation of a significant number of military standards in the mid 1990’s (see bottom of Table 1B). In many cases, these “Mil-Specs” were replaced by voluntary consensus standards (e.g. MIL-STD-45662A was replaced by Z540.1 in 1995). This section provides background information on government use of consensus standards as they relate to the NTTAA. W. Perry – Department of Defense “Perry Memo” (1994): [147] SUBJECT: Specifications & Standards - A New Way of Doing Business “…the Department of Defense must increase access to commercial state-of-the-art technology and must facilitate the adoption by its suppliers of business processes characteristic of world class suppliers. I have repeatedly stated that moving to greater use of performance and commercial specifications and standards is one of the most important actions that DoD must take to ensure we are able to meet our military, economic, and policy objectives in the future. Moreover, the Vice President's National Performance Review recommends that agencies avoid government-unique requirements and rely more on the commercial marketplace. To accomplish this objective, the Deputy Under Secretary of Defense (Acquisition Reform) chartered a Process Action Team to develop a strategy and a specific plan of action to decrease reliance, to the maximum extent practicable, on military specifications and standards… I wholeheartedly accept the Team's report and approve the report's primary recommendation to use performance and commercial specifications and standards in lieu of military specifications and standards… Waivers for the use of military specifications and standards must be approved… …encourage contractors to propose non-government standards and industry-wide practices that meet the intent of the military specifications and standards. These standards… shall be considered as alternatives to those military specifications and standards… Use of Non-Government Standards: …form partnerships with industry associations to develop nongovernment standards for replacement of military standards…. Reducing Oversight: …reduce direct Government oversight by substituting process controls and nongovernment standards in place of development and/or production testing and inspection and military-unique quality assurance systems. Education and Training: The Under Secretary of Defense (Acquisition and Technology) shall ensure that training and education programs throughout the Department are revised to incorporate specifications and standards reform. The Process Action Team's report… are a solid beginning that will increase the use of performance and commercial specifications and standards. I encourage you and your leadership teams to be active participants in establishing the environment essential for implementing this cultural change”. 2016 NCSL International Workshop & Symposium

Similar to the Perry Memo for the Department of Defense, the NTTAA set forth requirements for all federal agencies regarding the use of voluntary consensus standards. National Technology Transfer and Advancement Act (NTTAA) of 1995: [146] Public Law 104-113 “Utilization of Consensus Technical Standards by Federal Agencies; (1) In general.--Except as provided in paragraph (3) of this subsection, all Federal agencies and departments shall use technical standards that are developed or adopted by voluntary consensus standards bodies, using such technical standards as a means to carry out policy objectives or activities determined by the agencies and departments. (2) Consultation; participation.--In carrying out paragraph (1) of this subsection, Federal agencies and departments shall consult with voluntary, private sector, consensus standards bodies and shall, when such participation is in the public interest and is compatible with agency and departmental missions, authorities, priorities, and budget resources, participate with such bodies in the development of technical standards. (3) Exception.--If compliance with paragraph (1) of this subsection is inconsistent with applicable law or otherwise impractical, a Federal agency or department may elect to use technical standards that are not developed or adopted by voluntary consensus standards bodies if the head of each such agency or department transmits to the Office of Management and Budget an explanation of the reasons for using such standards. Each year, beginning with fiscal year 1997, the Office of Management and Budget shall transmit to Congress and its committees a report summarizing all explanations received in the preceding year under this paragraph. (4) Definition of technical standards.--As used in this subsection, the term ’technical standards’ means performance-based or design-specific technical specifications and related management systems practices”

The NTTAA was based on the White House Office of Management and Budget’s OMB Circular A119, which governs the use of voluntary consensus standards by federal agencies. The 1998 revision of Circular A-119 [148] stated: OMB Circular A-119 (1998): [148] “Standards developed by voluntary consensus standards bodies are often appropriate for use in achieving federal policy objectives and in conducting federal activities, including procurement and regulation. The policies of OMB Circular A-119 are intended to: (1) encourage federal agencies to benefit from the expertise of the private sector, (2) promote federal agency participation in such bodies to ensure creation of standards that are useable by federal agencies, (3) reduce reliance on government-unique standards where an existing voluntary standard would suffice. …the policy of the federal government, in its procurement and regulatory activities, is to: (1) rely on voluntary standards, both domestic and international, whenever feasible and consistent with law and regulation, (2) participate in voluntary standards bodies when such participation is in the public interest and is compatible with agencies' missions, authorities, priorities, and budget resources, (3) coordinate agency participation in voluntary standards bodies so that . . . the most effective use is made of agency resources . . . and that the views expressed by such representatives are in the public interest and do not conflict with the interests and established views of the agencies."

2016 NCSL International Workshop & Symposium

The most recent 2016 revision to OMB Circular A-119 [149] states: OMB Circular A-119 (2016): [149] “…this Circular directs agencies to use standards developed or adopted by voluntary consensus standards bodies rather than government-unique standards, except where inconsistent with applicable law or otherwise impractical. This Circular… describes procedures for satisfying the reporting requirements of the NTTAA. The policies in this Circular are intended to minimize the reliance by agencies on government-unique standards. The Circular also provides policy guidance to agencies on the use of conformity assessment in procurement, regulatory, and program activities. This Circular replaces Office of Management and Budget (OMB) Circular No. A-119, dated February 10, 1998. Many voluntary consensus standards are appropriate or adaptable for the Federal government's purposes… Conformity assessment includes sampling and testing, inspection, supplier’s declaration of conformity, certification, and management system assessment and registration. Conformity assessment also includes accreditation of the competence of those activities. This Circular applies to all agencies and agency representatives who use standards or conformity assessment and/or participate in the development of standards. “Agency” means any… establishment of the Federal government. It also includes any regulatory commission or board… Consistent with Section 12 (d)(1) of the NTTAA, all Federal agencies must use voluntary consensus standards in lieu of government-unique standards in their procurement and regulatory activities, except where inconsistent with law or otherwise impractical. In these circumstances, your agency must submit a report describing the reason(s) for its use of government-unique standards in lieu of voluntary consensus standards as explained in Sections 9-11. In evaluating whether to use a standard… an agency should consider the following factors…(f) the ongoing use of the standard by other agencies for the same or a similar requirement, the use of which in a particular instance would increase consistency across the Federal government… (h) the prevalence of the use of the standard in the national and international marketplaces… As a general matter, standards being considered for use in regulation that specify nomenclature, basic reference units, or methods of measurement or testing, and that are primarily empirical in their formulation, will ordinarily warrant less scrutiny by an agency than standards that embody factors that are less objective… In addition, the United States is obligated under the TBT [Technical Barriers to Trade] Agreement to use relevant international standards, except where such standards would be an ineffective or inappropriate means to fulfill the legitimate objective pursued. In particular, the TBT Agreement, Article 2.4, provides that: ‘Where technical regulations are required and relevant international standards exist or their completion is imminent, [WTO] Members shall use them, or the relevant parts of them, as a basis for their technical regulations…’ …agencies should consider using voluntary consensus standards, as described in this Circular, to achieve their regulatory, procurement, and program needs, including for test methods, sampling procedures, and protocols, if applicable. It may be appropriate for the agency to allow the use of multiple standards in order, for example, to permit greater flexibility… …agencies should recognize the possible contribution of private sector conformity assessment activities. When properly conducted, conformity assessments conducted by private sector conformity assessment bodies can increase productivity and efficiency in government and industry, expand opportunities for international trade, conserve resources, improve health and safety, and protect the environment. Working closely with NIST and OMB, agencies are encouraged to identify their conformity assessment needs in such areas as regulatory compliance and…to assess whether the use of private sector conformity assessment mechanisms in lieu of or in conjunction with government conformity assessment procedures would be beneficial…” 2016 NCSL International Workshop & Symposium

Prior to the 2016 revision of Circular A-119, the Office of Management and Budget stated that: U.S. Office of Management and Budget (OMB) (2014): [150] “For two decades, it has been the policy of the United States Government to support the development and use of efficient and effective standards and conformity assessment approaches that, when adopted by Federal agencies, can address important regulatory, procurement, and policy objectives, such as increasing the net benefits of Federal regulation. Promoting and using high-quality standards and standardization systems in turn supports the broader goals of enhancing economic growth, innovation, and competition and of facilitating international trade by avoiding the creation of unnecessary obstacles to trade.”

Encouragement of federal agencies to utilize voluntary consensus standards has been evident for nearly 30 years. The Administrative Council of the United States (ACUS) is an independent agency charged with providing recommendations for improving federal agency procedures. In 1978, ACUS stated: ACUS Recommendation 78-4 (1978): [151] “Federal Agency Interaction with Private Standard-Setting Organizations in Health & Safety Regulation (a ) Many federal agencies have authority to issue mandatory health or safety regulations relating to products, materials, processes, practices or services that may be the subjects of voluntary standards prepared by non-governmental organizations. Non-governmental standards, though not legally enforceable, have in fact gained wide acceptance and a high degree of observance… Standards developed by private organizations… or under the Department of Commerce voluntary standards program, are frequently referred to as "voluntary consensus standards," and are the subject of this recommendation… (c) Members of technical committees that develop voluntary consensus standards often have a wealth of technical knowledge and expertise that agency staffs do not possess… (f) The recommendation that follows is limited to agency interaction with standards-developing organizations and use of voluntary consensus standards in the context of regulation of health or safety. The recommendation may nevertheless be significant in relation to setting standards for other purposes: For example, in conservation of energy and resources, environmental issues, and formulation of test methods and definitions… 4. Each agency should, as a matter of general policy, regularly review standards or revisions proposed by technical committees active in the areas of regulatory concern of the agency… 6. Agencies with authority to issue health or safety regulations should consider the use of existing relevant voluntary consensus standards in developing mandatory standards. Voluntary consensus standards should be considered with due caution and on a case-by-case basis. Ordinarily, standards which embody judgmental factors should receive greater scrutiny when being considered by agencies for adoption into regulations than standards which specify nomenclature, basic reference units, or methods of measurement or testing, and which are primarily empirical in their formulation. (7a) If the voluntary consensus standard adequately addresses the questions of health or safety and is being substantially complied with by the affected industry, the agency may decide to take no further regulatory steps, or, alternatively, to adopt the standard into its regulations (see paragraph (f) below), and direct its primary regulatory efforts elsewhere… (7e) Agencies should consider the "regulatory guide" approach as a means of effectively making use of voluntary consensus standards. A "regulatory guide" is a formal declaration by the agency that compliance with designated portions, or all, of a voluntary consensus standard will be considered an acceptable method of compliance with a general mandatory standard appearing in either the governing statute or the agency's regulations. When taking this approach, the agency should suitably publicize its decision and reasons therefor. (f)

The agency may adopt a voluntary standard into its regulations either by placing the text of the standard in the regulations, or, preferably, by incorporating the standard by reference pursuant to 1 CFR part 51.”

2016 NCSL International Workshop & Symposium

Section (f) in the preceding ACUS Recommendation 78-4 references a concept known as “incorporation by reference”, whereby voluntary consensus standards can be directly referenced within the text of the CFR. ACUS recommendation 2011-5 [152] explains this option, now codified in Title 1 of the Code of Federal Regulations Section 51.1 [153]. The ACUS has stated: ACUS Recommendation 2011-5, Incorporation by Reference (2011): [152] “Incorporation by reference allows agencies to comply with the requirement of publishing rules in the Federal Register to be codified in the Code of Federal Regulations (CFR) by referring to material published elsewhere. The practice is first and foremost intended to—and in fact does—substantially reduce the size of the CFR. But it also furthers important, substantive regulatory policies, enabling agencies to draw on the expertise and resources of private sector standard developers to serve the public interest. Incorporation by reference allows agencies to give effect to a strong federal policy, embodied in the National Technology Transfer and Advancement Act of 1995 and OMB Circular A-119, in favor of agency use of voluntary consensus standards. This federal policy benefits the public, private industry, and standard developers”.

Examples of federal agencies incorporating ISO 17025 ‘by reference” into the Code of Federal Regulations are provided below. 46 CFR 162.060-36 – Quality Assurance Project Plan (QAPP) Requirements [154] Coast Guard / Department of Homeland Security “The approval testing and evaluation process must contain a rigorous Quality Assurance and Quality Control program consisting of a QAPP developed in accordance with ISO/IEC 17025:2005(E), as amended ISO/IEC 17025:2005/Cor.1:2006(E) (incorporated by reference, see § 162.060–5). The independent laboratory performing approval tests and evaluations is responsible for ensuring the appropriate Quality Assurance and Quality Control procedures are implemented.”

10 CFR 429.110 – Enforcement Testing [155] Department of Energy “(3) Testing will be conducted at a lab accredited to the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC), “General requirements for the competence of testing and calibration laboratories,” ISO/IEC 17025:2005(E) (incorporated by reference; see§ 429.4). If testing cannot be completed at an independent lab, DOE, at its discretion, may allow enforcement testing at a manufacturer's lab, so long as the lab is accredited to ISO/IEC 17025:2005(E) and DOE representatives witness the testing.”

47 CFR 2.948 – Description of Measurement Facilities [156] Federal Communications Commission “(d) A laboratory that has been accredited with a scope covering the required measurements shall be deemed competent to test and submit test data for equipment subject to verification, Declaration of Conformity, and certification. Such a laboratory shall be accredited by an approved accreditation organization based on the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Standard 17025, “General Requirements for the Competence of Calibration and Testing Laboratories.”

2016 NCSL International Workshop & Symposium

The definition of a “standard” or “technical standard” in OMB Circular A-119 is “a common and repeated use of rules, conditions, guidelines or characteristics for products or related processes and production methods, and related management systems practices.” Standards which are not considered voluntary consensus standards are, “government-unique standards, which are developed by the government for its own uses” and “standards mandated by law such as those contained in the United States Pharmacopeia and the National Formulary…” Regarding the use of voluntary consensus standards for quality systems in the healthcare industry, the following was published by the FDA upon the revision of the cGMPs in 1997: FDA Final Rule cGMP Quality System Regulation Preamble (1997): [157] “This revision follows the suggestion underlying many comments on specific provisions that FDA generally harmonize the CGMP requirements and terminology with international standards. ISO 9001:1994, ISO/CD 13485, and EN 46001 employ this terminology to describe the CGMP requirements. In addition, this title [Quality System Regulation] accurately describes the sum of the requirements, which now include the CGMP requirements for design, purchasing, and servicing controls. CGMP requirements now cover a full quality system. FDA notes that the principles embodied in this quality system regulation have been accepted worldwide as a means of ensuring that acceptable products are produced. While the regulation has been harmonized with the medical device requirements in Europe, Australia, and Japan, as well as the requirements proposed by Canada, it is anticipated that other countries will adopt similar requirements in the near future. FDA, however, did not adopt ISO 9001:1994 verbatim for two reasons. First, there were complications in dealing with the issue of copyrights and, second, FDA along with health agencies of other governments does not believe that for medical devices ISO 9001:1994 alone is sufficient to adequately protect the public health. Therefore, FDA has worked closely with the GHTF [Global Harmonization Task Force] and TC 210 to develop a regulation which is consistent with both ISO 9001:1994 and ISO/CD 13485. FDA made several suggestions to TC 210 on the drafts of the ISO/CD 13485 document in order to minimize differences and move closer to harmonization. In some cases, FDA has explicitly stated requirements that many experts believe are inherent in ISO 9001:1994. Through the many years of experience enforcing and evaluating compliance with the original CGMP regulation, FDA has found that it is necessary to clearly spell out its expectations. This difference in approach does not represent any fundamentally different requirements that would hinder global harmonization. In fact, numerous comments expressed their approval and satisfaction with FDA's effort to harmonize the quality system requirements with those of ISO 9001:1994 and ISO/CD 13485… FDA supports the international harmonization of standards and regulations governing medical devices and the eventual mutual recognition of CGMP inspections between major device markets. While full achievement of this goal is still in the future, the harmonization of quality standards is an important first step. FDA believes in a step wise approach toward harmonization and eventual mutual recognition”.

The National Institute of Standards and Technology (NIST), is charged with providing an annual report to the OMB summarizing the various federal agencies participation in the NTTAA. Excerpts from several of these annual reports are provided here and contain information relevant to FDA, NRC and other federal agency programs.

2016 NCSL International Workshop & Symposium

th

NIST 9 Annual Report on Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment (2005): [158] "The Department of Health and Human Services HHS reported several activities including: FDA’s participation in ongoing conformity assessment activities such as the ANSI Accreditation and International Conformity Assessment, as well as American Society for Testing and Materials (ASTM) Committee E-36 on Conformity Assessment allows FDA to ensure that its needs are met while utilizing existing recognition and accreditation criteria. FDA’s Center for Devices and Radiological Health allows a medical device manufacturer to submit a Declaration of Conformity to a “recognized standard” as described in ISO/IEC Guide 22 in its standards recognition program and has developed an MRA with the European Union on mutual recognition of each other's conformity assessment procedures related to manufacture and marketing of medical devices. This reduces costs for manufacturers and decreases the time to market for approved products. The FDA Office of Regulatory Affairs (FDA/ORA) actively participates in the National Cooperation for Laboratory Accreditation (NACLA), serving as a member of the NACLA Executive Board of Directors and participating in the NACLA Recognition Committee for Accrediting Bodies who apply for mutual recognition. This participation may form the basis for the future accreditation [to ISO 17025] of FDA laboratories. With the idea of enhancing international credibility and recognition, FDA’s Center for Food Safety and Applied Nutrition (CFSAN) is moving towards ISO [17025] accreditation of its own laboratories that perform regulatory work… The Department of Commerce reported that the Nuclear Regulatory Commission (NRC) now accepts accreditation [to ISO 17025] by qualified laboratory accreditation bodies as an acceptable alternative so a supplier audit, commercial-grade survey, or in-process surveillance for the qualification of commercial grade calibration service suppliers. This reduces resource burdens on industry and eliminates costs related to redundant audits. The Department also reported that: NIST provides technical support for the Inter-American Accreditation Cooperation (IAAC). Such arrangements/agreements are designed to harmonize conformity assessment practices and promote the global acceptance of conformity assessment results from qualified bodies to minimize the need for and cost of redundant conformity assessment activities. Federal agencies continue to experience personnel turnover at all organizational levels due to reorganizations, accelerated or early retirements, and normal attrition. These changes make it difficult for Federal agencies to retain high-level managers who understand the importance of standards and who visibly support standardsrelated activities. Staff turnover has also caused a decrease in “institutional memory” of standards policies, responsibilities, and practices. To address this issue, NIST recently developed and is now providing training for Federal employees who are engaged in developing standards and using them in regulation or procurement actions. NIST is creating a handbook for Standards Executives so that they will have readily available the information needed to make decisions about the use of standards. Sustained high-level Federal agency leadership has been identified as the primary driver of successful NTTAA implementation. Top agency leaders have the ability to direct policy and resources in ways that bring about other desirable outcomes, such as increased Federal participation and collaboration with the private sector. Ensuring that agency Standards Executives have the tools at hand to show how standards and the standards making process contribute to their agency’s mission is a continuing priority”.

2016 NCSL International Workshop & Symposium

NISTIR 7413: Tenth (2006) Annual Report on Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment - Addendum (2006): [159] “Standards may become part of conformance activities as they may provide an acceptable approach to be in compliance with applicable laws and regulations. Also, FDA laboratories which conduct official product testing are, or are in the process of becoming, ISO/IEC 17025 accredited. They have conducted staff training, are in the process of writing a Laboratory Quality Assurance Manual centrally documenting Center policies and procedures related to the official testing of regulated biological products, are implementing a quality management software tool to assist in the effort, under direction of quality assurance managers hired to coordinate the implementation of an ISO 17025-based quality system” NISTIR 7503: Eleventh (2007) Annual Report on Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment – Addendum (2008): [160] Establishment and use of standards result in benefits to FDA that include: standards that can assist reviewers with assessment of product applications; international standards that can be used by multiple regulatory regions, following our legal mandate to facilitate harmonization on an international level; standards that often result in better utilization of limited internal resources; and direct participation by various stakeholders in development of standards that results in a consensus among users, manufacturers and government regulators on safety and effective use of regulated products. NISTIR 7598: Twelfth (2008) Annual Report on Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment – Addendum (2009): [161] “The FD&C Act requires the Center for Devices and Radiological Health to annually publish a list of voluntary consensus standards ’recognized’ by the Agency for the use of manufacturers and others in meeting the regulatory requirements of the FDA. The list of recognized Standards and several applicable guidance documents are available at http://www.fda.gov/cdrh/stdsprog.html.”

As stated in the 12th annual NIST report addendum above, the FDA/CDRH maintains a database of recognized consensus standards [84] [85]. For example, the list below indicates specific calibration procedures which are formally recognized by the FDA. However, ISO 17025 and ANSI/NCSL Z540.3 for calibration programs are not included in the FDA database of recognized standards. FDA Recognition Number

Standard Developing Organization

4-164

ANSI

12-61

12-141

Standard Designation Number and Date

Title of Standard

Specialty Task Group

FR Publication Date

S3.7-1997 (Reaffirmed 2003) (Reaffirmed 2008)

Method for Coupler Calibration of Earphones

Dental/ENT

3/16/2012

IEC

61145 (1992-05)

Calibration and usage of ionization chamber systems for assay of radionuclides

Radiology

5/3/1999

IEEE

N42.13-2004

Calibration and Usage of "Dose Calibrator" Ionization Chambers for the Assay of Radionuclides

Radiology

3/31/2006

Radiology

7/9/2014

Sterility

8/5/2013

12-278

IEC

62127-2 Edition 1.0 2007-08

Ultrasonics -- Hydrophones -- Part 2: Calibration for ultrasonic fields up to 40 MHz [Including: Technical Corrigendum 1:2008 and Amendment 1:2013]

14-381

ASTM ISO

51261 Second Edition 201304-15

Practice for calibration of routine dosimetry systems for radiation processing

2016 NCSL International Workshop & Symposium

Likewise, and as previously stated herein, ISO 17025 is also not currently listed as a “recognized standard” by the IMDRF N15 document, “Final Report: List of International Standards Recognized by IMDRF Management Committee Members as of March 2014” [87] [88]. The FDA/CDRH has published guidance on the use of consensus standards [86]. This guidance states, “CDRH believes that conformance with recognized consensus standards can support a reasonable assurance of safety and/or effectiveness for many applicable aspects of medical devices”. Although ISO 17025 is not currently listed as a recognized consensus standard, the FDA has expressed positive statements towards laboratory accreditation to ISO 17025. However, this dialogue was directed more toward testing laboratories, rather than calibration laboratories [162]. FDA Guidance for Industry – Submission of Laboratory Packages by Accredited Laboratories (2009): [162] “Since the time we issued the proposed rule, significant changes in laboratory accreditation have occurred. For example, when we drafted the proposed rule, there was a trend towards the use of the International Organization for Standardization (ISO) standard ISO 17025, "General Requirements for the Competence of Testing and Calibration Laboratories," but no firm consensus (see 69 FR at 23461). Additionally, when we drafted the proposed rule, parties disagreed as to the value of laboratory accreditation, and FDA’s own laboratories were not accredited. Today, there is widespread agreement on ISO 17025, the laboratory industry favors accreditation, and FDA's own laboratories are accredited. Moreover, the Administration’s Strategic Framework expressly seeks better ways to ensure compliance with safety standards, while the Action Plan and the GAO testimony reflect greater support for the use of accredited laboratories. Thus, given these and other developments and the Action Plan's recommendation to issue guidance, we have decided to issue this guidance document instead of proceeding with a final rule at this time… Rigorous accreditation standards give us more confidence that accredited laboratories have the technical capability and trained personnel to perform the specific methods for which they are accredited.”

As stated above, the FDA’s Office of Regulatory Affairs (ORA) laboratories are accredited to ISO 17025. The quality policy of the ORA laboratories states that, “ORA laboratories are committed to laboratory accreditation according to the requirements of ISO/IEC 17025” [163]. The FDA has documented the historical precedent leading up to the decision to pursue accreditation, including legal testimony regarding admissibility of scientific laboratory evidence in litigation [164]. FDA: Allergenic Products Advisory Committee Meeting Background Document: ISO 17025 Accreditation of the Laboratory of Immunobiochemistry (2011): [164] “Following these media awareness events, the FDA Senior Science Council recommended in 1998 that FDA develop quality systems compliant to ISO 17025 (then known as ISO Guide 25) for all official testing activities. ISO 17025 is also known as “General Requirements for the Competence of Testing and Calibration Laboratories,” and merges the requirements for technical competence in testing and calibration with requirements for quality systems. ISO 17025 is the main standard used by testing laboratories and is broken into two main sections: management and technical requirements… In response to the Senior Science Council’s recommendations, the Center for Biologics Evaluation and Review (CBER) evaluated existing gaps in CBER testing and lot release activities, purchased and implemented quality system software, developed the quality system policy manual, and provided basic ISO 17025 training for select laboratory personnel. To facilitate the goal of ISO 17025 accreditation, CBER created the Division of Product Quality in 2006, and then decided that official testing be transferred to DPQ, thus satisfying the goal that the testing would be performed in an accredited laboratory according the standards of ISO 17025”. 2016 NCSL International Workshop & Symposium

The steps taken by the FDA/CBER Laboratory of Immunobiochemistry (LIB) to become accredited to ISO 17025 were documented in 2011 by Menzies [165, 166]. CBER’s Laboratory Quality System (LQS) officially received accreditation to ISO 17025 in October 2010 [167]. In 2012, the FDA’s Division of Federal-State Relations (DFSR) announced a Cooperative Agreement Program for the ISO 17025 accreditation of state food testing Laboratories (U18) [168] [169]. This program provides monetary grants for laboratories seeking ISO 17025 accreditation or to maintain existing accreditation. The FDA has also stated, “ISO/IEC 17025 is one of the most important standards for calibration and testing laboratories… Laboratory customers, regulatory authorities and accreditation bodies may also use it in confirming or recognizing the competence of laboratories [169A]. ” FDA: ISO/IEC 17025:2005 Accreditation for State Food Testing Laboratories (2012): [169] “The intended outcome of this FOA [Funding Opportunity Announcement] is for microbiological and chemical food analyses performed on behalf of State manufactured food regulatory programs to be conducted within the scope of an ISO/IEC 17025:2005 accredited laboratory and the goal of achieving a nationally integrated food safety system to be further advanced. This will be accomplished by preparing the primary food testing laboratories for State manufactured food regulatory programs to achieve and maintain ISO/IEC 17025:2005 laboratory accreditation. Currently accredited laboratories will also be prepared for accreditation enhancements. Increased laboratory analyses from ISO/IEC 17025:2005 accredited labs, as would be accomplished through this cooperative agreement, will in effect serve to increase the analytical capacity for FDA and enhance efforts to protect the food supply.”

The FDA has also recently issued a Final Rule on the Food Safety and Modernization Act (FSMA) regarding accredited third party certification [170]. FDA: Accreditation of Third-Party Certification Bodies To Conduct Food Safety Audits and To Issue Certifications (2015): [170] “At our own initiative, we are removing the requirement to use a laboratory consistent with section 422 of the FD&C Act and inserting a requirement in § 1.651(b)(3) to use a laboratory accredited under ISO/IEC 17025:2005 or another laboratory accreditation standard that provides at least a similar level of assurance in the validity and reliability of sampling methodologies, analytical methodologies, and analytical results.”

Regarding the FSMA, the American Council of Independent Laboratories (ACIL) has stated [171], ACIL Statement on the Food Safety Modernization Act (2011): [171] “ACIL has advocated for years the FDA establish uniform requirements for private laboratory submission… …ACIL has consistently and repeatedly urged FDA to adopt and recognize International Standards Organization (ISO) accreditation [ISO 17025] as the primary basis for qualification of laboratories to submit analytical data to FDA for any purpose. We are pleased that FSMA addressed these issues and look forward to working with FDA on their implementation.”

Recognition and use of ISO 17025 by federal agencies, as well as utilization of the associated network of private sector accreditation and certification bodies, is consistent with ACUS Recommendation 2012-7 [172] (below) and is addressed in an ACUS report by McAllister [173]. 2016 NCSL International Workshop & Symposium

ACUS Recommendation 2012-7 (2012): [172] “Federal agencies in diverse areas of regulation have developed third-party programs to assess whether regulated entities are in compliance with regulatory standards and other requirements. Through these programs, third parties are charged with assessing the safety of imported food, children’s products, medical devices, cell phones and other telecommunications equipment, and electrical equipment used in workplaces. In these regulatory third-party programs, regulated entities generally contract with third parties to carry out product testing, facility inspections, and other regulatory compliance assessment activities in the place of regulatory agencies. Regulatory agencies take on new roles in coordinating and overseeing these third-party actors. In some areas of regulation, Congress has directed federal agencies to develop a third-party program; in others, regulatory agencies have developed programs under existing statutory authority. Several broad reasons support the growing use of third-party programs in federal regulation. In many areas, federal regulatory agencies are faced with assuring the compliance of an increasing number of entities and products without a corresponding growth in agency resources. Third-party programs may leverage private resources and expertise in ways that make regulation more effective and less costly. In comparison with other regulatory approaches, third-party programs may also enable more frequent compliance assessment and more complete and reliable compliance data. Because agencies can authorize third parties located in other countries to undertake assessment activities, third-party programs may be particularly effective when regulated products or processes are international in scope. Regulatory third-party programs raise a host of important questions. Representing a partial privatization of the public function of implementing and enforcing regulatory law, they are a form of “public-private governance,” in which private actors play roles that are traditionally viewed as governmental in nature. Frequently, regulatory third-party programs use the practices and terminology of an international conformity assessment framework that has been developed by private-sector standards organizations. “Conformity assessment” is defined in international standards as the “demonstration that specified requirements relating to a product, process, system, person, or body are fulfilled.” International standards also set forth how the organizations that conduct conformity assessment – “conformity assessment bodies,” which are usually private organizations – should operate. International standards have been developed for various types of conformity assessment bodies, including testing bodies, certification bodies, and inspection bodies. Recognizing the assessment of regulatory compliance as a form of conformity assessment, many federal agencies that have established third-party programs have relied on conformity assessment standards and bodies. Agencies may require, for example, that third parties that certify conformity with regulatory requirements operate in accordance with the international standards for certification bodies. Federal agencies may also require that the third parties be accredited by accreditation bodies that operate in accordance with international accreditation standards. Accreditation bodies are established in many countries, and they may be either private or governmental. Agencies that establish third-party programs generally cannot or do not delegate their regulatory authority to conformity assessment bodies. Rather, agencies authorize conformity assessment bodies to perform certain technical tasks to assess conformity, and regulatory agencies rely on these assessments in their own enforcement of regulatory requirements. The goal is to leverage expertise and resources in the private sector to serve regulatory objectives. A key resource for agencies considering a regulatory third-party program is the National Institute of Standards and Technology (NIST), which has the responsibility under the National Technology Transfer and Advancement Act of 1995 to coordinate government conformity assessment activities with similar activities of private-sector entities, with the goal of avoiding unnecessary duplication and complexity. Following Office and Management and Budget (OMB) Circular A-119, NIST published guidance for federal agencies on conformity assessment activities in 2000”.

2016 NCSL International Workshop & Symposium

Such recommendations were preceded by efforts of NCSLI in 2001 as documented by Harris [36A]. Harris, NIST SP-986. NIST Workshop on Conformity Assessment for a Changing Government (2001): [36A] “Collaboration ▪ Met with Jim Turner, House Majority Counsel to Committee on Science, Commerce, Technology July 27, 2001. - Laboratory accreditation issues; lack of uniformity in acceptance of accreditation by Federal agencies, effectiveness of NCSLI in representing interests of members - Possible proposal to OMB [White House Office of Management and Budget] Suggestions Government agencies should recognize competence of laboratories accredited to international quality standards - Versus additional audits and requirements - Compare ISO/IEC 17025 and NRC 10 CFR 50, find out if there are gaps, then address the gaps by 17025 updates - Environmental, chemical, biological Collaboration among Federal & accredited labs to provide traceable calibration services”.

The recent implementation of the FDA’s Medical Device Single Audit program (MDSAP) pilot program and the FSMA are exemplary applications of using third party programs to assess regulatory compliance. Additional benefit could further be realized from formal recognition of ISO 17025 accredited calibration laboratories during FDA audits, to include recognition as part of the MDSAP program. This would be analogous to the current recognition of such accredited calibration laboratories by the DOD, DOE, NRC, FAA, the automotive industry, and other federal agencies. Perhaps the most direct indication of FDA support and recognition of ISO 17025 accredited calibration laboratories was provided in March 2011 [173A] when addressing comments received in response to a proposed FDA rule regarding temperature indicating devices for particular food processing and packaging requirements. FDA, Response to Comment 11: Docket FDA-2007-N-0265 (2011): [173A] “One comment expressed concern about… documentation of accuracy of temperature-indicating devices and reference devices. The comment suggested that the final rule should instead require documentation that conforms to the standards established by the American National Standards Institute, National Conference of Standards Laboratories (ANSI/NCSL) or the International Organization for Standardization, International Electrotechnical Commission (ISO/IEC) for accrediting calibration laboratories. The comment stated that the laboratory accreditation standards indicate acceptable reporting practices. The comment acknowledged that the standards may be too prescriptive for food processors who perform their own calibrations. (Response) We do not agree that the regulation should require the documentation of accuracy of temperatureindicating devices and reference devices to conform to the standards specified in the comment for accrediting calibration laboratories. Although FDA supports use of accredited calibration laboratories and recognizes that the laboratories must maintain certain documentation for the accreditation, the records required by this final rule are appropriately limited to those necessary to document that the temperature-indicating device was tested for accuracy at sufficient frequency to ensure accuracy during processing. As acknowledged by the comment, a requirement for processors to adhere to accreditation standards would impose an unnecessary burden on those who successfully perform their own calibrations but are not accredited by ANSI/NCSL or ISO/IEC”.

2016 NCSL International Workshop & Symposium

The FDA also commented on the use of measurement uncertainty and test accuracy/uncertainty ratios. FDA, Response to Comment 9: Docket FDA-2007-N-0265 (2011): [173A] “One comment expressed concern that the proposed rule did not mention measurement uncertainties or test accuracy ratio, which are essential parameters for assuring an accurate calibration that are specified in standards issued by the American National Standards Institute (ANSI) and the International Organization for Standardization (ISO) for certification of calibration laboratories. The comment stated that the ANSI and ISO standards provide a limit for measurement uncertainty and establish a minimum test accuracy ratio that is commonly used by calibration facilities. According to the comment, although the proposed rule requires use of a calibrated accurate reference device, the lack of specific calibration parameters may lead to inaccurate calibrations for temperature indicating devices. (Response) …We do not agree that the regulations should specify calibration parameters, such as those relating to measurement uncertainties or test accuracy ratio, or require use of specific calibration standards, such as the ANSI and ISO standards suggested by the comment. Metrology authorities, in addition to ANSI and ISO, issue calibration standards, which may be revised or replaced. It would be impractical for FDA to maintain in the regulations a current list of acceptable calibration standards.”

However, as stated previously, certain branches of the FDA do indeed maintain a list of recognized consensus standards [84] [85] [86], as does the IMDRF [87] [88]. Both the NTTAA [146] and the Food and Drug Modernization Act encourage the use of such standards. Z540 has existed for 22 years (1994 to 2016) with a single revision in 2006 when Z540.3 replaced Z540.1. Likewise, ISO 17025 has existed for 17 years (1999 to 2016) with a single revision in 2005. While hundreds of documentary calibration standards exist for individual equipment types (i.e., calibration procedures/methods), Z540.3 and ISO 17205 are the two standards for calibration systems. Revision and/or replacement of these standards has been relatively infrequent for nearly two decades. Z540.3 addresses how measurement uncertainty and test uncertainty ratio (TUR) can influence false accept/reject decisions for all calibration and measurement or test processes, resulting in incorrect inor-out of tolerance decisions (see also appendices A & B). The U.S. Pharmacopeia (USP) has also recognized the importance ISO 17025 and of test accuracy (or uncertainty) ratios [173B] [173C] [173D] [174E], regarding their effect on the risk of making false accept and false reject decisions. USP Subcommittee on Certified Reference Materials [RM] (2007): [173B] “The impact of an RM uncertainty, unless it is negligible, is that it will alter the likelihood that a quality control laboratory will make an incorrect administrative decision – passing an item that does not meet its acceptance criteria (consumer risk) or failing to pass an item that does (producer risk)… A criterion… can be considered… in terms of the relationship between the expanded uncertainty and the acceptance criteria… This ratio is sometimes referred to as the test accuracy ratio (TAR) or test uncertainty ratio... The choice of 4:1 is commonly used and was part of MIL-STD 45662A… …ASME [B89.7.3.1-2001]... suggests a range of 10:1 to 3:1, and 4:1 and 3:1 have been more commonly used in recent years. Based on these considerations, a choice of 4:1 seems to be a reasonable default choice and is the TAR used by the European Pharmacopeia (U. Rose, written communication, June 2007)”. "A TAR of 4:1 is most commonly used and is USP’s choice” [173C].

Along with historical instrument reliability data (EOPR) [37] [38] [39] [39A] [A96], few aspects of a calibration program or measurement science are more important than test uncertainty ratio for ensuring that measuring instruments are “suitable for their intended use and capable of providing valid results”. 2016 NCSL International Workshop & Symposium

8. Summary and Conclusion The formal origins of modern quality system documentary standards were born in the military and aerospace industries of the 1950’s, of which, calibration systems were an integral part. Over time, these military quality standards evolved into ISO 9001 with a multitude of organizations choosing to become registered. However, specific language prescribing technical competency requirements for calibration was not included in ISO 9001 (or similar standards such as ISO 13485). When the FDA introduced equipment calibration requirements into the Good Manufacturing Practices in 1978 (codified in 21 CFR 820.61), the calibration requirements were based to some extent on MIL-C-45662 from 1960; however, no accuracy ratio or uncertainty requirements for decision rules were specified. The FDA was a member of the NCSL TQM committee and a contributor to Z540.1, which replaced MIL-STD-45662A; both standards required a 4:1 test accuracy ratio. Since then, standards for calibration systems have evolved considerably, the latest of which (Z540.3) is a direct descendent of these earlier standards and focuses on limiting the probability of incorrect acceptance decisions. Companies registered to ISO 9001 or ISO 13485 are not necessarily required to demonstrate that measuring equipment has been calibrated in a technically competent manner or that the risks of erroneously accepting non-conforming instruments have been adequately controlled. Essentially, only basic documentation requirements for calibrated equipment are required by ISO 9001, ISO 13485, and 21 CFR §820.72. This is contrasted with other federal agencies or regulated industries which employ ISO 17025 and/or Z540.3 as guidance or requirements documents to ensure technical competence of calibrations. Thus, there exists a significant disparity in the implementation of voluntary consensus standards for calibration requirements among federal (or federally-regulated) entities, many of which are engaged in critical operations which may affect the health and welfare of the general public. The National Technology Transfer and Advancement Act of 1995 was instituted to encourage federal agencies to adopt voluntary consensus standards for use in carrying out their responsibilities and regulatory functions. Federal agencies are required to use such standards in lieu of “government unique” standards, where such use is consistent with the agencies’ mission and objectives. Adoption of such voluntary consensus standards harnesses the expertise of the private sector to aid in carrying out agency objectives. The DOD, NASA, the DOE, the Nuclear Regulatory Commission, the Federal Aviation Administration, and the automotive industry are examples of federal agencies, or industries regulated by them, where ISO 17025 and/or Z540.3 have been officially recognized and adopted for use in demonstrating that equipment has been calibrated in a technically competent manner. Not all federal agencies have utilized these standards, as guidance documents or otherwise, and some agencies rely on government unique standards to prescribe equipment calibration requirements. Many federal agencies formally recognize calibrations which have been performed by a laboratory accredited to ISO 17025 or Z540.3. Accreditation bodies which are signatories to the ILAC MRA enjoy a wide degree of acceptance. Global recognition of accreditation bodies is facilitated via peer review by other MRA signatories to ensure that all members are competent to accredit calibration laboratories. Laboratory accreditation is a mature system, supported by an infrastructure of ILAC policy documents on important issues such as metrological traceability, measurement uncertainty, calibration measurement capability, decision rules, etc. These policies leverage the significant contributions and expertise of groups such as the Joint Committee on Guides in Metrology (JCGM), consisting of eight prominent scientific organizations representing diverse areas of chemistry, physics, legal metrology, clinical chemistry, laboratory medicine, etc. The question could be posed, “If critical government/regulated industries do not use these standards for guidance, or do not formally recognize accredited calibration laboratories, then who should?” 2016 NCSL International Workshop & Symposium

1950

MIL-STD-120 1955

No Cal Requirements

1978

FDA 21 CFR §820.61 (GMP)

MIL-C-45662 MIL-HDBK-52

NATO AQAP-6 NATO AQAP-7 1973

1997

1976

DEFSTAN 05-26 DEFSTAN 05-27

FDA 21 CFR §820.72

DEFSTAN 05-32

Inspection, Measuring, and Test Equipment

1979

ISO 10012:2003 Measurement Management Systems: Requirements for Measurement Processes and Measuring Equipment

ISO CERTICO ISO 9000 Series (Quality Standards)

BS 5781-1 BS 5781-2

1971

~1977 [1]

(Now CASCO)

NTTAA (1995) Guidelines

“…all Federal agencies and departments shall use technical standards that are developed or adopted by voluntary consensus standards bodies…”

NCSL RP-2 NCSL RP-4 NCSL RP-11 1987

ANSI/ASQC M1

Public1978 Law 104-113

ISO/IEC 17025:2005 General Requirements for the Competence of Testing and Calibration Laboratories

ISO Guide 25 ASTM E548 UKAS M10 BS 6460-1 HB 18.25 BS 7501/7502 EN 45001/45002

1994

ANSI/NCSL Z540.1 & Z540.1 Handbook

ANSI/NCSL Z540.3-2006 Requirements for the Calibration of Measuring and Test Equipment

Figure 4. Should federal agencies formally recognize voluntary consensus standards for calibration?

2016 NCSL International Workshop & Symposium

(for calibration)

1960

1970

(for calibration)

FDA 21 CFR §133.4 (GMP)

Voluntary Consensus Standards

Army: AR 750-25 Air Force: AFR 74-2 & BU-520 Navy: SLIM & SecNav 4355.11 Navy: Tech Memo 63-106

Government Standards

1963

This paper has been an attempt to provide relevant background information for federal agencies, accreditation bodies, auditors, metrologists and quality personnel involved with calibration actives which may benefit from the use of voluntary consensus standards Z540.3 and/or ISO 17025. Federal agencies may derive efficiencies from the sharing of information and by maintaining a broad awareness of inter-agency efforts to achieve common goals. Objective knowledge of calibration programs among different agencies provides cross-sector insight to solutions implemented outside of one’s own office of responsibility. Many common challenges may have already been addressed by committees, colleagues, or counterparts in neighboring government agencies. Generating, managing, and regulating calibration program requirements from the ground up can be costly and inefficient, particularly when an elegant framework may already exist. If developed independently and without the benefit of lateral awareness, these individual programs can become splintered, yielding incompatibilities without mutual recognition and/or acceptance. There is a deep and rich history of the development of calibration standards and guidelines which have found application among regulated and non-regulated industries alike. When consideration is extended to the implications of the National Technology Transfer Advancement Act, the use of such standards becomes an attractive method of improving calibration quality while reducing the reliance on government-unique standards for the prescription of calibration requirements and/or guidance. Utilizing such consensus standards as guidance documents can mitigate the risk of incorrect decisions during calibration and subsequent measurement processes. Standards such as Z540.3 provide a consistent approach for ensuring that measuring processes/instruments are “suitable for their intended use” by managing false-accept risks and incorrect decisions to acceptably low levels. The U.S. Pharmacopeia has also embraced a similar philosophy to limiting incorrect measurement decisions via target uncertainties & TUR. Standards such as ISO 17025 could provide utility in the implementation of the Medical Device Single Audit Program (MDSAP) by streamlining acceptance of calibrations in the medical industry on an international/global scale via recognition of accredited laboratories. Complying with government-unique calibration standards has been a challenge for industry dating back to the 1960’s. Reliance and recognition of voluntary consensus standards for calibration has many benefits for both government and industry alike and can provide utility in determining “suitability for intended use” by mitigating measurement decision risk. Leveraging these standards as guidance documents, rather than requirements documents, along with formal recognition of accredited calibration laboratories could foster a degree of harmonization without undue industry burden. In the 21st century “risk-based” environment, the approach to calibration incorporated into Z540.3 is complementary to this risk-based philosophy. Consensus standards for calibration provide a framework for making sound decisions throughout the lifecycle of products and instruments, from design through end-use application. 9. Acknowledgement and Disclosures The author thanks Nancy Mescher and Tom Waltrich of Baxter Healthcare Corporation for their support and guidance in a multitude of healthcare metrology issues, many of which have been discussed in this paper. Much gratitude is extended to Shawn Schmitt at Informa, Marcus McNeely at Blue Mountain Quality Resources, and Roger Burton of Sandia National Laboratories for permission to reproduce previously published material in its entirety. Rules, regulations, and guidance discussed in this paper are provided for reference only, are not authoritative, and may be outdated or withdrawn at the time of publication. Original source documents should be consulted for use in any official capacity. While every effort has been made to ensure accuracy of information, it is presented “as-is” without any implied warranty or suitability. The opinions and inferences provided in this paper are exclusively those of the author and do not necessarily reflect those of any agency or entity. 2016 NCSL International Workshop & Symposium

10. Bibliography [1] Code of Federal Regulations, Title 21 – Food and Drugs: [1A] Part 820 – Quality System Regulation, Subpart G – Production and Process Controls, Section §820.72 – Inspection, Measuring, and Test Equipment, paragraphs (a/b/1/2) [1B] Part 211 – Current Good Manufacturing Practice for Finished Pharmaceuticals, Subpart C- Buildings and Facilities, Section §211.68 – Automatic, Mechanical, and Electronic Equipment, Paragraph (a) [1C] Part 211 – Current Good Manufacturing Practice for Finished Pharmaceuticals, Subpart I- Laboratory Controls, Section §211.160 – General Requirements, Paragraph (b/4) [1D] Part 606 – Current Good Manufacturing Practice for Blood and Blood Components, Subpart D – Equipment, Section §606.60 – Equipment, Paragraphs (a/b/c) [1E] Part 58 – Good Laboratory Practice for Nonclinical Laboratory Studies, Subpart D – Equipment, Section §58.63 – Maintenance and Calibration of Equipment, Paragraphs (a/b/c) [1F] Part 106 – Infant Formula Requirements Pertaining to Current Good Manufacturing Practice, Quality Control Procedures, Quality Factors, Records and Reports, and Notifications, Subpart B – Current Good Manufacturing Practice, Section §106.30 – Controls to Prevent Adulteration Caused by Equipment or Utensils, Paragraphs (d 1/2/3) [2] MIL-STD-120, Military Standard: Gage Inspection. U.S Department of Defense. Washington D.C. December 1950. [3] H. Castrup, A Note on the Accuracy Ratio Requirements. Integrated Science Group. Bakersfield CA. June 2006. [4] J. Condon, NASA Calibration Practices. Proceedings of the 1966 Standards Laboratory Conference, National Conference of Standards Laboratories – United States Department of Commerce – National Bureau of Standards, NBS Miscellaneous Publication 291, pp 119 – 120, Washington DC, May 1966. [5] F. Russell, Industry’s View of the 10:1 Ratio-of-Accuracy Requirement. Proceedings of the 1966 Standards Laboratory Conference, National Conference of Standards Laboratories – United States Department of Commerce – National Bureau of Standards. NBS Special Publication 313, pp 121 – 123, Washington DC. May 1966. [6] M. Fruechtenicht, Metrology and Calibration in DOD Quality and Reliability Operations. Making Valuable Measurements - Proceedings of the 1968 Standards Laboratory Conference, National Conference of Standards Laboratories. United States Department of Commerce – National Bureau of Standards, pp 11 – 13, Washington DC. August 1968. [7] MIL-C-45662A, Military Specification: Calibration System Requirements. U.S Department of Defense. February 1962. [8] A. Lowery, Device Good Manufacturing Practices – A Workshop Manual. U.S. Department of Health and Human Services – Public Health Service – Food and Drug Administration – Bureau of Medical Devices – Office of Small Manufacturers Assistance. Chapter 4: Buildings, Equipment and Calibration. October 1982. [9] W. F. Hooten, Measurement Assurance and Medical Devices. National Conference of Standards Laboratories. NCSL Newsletter. Vol. 20, No. 4, pp 36 – 37. Boulder CO. December 1980. 2016 NCSL International Workshop & Symposium

[10] 10 CFR 50, Code of Federal Regulations. Title 10 – Energy, Part 50 – Domestic Licensing of Production and Utilization Facilities, Appendix B – Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants, Basic Requirement XII – Control of Measuring and Test Equipment. [11] 14 CFR §145.109, Code of Federal Regulations. Title 14 – Aeronautics and Space – Chapter I – Federal Aviation Administration – Department of Transportation, Subchapter H – Schools and Other Certificated Agencies, Part 145 – Repair Stations, Subpart C – Housing, Facilities, Equipment, Materials, and Data, §145.109 – Equipment, Materials, and Data Requirements. [12] MIL-C-45662 (Ord), Military Specification: Calibration of Standards. U.S Department of Defense. Washington D.C. March 1960. [13] MIL-HDBK-52, Military Handbook: Evaluation of Contractor’s Calibration System. U.S Department of Defense. Washington D.C. July 1964. [14] NATO AQAP-6, Measurement and Calibration System Requirements for Industry. North Atlantic Treaty Organization. Brussels Belgium. December 1970. [15] NATO AQAP-7, Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with AQAP-6. North Atlantic Treaty Organization. Brussels Belgium. October 1978. [16] DEFSTAN 05-26/1:1973, Measurement and Calibration System Requirements for Industry. British Ministry of Defence. London England. 1973. [17] DEFSTAN 05-27/1:1973, Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with Def Stan 05-26/2. British Ministry of Defence. London England. 1973. [18] BS 5781-1:1979, Measurement and Calibration Systems. Specification for System Requirements. British Standards Institute. London England. October 1979. [19] BS 5781-2:1981, Quality Assurance Requirements for Measuring Equipment. Guide to the Use of BS 5781-1 ‘Specification of System Requirements’. British Standards Institute. London England. February 1981. [20] ISO 10012-1:1992, Quality Assurance Requirements for Measuring Equipment – Part 1: Metrological Confirmation System for Measuring Equipment. International Organization for Standardization. Geneva Switzerland. January 1992. [21] ISO 10012-2:1997, Quality Assurance for Measuring Equipment – Part 2: Guidelines for Control of Measurement Process. International Organization for Standardization. Geneva Switzerland. January 1997. [22] ISO 10012:2003, Measuring Management Systems – Requirements for Measurement Processes and Measuring Equipment. International Organization for Standardization. Geneva Switzerland. April 2003. [23] ISO Guide 25-1978(E), Guidelines for Assessing the Technical Competence of Testing Laboratories, first edition. International Organization for Standardization. Geneva Switzerland. October 1978. [24] ILAC/78, Summary of Proceedings – Second International Conference on Recognition of National Programs for Accreditation of Testing Laboratories. United States Department of Commerce – Office of Product Standards. Washington D.C. October 1978. 2016 NCSL International Workshop & Symposium

[25] ISO/IEC Guide 25:1990(E), General Requirements for the Competence of Calibration and Testing Laboratories, third edition. International Organization for Standardization. Geneva Switzerland. December 1990. [26] ISO/IEC 17025:1999(E), General Requirements for the Competence of Testing and Calibration Laboratories, first edition. International Organization for Standardization. Geneva Switzerland. December 1999. [27] ISO/IEC 17025:2005(E), General Requirements for the Competence of Testing and Calibration Laboratories, second edition. International Organization for Standardization. Geneva Switzerland. May 2005. [28] ISO/IEC 17025 CD2, General Requirements for the Competence of Testing and Calibration Laboratories. CASCO WG44 N75, Committee Draft 2. International Organization for Standardization. Geneva Switzerland. March 2016. [29] MIL-STD-45662A, Military Standard: Calibration Systems Requirements. U.S Department of Defense. Washington D.C. August 1988. [30] MIL-STD-45662, Military Standard: Calibration Systems Requirements. U.S Department of Defense. Washington D.C. June 1980. [31] MIL-HDBK-52A, Military Handbook: Evaluation of Contractor’s Calibration System. U.S Department of Defense. Washington D.C. August 1984. [32] MIL-STD-45662 Notice 2, Notice of Cancellation. U.S Department of Defense. Washington D.C. February 1995. [32A] R. Schumacher, New American National and International Standards on Calibration Control. Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. Albuquerque NM. August 1991. [32B] R. Schumacher, Metrology Standards – In ANSI and ISO. Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. Dallas TX. July 1995. [33] T. Held, Metrology and Calibration Journal. Issue No. 13. Abbot Laboratories. Abbott Park IL. September 1992. [34] R. Randall, ISO 9000 – Questions & Answers. GE Electronic Services. Buford GA. June 1991, Revised April 1992. [35] ANSI/NCSL Z540.1-1994, American National Standard for Calibration – Calibration Laboratories and Measuring and Test Equipment – General Requirements. American National Standards Institute / National Conference of Standards Laboratories. Washington D.C. / Boulder CO. July 1994 [36] J. Cigler, NVLAP Calibration Laboratory Program. Proceedings of the Workshop on Measurement Quality Assurance for Ionizing Radiation. International Atomic Energy Agency (IAEA) – International Nuclear Information System (INIS) Vol. 27, No. 19. Report No. PNL10076. March 1993. (Note: Similar version presented at 1993 NCSL Workshop & Symposium). [36A] L. Neumann, A History of Laboratory Accreditation in the United States. Cal Lab Magazine. Vol. 8, No. 3, pp 22 – 25. July/August/September 2001. [36B] NIST SP-986, Proceedings of the NIST Conformity Assessment for a Changing Government Workshop. NIST Special Publication 986. U.S. Department of Commerce –

2016 NCSL International Workshop & Symposium

National Institute of Standards and Technology – Standards Services Division. Gaithersburg MD. May 2003. [37] P. Reese, J. Harben. Implementing Strategies for Risk Mitigation in the Modern Calibration Laboratory, Proceedings of the NCSL International Workshop and Symposium, National Harbor MD. August 2011. [38] P. Reese, J. Harben. Risk Mitigation Strategies for Compliance Testing, Measure – The Journal of Measurement Science, NCSL International Vol.7, No.1 pp 38-49. March 2012 [39] S. Mimbs, Using Reliability to Meet Z540.3’s 2 % Rule. Proceedings of the NCSL International Workshop and Symposium. National Harbor, MD. August 2011. [39A] NCSLI, Request for Interpretation of ANSI/NCSL Z540.3-2006. NCSLI 174 Accredited Standards Committee. NCSL International. Boulder CO. 2011. [40] ISO/IEC Guide 25 Draft V, General Requirements for the Competence of Calibration and Testing Laboratories, Draft 5 (unpublished). August 1996. [41] ISO/IEC 17025 CD1, General Requirements for the Competence of Testing and Calibration Laboratories. CASCO WG44, Committee Draft 1 (unpublished). International Organization for Standardization. Geneva Switzerland. September 2015. [42] ISO/IEC Guide 98-4, Uncertainty of Measurement – Part 4: Role of Measurement Uncertainty in Conformity Assessment. First edition. International Organization for Standardization. Geneva Switzerland. 2012. [43] JCGM 106:2012, Evaluation of Measurement Data – The Role of Measurement Uncertainty in Conformity Assessment. Joint Committee for Guides in Metrology – BIPM, IEC, IFCC, ILAC, ISO, IUPAC, IUPAP, OIML. October 2012. [44] ANSI/NCSL Z540.3-2006. Requirements for the Calibration of Measuring and Test Equipment. American National Standards Institute / NCSL International, 2006. [45] NCSL Z540.3-2006 Handbook, Handbook for the Application of ANSI/NCSL Z540.3-2006 – Requirements for the Calibration of Measuring and Test Equipment. NCSL International. February 2009. [46] NCSLI RP-18, Recommended Practice – Estimation and Evaluation of Measurement Decision Risk. NCSL International. Boulder CO. July 2014. [47] NCSLI RP-21, Recommended Practice for the Assessment of ANSI/NCSL Z540.3-2006 Subclause 5.3. NCSL International. Boulder CO. July 2014. [48] D. Caldwell, ANSI/NCSL Z540.3:2006: Requirements for the Calibration of Measuring and Test Equipment. NCSLI Measure – The Journal of Measurement Science. Vol. 1, No. 4, pp 26 – 30. December 2006. [49] S. Mimbs, Measurement Decision Risk - The Importance of Definitions. Proceedings of the NCSL International Workshop and Symposium. St. Paul MN. July 2007. [49A] S. Mimbs, Measurement Decision Risk - The Importance of Definitions. Proceedings of the Measurement Science Conference. Anaheim CA. March 2008. [50] ISO, Risk-Based Thinking in ISO 9001:2015. Technical Committee 176, Sub-Committee 2. Document N1269. International Organization for Standardization. Geneva Switzerland. 2015

2016 NCSL International Workshop & Symposium

[51] S. Weber, A. Hillstrom, Economic Model of Calibration Improvements for Automatic Test Equipment. NBS Special Publication 673. U.S. Department of Commerce – National Bureau of Standards. April 1984. [52] NASA RP-1342. H. Castrup, W. Eicke, J. Hayes, A. Mark, R. Martin, J. Taylor. Metrology – Calibration and Measurement Processes Guidelines. National Aeronautics and Space Administration. Jet Propulsion Laboratory. Reference Publication 1342. Pasadena CA. June 1994. [53] ASME B89.7.3.1-2001, Guidelines for Decision Rules: Considering Measurement Uncertainty in Determining Conformance to Specifications. American Society of Mechanical Engineers. New York NY. March 2002. [54] UKAS LAB34, The Expression of Uncertainty in EMC Testing. United Kingdom Accreditation Service. Middlesex UK. August 2002. [55] ISO 10576-1:2003, Statistical Methods – Guidelines for the Evaluation of Conformity With Specified Requirements – Part 1: General Principles. First edition. International Organization for Standardization. Geneva Switzerland. March 2003. [56] ASME B89.7.4.1-2005 (Technical Report), Measurement Uncertainty and Conformance Testing: Risk Analysis. American Society of Mechanical Engineers. New York NY. February 2006. [57] Eurachem/CITAC Guide, Use of Uncertainty Information in Compliance Assessment. First edition. Eurachem / Co-Operation on International Traceability in Analytical Chemistry (CITAC). Leoben Austria. 2007 [58] Eurolab Cook Book Doc No. 8.0, Determination of Conformance with Specifications or Limit Values with Particular Reference to Measurement Uncertainties – Possible Strategies. Brussels Belgium. May 2008. [59] ILAC-G8:03/2009, Guidelines on the Reporting and Assessment of Compliance with Specification. International Laboratory Accreditation Cooperation. 2009 [60] NASA HDBK 8739.19-4, Estimation and Evaluation of Measurement Decision Risk: NASA Measurement Quality Assurance Handbook – ANNEX 4. National Aeronautics and Space Administration. Washington D.C. July 2010. [61] VDA-5, Capability of Measurement Processes, Quality Management in the Automobile Industry – Volume 5. Verband der Automobilindustrie (German Association of the Automobile Industry). July 2011 [62] AIAG MSA-4, Measurement System Analysis – Reference Manual. Fourth edition. ISBN 978-1-60-534211-5. Chrysler Group LLC, Ford Motor Company, General Motors Corporation. Automotive Industry Action Group. Southfield MI. June 2010. [63] UKAS:M3003, The Expression of Uncertainty and Confidence in Measurement, Appendix M – Assessment of Compliance With Specification. Third edition. United Kingdom Accreditation Service. Middlesex UK. November 2012. [64] ISO 22514-7:2012, Statistical Methods in Process Management – Capability and Performance – Part 7: Capability of Measurement Processes. First edition. International Organization for Standardization. Geneva Switzerland. March 2003.

2016 NCSL International Workshop & Symposium

[64A] Minitab 17 Support, A Type 1 Gage Study Assesses the Capability of a Measurement Process. Minitab 17 Support. Topic Library – Quality Tools – Measurement System Analysis – Other Gage Studies and Measures – Type 1 Gage Study. Minitab Inc. Retrieved May 2016. [65] ISO-14253-1:2013(E), Geometrical Product Specifications (GPS) - Inspection by Measurement of Workpieces and Measuring Equipment - Part 1: Decision Rules for Proving Conformance or Non-conformance with Specifications. International Organization for Standardization. Geneva Switzerland. 2013. [66] WADA TD2014DL, Decision Limits for the Conformity Quantification of Threshold Substances. Technical Document TD2014DL, Version 1.0. World Anti-Doping Agency. Quebec Canada. September 2014. [67] FDA, Guide to Inspections of Medical Device Manufacturers. Section 8, pg 19, Inspection Measuring and Test Equipment 21 CFR 820.72. U.S. Department of Health and Human Services. Food and Drug Administration. Silver Spring MD. December 1997. [68] USP , Analytical Instrument Qualification. U.S. Pharmacopeia National Formulary. USP 39, NF 34. General Chapter . United States Pharmacopeial Convention. Rockville MD. May 2016. (hyperlink is for previous edition).

[69] ISPE GAMP-5, Good Practice Guide – A Risk-Based Approach to GxP Compliant Laboratory Computerized Systems. Second edition. ISBN 978-1-936379-49-1. International Society for Pharmaceutical Engineering (ISPE). Community of Practice (COP) Technical Subcommittee on Good Automated Manufacturing Practice (GAMP). Tampa FL. 2012. [70] L.Vuolo-Schuessler, M. Newton, P. Smith, C. Burgess, R. McDowall, Harmonizing USP and GAMP for Analytical Instrument Qualification. Pharmaceutical Engineering. International Society for Pharmaceutical Engineering (ISPE). Vol. 34, No. 1, pp 1 – 9. January/February 2014. [71] FDA, Code of Federal Regulations, Title 21 – Food and Drugs, Part 133 Drugs; Current Good Manufacturing Practice in Manufacture, Processing, Packing, or Holding. Section §133.4 – Equipment. Office of the Federal Register, National Archives and Records Service, General Services Administration. Washington DC. January 1965. [72] FDA, Code of Federal Regulations, Title 21 – Food and Drugs: Subchapter H – Medical Devices. Part 820 – Good Manufacturing Practice for Medical Devices. Subpart D – Equipment, Section §820.61 – Measurement Equipment. Federal Register, 43 FR 31 508. July 1978. (hyperlink is for a subsequent edition).

[73] AAMI, Quality System Compendium, GMP Requirements & Industry Practice. Chapter 15 – Inspection, Measuring, and Test Equipment (Subpart G). First edition. Association for the Advancement of Medical Instrumentation (AAMI). ISBN 1-57020-076-9. Arlington VA. 1998 [74] ANSI/ASQC M1-1987, American National Standard for Calibration Systems. American Society for Quality Control. Milwaukee WI. June 1987. (Reaffirmed in February 1996 as ANSI/ASQC M1-1997). [75] FDA, Guidance for Industry PAT – A Framework for Innovative Pharmaceutical Development, Manufacturing, and Quality Assurance. U.S. Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research. Center for Veterinary Medicine. Office of Regulatory Affairs. Rockville MD. September 2004. [76] ISPE/GAMP, Good Practice Guide: A Risk-Based Approach to Calibration Management. Second edition. ISBN 978-1-936379-00-2. International Society for Pharmaceutical Engineering 2016 NCSL International Workshop & Symposium

(ISPE) – Community of Practice (COP) Technical Subcommittee on Good Automated Manufacturing Practice (GAMP). Tampa FL. 2010. [77] NCSL RP-6, Recommended Practice for Calibration Quality Systems for the Healthcare Industries. NCSL International. ISBN 978-1-58464-069-1. Boulder CO. 2015. [78] M. McNeely, An Overview of the NCSLI 151 Healthcare Metrology Committee. Proceedings of the NCSL International Workshop and Symposium. Nashville TN. July 2013. [79] FDA, Code of Federal Regulations, Title 21 – Food and Drugs: Chapter I – Food and Drug Administration, Department of Health and Human Services, Subchapter A – General, Part 10, Administrative Practices and Procedures, Subpart B – General Administrative Procedures, Section §10.115 – Good Guidance Practices. Federal Register 65 FR 56477. September 2000. [80] P. James, Comments to the Proposed Rule: Medical Devices; Refurbishers, Servicers, and ‘As Is’ Remarketers of Medical Devices; Review and Revision of Compliance Policy Guides and Regulatory Requirements; Request for Comments and Information. FDA Docket No. 97N-0477. Food and Drug Administration. February 1998. [81] GHTF.SG3.N99-8, Guidance on Quality Systems for the Design and Manufacture of Medical Devices. Section 4.11 – Control of Inspection, Measuring, and Test Equipment. pg 31. Global Harmonization Task Force (GHTF), Study Group 3. June 1999 [82] ISO 13485:2016, Medical Devices – Quality Management Systems – Requirements for Regulatory Purposes. International Organization for Standardization. Geneva Switzerland. March 2016. [82A] IAF B5, The IAF Initiative for Accredited Certification to ISO 13485 – Medical Devices. International Accreditation Forum. Quebec Canada. January 2012. [83] MDSAP G0002.1.003, Medical Device Single Audit Program Companion Document. Chapter 6 – Process: Production and Service Controls. Audit Task 13 – Maintenance and Calibration, pp 69 - 70. International Medical Device Regulators Forum (undated). [84] FDA, Recognized Consensus Standards - Databases. U.S. Department of Health and Human Services, Food and Drug Administration. CDRH Standards Program. Rockville MD. May 2016. [85] FDA, Federal Register Documents, Modifications to the Initial List of Recognized Standards.. U.S. Department of Health and Human Services, Food and Drug Administration. Rockville MD (continuously updated). [86] FDA, Guidance for Industry and Staff – Recognition and Use of Consensus Standards. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, Standards Management Staff, Office of Science and Engineering Laboratories. September 2007. [87] IMDRF N15, Final Report: List of International Standards Recognized by IMDRF Members as of March 2014. IMDRF Standards Working Group Document N15. International Medical Device Regulators Forum. September 2014. [88] IMDRF N15 (USA list), Final Report: List of International Standards Recognized by IMDRF Members as of March 2014. IMDRF Standards Working Group Document N15. International Medical Device Regulators Forum. September 2014.

2016 NCSL International Workshop & Symposium

[89] ISO/TR 14969:2004, Medical Devices – Quality Management Systems – Guidance on the Application of ISO 13485:2003. Section 7.6 – Control of Monitoring and Measuring Devices. International Organization for Standardization. Geneva Switzerland. 2004. [90] C. Grim, The Cost of Inaccurate Blood Pressure Measurement. Summary Report – National High Blood Pressure Education Program (NHBPEP)/National Heart, Lung, and Blood Pressure Institute (NHLBI) and American Heart Association (AHA) Working Meeting on Blood Pressure Measurement. pp 30 – 31. National Institutes of Health. Bethesda MD. April 2002. [91] M. Lozy, Simple Computation of a Bivariate Normal Integral Arising From a Problem of Misclassification with Applications to the Diagnosis of Hypertension. Communications in Statistics – Theory and Methods, Vol. 11, No. 19, pp 2195-2205. 1982. [91A] C. Friedman-Gerlicz, I. Lilly, Misclassification Rates in Hypertension Diagnosis Due to Measurement Errors. Society for Industrial and Applied Mathematics (SIAM) , Undergraduate Research Online (SIURO). Vol. 2, No. 2. August 2010. [92] R. Greaves, Common Reference Intervals. 4th Vietnam Chemical Pathology Course. Roche Diagnostics. Ho Chi Minh City, Vietnam. July 2012. [93] R. Forsman, Why is the Laboratory an Afterthought for Managed Care Organizations? Clinical Chemistry, Vol. 42, No. 5, pp 813-816. May 1996. [94] I. Kuselman, A. Weisman, W. Wegscheider, Traceability Without Uncertainty: Current Situation in the Pharmaceutical Industry. Accreditation and Quality Assurance. Vol. 8, No. 11, pp 530-531. November 2003. [95] E. Simpson, FDA Standards on Calibrating Lab Equipment Are Not So Tough”. Engineering.com – Designer Edge. January 2015. [96] U.S. Department of Health and Human Services, Food and Drug Administration, Form 483 Observation #11 – Control of Inspection, Measuring, and Test Equipment, Commander S. Creighton, Consumer Safety Officer. Issued to St. Jude Medical IESD, Sylmar CA. October 17, 2012.  

Response (November 7, 2012) – Observation #11 Response (March 13, 2013) – Observation #11

[97] JCGM 200:2012 (ISO/IEC Guide 99-12:2007). International Vocabulary of Metrology — Basic and General Concepts and Associated Terms (VIM). Joint Committee for Guides in Metrology - Working Group 2, 3rd Edition. 2008. [98] P. Reese, Instrument Adjustment Policies. Proceedings of the 2015 NCSL International Workshop and Symposium. Grapevine TX. July 2015. [99] J. Bucher, Measure for Measure - Out of Sync. The Standard. American Society for Quality (ASQ) Measurement Quality Division. Vol. 27, No. 2, pp 21-22. June 2013. [99A] The preceding paper was also published in ASQ Quality Progress, pp 52-53. March 2010. [100] J. Bucher, Debunking The Two Great Myths About Calibration: Traceability to NIST: If You Cannot Adjust, You Cannot Calibrate. Proceedings of the NCSL International Workshop and Symposium, National Harbor MD. Aug 2011. [101] J. Bucher., Where Does It Say That? Clearing Up the FDA’s Calibration Requirements. American Society for Quality, Measurement Quality Division, The Standard, Vol. 27 No. 2, PDF pp 31-32. June 2013. 2016 NCSL International Workshop & Symposium

[102] S. Schmitt, Pay Close Attention to Equipment Calibration Certificates, FDA Says. The Silver Sheet. Article # 09140061001. Informa Business Intelligence, Inc. June 2010. [103] NASA NPD 8730.1C, Nasa Policy Directive, Metrology and Calibration. National Aeronautics and Space Administration. Washington D.C. June 2011. [104] NASA Solicitation No. NNK07189729R, Institutional Services Contract (ISC). National Aeronautics and Space Administration. Kennedy Space Center FL. October 2008. [105] NASA Solicitation No. NNJ15532524R, Calibration and Metrology Services (CAMS) III Contract. National Aeronautics and Space Administration – Johnson Space Center. Houston TX. July 2015. [106] NASA Solicitation No. NNC14ZC013R, Technical Information Administration and Logistics Services (TIALS) Contract. National Aeronautics and Space Administration – Glenn Research Center. Cleveland OH. September 2014. [107] ANAB AC-1494, Certificate of Accreditation. Measurement Standards and Calibration Laboratory (MSCL), NASA – Johnson Space Center, Houston TX. ANSI-ASQ National Accreditation Board. December 2015. [108] Sandia, Sandia Primary Standards Laboratory, DOE/AL Approve Use of Accredited Commercial Calibration Laboratories. Sandia National Laboratories. Sandia Lab News. Vol. 53, No. 21, pg 9. Albuquerque NM. October 2001. [109] R. Pettit, Laboratory Accreditation. Report Sand-98-1739C, Conf-980822. Sandia National Laboratories – Primary Standards Laboratory. Albuquerque NM. August 1998. [110] ANSI/ASME NQA-1-1986, Nuclear Quality Assurance Program – Supplement 12 S-1, Supplementary Requirements for Control of Measuring and Test Equipment. American National Standards Institute / American Society of Mechanical Engineers. New York NY. 1986. [111] DOE, NQA-1: An Overview for Federal Project Directors. U.S. Department of Energy. Undated. [112] IEEE STD 498-1975, IEEE Standard Supplementary Requirements for the Calibration and Control of Measuring Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations. Institute of Electrical and Electronics Engineers. New York NY. July 1975. [113] IEEE STD 498-1980, IEEE Standard Requirements for the Calibration and Control of Measuring Equipment Used in the Construction and Maintenance of Nuclear Power Generating Stations. Institute of Electrical and Electronics Engineers. New York NY. October 1980. [114] IEEE STD 498-1985, IEEE Standard Requirements for the Calibration and Control of Measuring Equipment Used in Nuclear Facilities. Institute of Electrical and Electronics Engineers. New York NY. June 1985. [115] IEEE STD 498-1990, IEEE Standard Requirements for the Calibration and Control of Measuring Equipment Used in Nuclear Facilities. Institute of Electrical and Electronics Engineers. New York NY. January 1991. [116] NRC 35705, NRC Inspection Manual: Inspection Procedure 35750, QA Program Measuring and Test Equipment. LPEB Nuclear Regulatory Commission. Washington D.C. April 1992. [117] NUPIC Doc No. 28, Commercial Grade Calibration Service Checklist. Nuclear Procurement Issues Committee. Revision 2. Undated. 2016 NCSL International Workshop & Symposium

[118] NUPIC Doc No. 29, Commercial Grade Calibration Service Checklist Implementation Guidelines. Nuclear Procurement Issues Committee. Revision 2. August 2006. [119] NIST IR-6989, Comparison of ISO/IEC 17025 with the NUPIC Audit Checklist. Department of Commerce – National Institute of Standards and Technology Interagency/Internal Report. Gaithersburg MD. May 2003. [120] BLN – COL – FSAR, Final Safety Analysis Report – Chapter 17, Quality Assurance, Section 17.5.4.7. Bellefonte Nuclear Generating Station (BLN) – Combined License (COL). Nuclear Regulatory Commission. Hollywood AL. 2009. [121] P. Hiland, Letter from NRC to P. McCullen of International Accreditation Service Inc. confirming acceptance of IAS accredited calibrations for commercial-grade calibration suppliers. Nuclear Regulatory Commission – Office of Nuclear Reactor Regulation – Division of Engineering. Washington D.C. May 2008. [122] P. Hiland, Letter from NRC to R. Leonard of Laboratory Accreditation Bureau confirming acceptance of L-A-B accredited calibrations for commercial-grade calibration suppliers. Nuclear Regulatory Commission – Office of Nuclear Reactor Regulation – Division of Engineering. Washington D.C. April 2008. [123] P. Hiland, Letter from NRC to K. Greenaway of Assured Calibration and Laboratory Accreditation Select Services confirming acceptance of ACLASS [now ANAB] accredited calibrations for commercial-grade calibration suppliers. Nuclear Regulatory Commission – Office of Nuclear Reactor Regulation – Division of Engineering. Washington D.C. December 2007. [124] P. Hiland, Letter from NRC to T. Szerszen of Perry Johnson Laboratory Accreditation Inc. confirming acceptance of PJLA accredited calibrations for commercial-grade calibration suppliers. Nuclear Regulatory Commission – Office of Nuclear Reactor Regulation – Division of Engineering. Washington D.C. January 2010. [125] ILAC MRA, The ILAC Mutual Recognition Arrangement. International Laboratory Accreditation Cooperation. New South Wales Australia. 2016. [126] ILAC P4:02/2016, ILAC Mutual Recognition Arrangement: Policy and Management. International Laboratory Accreditation Cooperation. New South Wales Australia. 2016. [127] ILAC P5:02/2016, ILAC Mutual Recognition Arrangement: Scope and Obligations. International Laboratory Accreditation Cooperation. New South Wales Australia. 2016. [128] ILAC P8:02/2016, ILAC Mutual Recognition Arrangement: Supplementary Requirements and Guidelines for the Use of Accreditation Symbols and for Claims of Accreditation Status by Accredited Laboratories and Inspection Bodies. International Laboratory Accreditation Cooperation. New South Wales Australia. 2016. [129] NCSLI, Nuclear Power Industry Calibration Suppliers Confused About Requirements. NCSLI Measure – The Journal of Measurement Science. Vol. 5, No. 4, pp 14 – 17. NCSL International. Boulder CO. December 2010. [130] NEI 14-05A, Guidelines for the Use of Accreditation in Lieu of Commercial Grade Surveys for Procurement of Laboratory Calibration and Test Services.” Nuclear Energy Institute. Revision 0. Washington DC. March 2015. [131] AFD-130625-019, AFMETCAL (AFLCMC/WNM) Air Force Metrology and Calibration Traceability. U.S. Air Force, Wright Patterson Air Force Base OH. June 2013 2016 NCSL International Workshop & Symposium

[132] AR 750-43, Maintenance of Supplies and Equipment: Army Test, Measurement, and Diagnostic Equipment. Army Regulation 750-43, Headquarters – Department of the Army. Washington DC. January 2014. [133] R. Fritzsche, NACLA Navy Partnership. Naval Surface Warfare Center – Corona Division. Measurement Science Department. Corona CA. Undated. [134] R. Fritzsche, NACLA Value – Navy METCAL Perspective. Naval Surface Warfare Center – Corona Division. Measurement Science Department. Corona CA. Undated. [135] NACLA, Accreditation Body Evaluation Procedure for ANSI/NCSL Z540.3-2006 subclause 5.3. Rev A. National Cooperation for Laboratory Accreditation. Washington DC. April 2010. [136] A2LA R205a, Annex to Specific Requirements: ANSI/NCSLI Z540.3-2006. American Association Laboratory Accreditation. Frederick MD. June 2010. [137] L-A-B LABPR-401.1, METCAP Program Requirements Metrology and Calibration Accreditation Program ANSI/NCSL Z540.3-2006 subclause 5.3. Rev 3. Laboratory Accreditation Bureau. Fort Wayne IN. May 2011. [138] ANAB CL 2043, Z540.3 Checklist. ANSI-ASQ National Accreditation Board (formerly ACLASS). Milwaukee WI. Undated. [139] IAS CL/014, Policy Guide on Calibration, Traceability, and Measurement Uncertainty for Calibration Laboratories. International Accreditation Service, Inc. Brea CA. January 2016. [140] PJLA LF-1, Application / Questionnaire. Rev 1.10. Perry Johnson Laboratory Accreditation, Inc. Troy MI. November 2015. [141] ISO/TS 16949:2009(E), Quality Management Systems — Particular Requirements for the Application of ISO 9001:2008 for Automotive Production and Relevant Service Part Organizations. Third edition. International Organization for Standardization. June 2009. [141A] QS 9000, Quality System Requirements. Chrysler Corporation, Ford Motor Company, General Motors Corporation. Third edition. Auburn Hills / Allen Park / Flint MI. March 1998. [142] FAA 8900.1, Flight Standards Information System (FSIMS). Volume 6 – Surveillance, Chapter 9 – Part 145 Inspections, Section 9 – Safety Assurance System: Inspect a Part 145 Repair Station’s Tools and Equipment, Paragraph 6-1800 – Procedures. Chg 329. Federal Aviation Administration. August 2014. [143] FAA Order 6200.4G, National Test Equipment Program Management. Section 11 – Calibration of Test Equipment. U.S. Department of Transportation – Federal Aviation Administration. Washington D.C. March 2010. [144] FAA, Performance Work Statement – FAA Test, Measurement & Diagnostic Equipment Calibration Services. Mike Monroney Aeronautical Center – FAA Logistics Center. Federal Aviation Administration. Oklahoma City OK. January 2008. [145] NASA, System Failure Case Studies – The Unknown Known. Vol. 3, No. 1. National Aeronautics and Space Administration – NASA Safety Center (NSC). Cleveland OH. January 2009. [146] NTTAA, National Technology Transfer and Advancement Act of 1995. One Hundred Fourth Congress of the United States of America. H.R. 2196. Public Law 104-113. Washington D.C. March 1996. 2016 NCSL International Workshop & Symposium

[147] W. Perry, Specifications & Standards - A New Way of Doing Business (The Perry Memo). U.S. Secretary of Defense, White House National Security Council, Department of Defense. Washington DC. June 1994. [148] OMB Circular A-119, Federal Participation in the Development and Use of Voluntary Consensus Standards and in Conformance Assessment Activities. U.S. White House (Executive Office of the President) – Office of Management and Budget. Washington DC. February 1998 (obsolete). [149] OMB Circular A-119, Federal Participation in the Development and Use of Voluntary Consensus Standards and in Conformance Assessment Activities. U.S. White House – (Executive Office of the President) Office of Management and Budget. Washington DC. 2016 (revised). [150] OMB, Proposed Revisions to Circular A-119 – Public Comments. U.S. White House – (Executive Office of the President) Office of Management and Budget. Washington DC. 2014. [151] ACUS 78-4, Federal Agency Interaction with Private Stand-Setting Organizations in Health and Safety Regulation. Recommendation 78-4. Administrative Conference of the United States (ACUS). Adopted December 14-15, 1978. [152] ACUS 2011-5, Incorporation by Reference. Recommendation 2011-5. Administrative Conference of the United States (ACUS). Adopted December 8, 2011. [153] 1 CFR §51.1, Code of Federal Regulations. Title 1– General Provisions, Chapter II – Office of the Federal Register, Part 51 – Incorporation by Reference, Section 51.1 – Policy. [154] 46 CFR §162.060-36, Code of Federal Regulations. Title 46 – Coast Guard, Department of Homeland Security, Subchapter Q – Equipment, Construction, and Materials: Specifications and Approval, Part 162 – Engineering Equipment, Subpart 160.060 – Ballast Water Management Systems, §162.160.36 – Quality Assurance Project Plan (QAPP) Requirements. [155] 10 CFR §429.110, Code of Federal Regulations. Title 10 – Energy, Chapter II – Department of Energy, Subchapter D – Energy Conservation, Part 429 – Certification, Compliance, and Enforcement for Consumer Products and Commercial and Industrial Equipment, Subpart C – Enforcement, §429.110 – Enforcement Testing, paragraph (3). [156] 47 CFR §2.948, Code of Federal Regulations. Title 47 – Telecommunication, Chapter I – Federal Communications Commission, Subchapter A – General, Part 2 – Frequency Allocations and Radio Treaty Matters; General Rules and Regulations, Subpart J – Equipment Authorization Procedures, §2.948 – Description of Measurement Facilities, Paragraph (d). [157] FDA, Medical Devices; Current Good Manufacturing Practice (CGMP) Final Rule; Quality System Regulation. Federal Register. Vol. 61, No. 195, pp 52601 – 52662. Washington D.C. October 1996. [158] NIST 9th NTTAA Report, Ninth Annual Report of Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment. U.S. Department of Commerce – National Institute of Standards and Technology. Washington D.C. December 2006. [159] NIST IR7413, Addendum to the Tenth (2006) Annual Report of Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment. U.S. Department of Commerce – National Institute of Standards and Technology. Washington D.C. April 2007. [160] NIST IR7503, Addendum to the Eleventh (2007) Annual Report of Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment. U.S. Department of Commerce – National Institute of Standards and Technology. Washington D.C. May 2008. 2016 NCSL International Workshop & Symposium

[161] NIST IR7598, Addendum to the Twelfth (2008) Annual Report of Federal Agency Use of Voluntary Consensus Standards and Conformity Assessment. U.S. Department of Commerce – National Institute of Standards and Technology. Washington D.C. May 2009. [162] FDA, Guidance for Industry – Submission of Laboratory Packages by Accredited Laboratories. (draft). U.S. Department of Health and Human Services – Food and Drug Administration – Office of Regulatory Affairs and Office of Policy, Planning and Preparedness. Silver Spring MD. January 2009. [163] FDA, Laboratory Manual of Quality Policies – Volume 1: ORA Laboratory Manual of Quality Policies. Document No. ORA-LAB.QM, Version 2.0, Effective Date 10-01-03, Revised 05-02-14. U.S. Department of Health and Human Services – Food and Drug Administration – Office of Regulatory Affairs. Silver Spring MD. May 2014. [164] FDA: Allergenic Products Advisory Committee Meeting Background Document: ISO 17025 Accreditation of the Laboratory of Immunobiochemistry. U.S. Department of Health and Human Services – Food and Drug Administration. Silver Spring MD. May 2011. [165] S. Menzies, Laboratory of Immunobiochemistry and ISO 17025 Accreditation. (Presentation). U.S. Department of Health and Human Services, Food and Drug Administration. Center for Biologics Evaluation and Research (CBER). Laboratory of Immunobiochemistry. Allergenic Products Advisory Committee Meeting. Bethesda MD, May 2011. [166] S. Menzies, Laboratory of Immunobiochemistry and ISO 17025 Accreditation. (Meeting minutes). U.S. Department of Health and Human Services, Food and Drug Administration. Center for Biologics Evaluation and Research (CBER). Laboratory of Immunobiochemistry. Allergenic Products Advisory Committee Meeting. Bethesda MD, May 2011 [167] FDA, Advancing Regulatory Science: CBER Receives Accreditation for Laboratory Testing of Influenza Vaccines and Evaluation of Blood Donor Screening Kits. U.S. Department of Health and Human Services, Food and Drug Administration. Center for Biologics Evaluation and Research (CBER). Silver Spring MD. Undated. [168] FDA, ISO/IEC 17025:2005 Accreditation for State Food Testing Laboratories (U18) Cooperative Agreement Program.. U.S. Department of Health and Human Services, Food and Drug Administration, Division of Federal-State Relations (DFSR). Silver Spring MD. 2012. [169] FDA, ISO/IEC 17025:2005 Accreditation for State Food Testing Laboratories (U18). Funding Opportunity Announcement (FOA) RFA-FD-12-008. U.S. Department of Health and Human Services, Food and Drug Administration, Division of Federal-State Relations (DFSR). Silver Spring MD. 2012. [169A] FDA, ISO/IEC 17025:2005 CAP (Cooperative Agreement Program). U.S. Department of Health and Human Services, Food and Drug Administration, Office of Partnerships. Silver Spring MD. 2012. [170] FDA: Accreditation of Third-Party Certification Bodies To Conduct Food Safety Audits and To Issue Certifications. Federal Register. Vol. 80, No. 228, pp7450 – 74667. Washington D.C. November 2015. [171] ACIL, Statement on the Food Safety Modernization Act. American Council of Independent Laboratories. Washington D.C. 2011 [172] ACUS 2012-7, Agency Use of Third-Party Programs to Assess Regulatory Compliance. Administrative Conference of the United States. Recommendation 2012-7. Adopted December 6, 2012. 2016 NCSL International Workshop & Symposium

[173] L. McAllister, Third-Party Programs to Assess Regulatory Compliance. University of San Diego School of Law (Report prepared for ACUS). San Diego CA. October 2012. Note: Hyperlink is to September 2012 “draft” report.

[173A] FDA, Docket No. FDA-2007-N-0265 (Formerly Docket No. 2007P-0026). Final Rule – Temperature-Indicating Devices: Thermally Processed Low-Acid Foods Packaged in Hermetically Sealed Containers. Comment 9 and Comment 11. Federal Register. Vol. 76, No. 42, pp 11896. U.S. Department of Health and Human Services – Food and Drug Administration. March 2011. [173B] R. Manning, S. Lane, S. Dressman, W. Hauk, R. Williams, The Application of Uncertainty to USP’s Compendial Reference Standards Program: Certified Reference Materials. Pharmacopeial Forum. Vol. 33, No. 6, pp 2 – 14. November – December 2007. [173C] S. Dressman, W. Koch, R. Manning, S. Lane, W. Hauk, R. Williams, The Application of Uncertainty to USP’s Compendial Reference Standards Program: Certified Reference Materials. CITAC News. pp 22 – 24. February 2009. [173D] M. Weitzel, How to Monitor Reference Substances for Continuous Fitness for Purpose: Novel Approaches or Solutions to Challenges? USP Expert Committee Member Presentation. Pharmaceutical Reference Standards 11th International Symposium – Session 2b. European Directorate for the Quality of Medicines & Healthcare (EDQM), Council of Europe. Strasbourg France. September 2012. [173E] M. Weitzel, Using Design of Experiment Studies and QbD for Robustness Testing. IVT Lab Compliance Week. Philadelphia PA. December 2013. [174] NASA NPC 200-2, Quality Program Provision for Space System Contractors. National Aeronautics and Space Administration. Section 9 – Inspection Measuring and Test Equipment. Washington D.C. April 1962. [175] NASA NHB 5300.2, Apollo Metrology Requirements Manual. NASA Handbook. Apollo Documentation No. RA-006-008-1. National Aeronautics and Space Administration. Washington D.C. December 1965. [176] NASA NHB 5300.4(1B), Quality Program Provisions for Aeronautical and Space System Contractors. NASA Handbook – Reliability and Quality Assurance Publication N93-72440. (Formerly NPC 200-2). National Aeronautics and Space Administration. Washington D.C. April 1969. [177] MIL-HDBK-52B, Military Handbook: Evaluation of Contractor’s Calibration System. U.S Department of Defense. Washington D.C. August 1989. [178] MIL-PRF-38793A, Military Performance Specification: Manuals, Technical: Calibration Procedures, Preparation of. U.S. Department of Defense. Washington D.C. March 1993. [179] DOE-STD-1054-93, Guideline to Good Practices for Control and Calibration of Measuring and Test Equipment (M&TE) at DOE Nuclear Facilities. U.S. Department of Energy. Washington D.C. March 1993. [180] DOE 4330.4B, Maintenance Management Program, Section 12 – Control and Calibration of Measuring and Test Equipment. U.S. Department of Energy. Washington D.C. February 1994 [181] NASA KHB 5330.9A, Metrology & Calibration Handbook. National Aeronautics and Space Administration. Kennedy Space Center FL. October 1994

2016 NCSL International Workshop & Symposium

[181A] NCSL Z540.1 Handbook, Handbook for the Interpretation and Application of ANSI/NCSL Z540-1-1994. First edition. NCSL International. Boulder CO. October 1995. [182] DOE RW-0333P, Quality Assurance Requirements and Description. Section 12.0 – Control of Measuring and Test Equipment. Rev.5. U.S. Department of Energy – Office of Civilian Radioactive Waste Management. Washington D.C. October 1995. [183] MIL-HDBK-1839, Handbook – Calibration and Measurement Requirements. U.S Department of Defense. Washington D.C. August 1996. [184] MIL-HDBK-1839A, Handbook – Calibration and Measurement Requirements. U.S Department of Defense. Washington D.C. November 2000. [185] NASA NSTS 5300.4(1D-2), Space Shuttle: Safety, Reliability, Maintainability, and Quality Provisions for the Space Shuttle Program. Change No. 9 (June 2006). Section 1D507 – Metrology. Formerly NHB 5300.4(1D-2). National Aeronautics and Space Administration – Lyndon B. Johnson Space Center. Houston TX. September 1997 (Baseline). Note: Hyperlink is to older version (Chg No. 2 - 2000)

[186] DOE RW-0333P, Quality Assurance Requirements and Description. Section 12.0 – Control of Measuring and Test Equipment. Rev.21. U.S. Department of Energy – Office of Civilian Radioactive Waste Management. Washington D.C. January 2009. [187] MIL-STD-1839D. Standard Practice – Calibration and Measurement Requirements. U.S. Department of Defense. Washington D.C. May 2010. [188] T.O. 00-20-14, Air Force Metrology and Calibration Program. Technical Manual. U.S. Department of Defense. Secretary of the Air Force. Washington D.C. September 2011. [189] L-A-B LABAG-502, Accreditation Guidance. (2011): [190] A2LA P102, A2LA Policy on Metrological Traceability. American Association for Laboratory Accreditation. Frederick MD. May 2015. [191] A2LA R205, Specific Requirements: Calibration Laboratory Accreditation Program. American Association for Laboratory Accreditation. Frederick MD. June 2015. [192] IAS AC204 CL/009, Accreditation Criteria for Calibration Laboratories. International Accreditation Service, Inc. Brea CA. February 2016. [193] ANAB CL 2043 – Z540.3 Checklist ANSI-ASQ National Accreditation Board (previously ACLASS). Milwaukee WI. 2016 [194] ANAB MA 2002: ISO/IEC 17025 Accreditation Requirements for Calibration Laboratories. ANSI-ASQ National Accreditation Board (previously ACLASS). Milwaukee WI. January 2016. [195] IAS CL/014, Policy Guide on Calibration, Traceability, and Measurement Uncertainty for Calibration Laboratories. International Accreditation Service, Inc. Brea CA. February 2016. [196] IAEA STR-368, International Target Values 2010 for Measurement Uncertainties in Safeguarding Nuclear Materials. International Atomic Energy Agency. Vienna Austria. November 2010. [197] N. Majcen, I. Skubic, P. DeBièvre, Target Measurement Uncertainties for Results of Chemical Measurements in Slovenia – Part 1: Legislation Related to Quality of Food. Accreditation and Quality Assurance. Vol. 9, No. 1, pp 106 – 111. January 2004. 2016 NCSL International Workshop & Symposium

[198] P. DeBièvre, Target Measurement Uncertainties I: Are They Coming? Accreditation and Quality Assurance. Vol. 9, No. 10, pp 589 – 590. September 2004. [199] P. DeBièvre, Target Measurement Uncertainties II: Are They Coming? Accreditation and Quality Assurance. Vol. 9, No. 11, pp 651 – 652. October 2004. [200] R. DaSilva, A. Williams, Setting the Target (Maximum Admissible) Measurement Uncertainty for Different Analytical Goals. Poster Presentation – Pittsburgh Conference on Analytical Chemistry and Applied Spectroscopy (Pittcon). New Orleans LA. October 2015. [201] A2LA P903, Policy on Estimating Measurement Uncertainty for ISO 15189 Testing Laboratories. March 2014. [202] Eurachem Guide – V. Barwick, E. Prichard, Terminology in Analytical Measurement – An Introduction to VIM 3. First Edition. ISBN 978-0-948926-29-7 [203] RDT F3-2T, RDT Standard - Calibration System Requirements. United States Atomic Energy Commission (AEC) – Division of Reactor Development and Technology (RDT). Germantown MD. February 1969.

2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

See also preceding bibliographical references [2 – 7 , 42 – 47, 49, 51 – 66, and 173B-E] 1950’s [A1] A. Eagle, A Method for Handling Errors in Testing and Measuring. Industrial Quality Control, Vol. 10, No. 5, pp 10 – 15. March 1954. [A2] F. Grubbs, H. Coon, On Setting Test Limits Relative to Specification Limits. Industrial Quality Control, Vol. 10, No. 5, pp 15 – 20. March 1954. [A3] J. Wiesen, C. Clark, Determining Allowable Test Set Errors. Sandia Corporation, Technical Memorandum 201-54-51. Albuquerque NM. September 1954. [A4] L. Mandel, Grading With a Gauge Subject to Random Output Fluctuations. Journal of the Royal Statistical Society, Series B (Methodological). Vol. 16, No. 1, pp 118-130. 1954. [A5] J. Hayes, Factors Affecting Measurement Reliability. Technical Memorandum No. 63-106. U.S. Naval Ordnance Laboratory – Measurements Reliability Branch – Production Quality Division – Missile Evaluation Department. Corona CA. October 1955. [A6] F. Tingey, J. Merrill, Minimum Risk Specification Limits. Atomic Energy Commission (AEC) Research and Development Report IDO-16396. Phillips Petroleum Company – Atomic Energy Division. Idaho Falls ID. July 1957. [A7] H. David, E. Fay, J. Walsh, Acceptance Inspection by Variables When the Measurements are Subject to Error. Annals of the Institute of Statistical Mathematics. Vol. 10, No. 2, pp 107129. June 1959. [A8] J. Wiesen, D. Owen, A Method of Computing Bivariate Normal Probabilities with an Application to Handling Errors in Testing and Measuring, Bell System Technical Journal, Volume 38, No. 2, pp 553-572. March 1959. [A9] F. Tingey, J. Merrill, Minimum Risk Specification Limits. Journal of the American Statistical Association, Vol. 54, No. 285, pp 260 – 275. March 1959. [A10] J. Gant, Let’s Take the Guesswork out of Inspection! American Machinist. Vol 103, pp 117-122. March 9, 1959. 1960’s [A11] R. Traver, Measuring Equipment Repeatability – The Rubber Ruler. Transactions of the Annual Convention of the American Society for Quality Control (ASQC). pp 25 – 32. Milwaukee WI. May 1962. (reprinted in Quality Engineering Vol. 5, No. 1, 1992, pp 181-190). [A11A] T. Diviney, N. David, A Note on the Relationship Between Measurement Error and Product Acceptance. The Journal of Industrial Engineering. Vol 14, No. 4, pp 218-219. August 1963. [A11B] H. Singh, Producer and Consumer Risks for Asymmetrical Test and Specification Limits. Journal of the American Statistical Association. Vol. 61, No. 314. 1966. [A12] H. Singh, Producer and Consumer Risks in Non-Normal Populations. Technometrics. Vol. 8, No. 2, pp 335-343. May 1966. [A13] A. Marshall, I. Olkin. A General Approach to Some Screening and Classification Problems. Journal of the Royal Statistical Society – Series B, Methodological. Vol. 30, No. 3, pp 407 – 443. 1968. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

1970’s [A14] T. Mukaihata, Administration of a Standards Laboratory, Section 3 – Special Problems of Technical and Administrative Nature. Proceedings of the Measurement Science Conference. San Luis Obispo CA. December 1976. [A15] ASTM STP-616, B. Marguglio, Quality Systems in the Nuclear Industry, Chapter 14 – Measurement Control, Section – Multiple (4 times to 10 times rule). Special Technical Publication No. 616, pg 344. American Society for Testing and Materials. ISBN-13: 978-0-80310197-5. Philadelphia PA. 1977 [A16] H. Castrup, Evaluation of Customer and Manufacturer Risk vs. Acceptance Test Instrument In-Tolerance Level. TRW Technical Report No. 99900-7871-RU-00. Redondo Beach CA. April 1978. [A17] NCSL, Uncertainty, Accuracy, and Accuracy Ratios. NCSL Newsletter, National Conference of Standards Laboratories. Vol. 18, No. 3, pp 22 – 24. Boulder CO. December 1978. [A17A] K. Kuskey, Cost-Benefit Model for Analysis of Calibration System Designs in the Case of Random-Walk Equipment Behavior. SAI Comsystems Technical Report. Prepared for U.S. Navy Metrology Engineering Center. Contract N000123-76-C-0589. January 1979. [A18] M. Lay, There are No Magic Accuracy Ratios in Metrology and Calibration. NCSL Newsletter. Vol. 19, No. 1. pp 26 – 27. National Conference of Standards Laboratories. Boulder CO. March 1979. [A19] J. Ferling, A Calibration Interval Analysis Model – Expected Costs Due to Wrong Test Decisions as a Function of the Calibration Interval. SAI Comsystems Corp. Prepared for U.S. Navy Metrology Engineering Center (Contract No. N00123-76-C-0589). Corona CA. April 1979 [A20] C. Koop, Accuracy Ratio Requirements. NCSL Newsletter. Vol. 19, No. 1. pp 28 – 29. National Conference of Standards Laboratories. Boulder CO. March 1979. [A21] J. Ferling, Accuracy Ratios Related to Consumer’s and Producer’s Risks. NCSL Newsletter. Vol. 19, No. 1. pp 37 – 39. National Conference of Standards Laboratories. Boulder CO. March 1979. [A22] R. Schumacher, Another Viewpoint on Accuracy Ratios. NCSL Newsletter. Vol. 19, No. 2 (mislabeled No. 1). pp 32 – 34. National Conference of Standards Laboratories. Boulder CO. June 1979. 1980’s [A23] H. Castrup, Evaluation of Customer and Manufacturer Risk vs. Acceptance Test Instrument In-Tolerance Level. Proceedings of the National Conference of Standards Laboratories. Gaithersburg MD. September 1980. [A24] G. Hahn, Removing Measurement Error in Assessing Conformance to Specifications. Journal of Quality Technology. Vol. 14, No. 3, pp117 – 121. July 1982. [A24A] M. Lotti, Measurement Variability and Product Acceptance. Quality Progress. Vol. 15, No. 9, pp 38 – 41. September 1982. [A25] R. Mee, D. Owen, A Simple Approximation for Bivariate Normal Probabilities. Technical Report No. 169. Department of Statistics, Office of Naval Research, Contract No. N00014-76-C0613. Dallas TX. December 1982. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A26] K. Kuskey, New Capabilities for Analyzing METCAL Decisions. Proceedings of the Measurement Science Conference. Long Beach CA. January 1984. [A27] J. Ferling, The Role of Accuracy Ratios in Test and Measurement Processes. Proceedings of the Measurement Science Conference. Long Beach CA. January 1984. [A28] R. Mee, D. Owen, J. Shyu, Confidence Bounds for Misclassification Probabilities Based on Data Subject to Measurement Error. Journal of Quality Technology. Vol. 18, No. 1, pp 29 – 40. January 1986. [A29] D. Jackson, Instrument Intercomparisons and Calibration. Proceedings of the Measurement Science Conference. Anaheim CA. 1987. [A30] J. Hayes, Decisions That Influence Quality. Proceedings of the Measurement Science Conference. Anaheim CA. 1988. [A31] R. Schumacher, MIL-STD-45662: Conflicting Positions on Out-Of-Tolerance Measurements. Proceedings of the Measurement Science Conference. Anaheim CA. 1988. [A32] F. Capell, From 4:1 to SPC, Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. Washington DC. August 1988. [A33] F. Capell, More on ‘From 4:1 to SPC’, Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. pp 8-1 to 8-15. Denver CO. July 1989. [A34] B. Gunter, The Use and Abuse of Cpk: Part 4. Quality Progress. Vol. 22, No. 7, pp 86 – 87. July 1989. [A35] H. Castrup, Calibration Requirements Analysis System. Proceedings of the NCSL Workshop and Symposium. National Conference of Standards Laboratories. Denver CO. July 1989. [A36] J. Ferling, D. Caldwell, Analytic Modeling for Electronic Test Equipment Adjustment Policies, Proceedings of the NCSL Workshop and Symposium. National Conference of Standards Laboratories. Denver CO. July 1989. [A36A] J. Ferling, On the Determination of Optimum Tolerance Test Limits Under Measurement Uncertainty Conditions. Proceedings of the 35th International Instrumentation Symposium. Orlando FL. 1989. 1990’s [A37] T. Mukaihata, The Accuracy Ratio Dilemma. Presentation at NCSL Region 8 San Diego Section Meeting. National Conference of Standards Laboratories. Montgomery Field CA. May 1990. [A38] S. Navard, Evaluating the Effect of Accuracy Ratios on the Percent of Calibrations Which Are Out of Tolerance. National Aeronautics and Space Administration (NASA) – Johnson Space Center. Final Report N91-27107. Contract No. NGT 44-055-803. Houston TX. August 1990. [A38A] P. Mullenix, The Capability of Capability Indices With an Application to Guardbanding in a Test Environment. IEEE Proceedings of the International Test Conference. pp 907 – 915. ISBN 0-8186-9064-X. Washington D.C. September 1990. [A38B] R. Easterling, M. Johnson, T. Bement, C. Nachtsheim, Statistical Tolerancing Based on Consumer’s Risk Considerations. 34th Annual Fall Technical Conference the American Society for Quality (Chemical and Process Industries Division and Statistics Division). October 1990. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A39] F. Capell, From 4:1 to SPC, International Society of Automation. ISA Transactions. Vol. 29, No. 4. November 1990. [A40] F. Capell, How Good Is Your TUR? Evaluation Engineering. pp 81-84. January 1991. [A41] H. Castrup, Analytical Metrology SPC Methods for ATE Implementation. Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. Albuquerque NM. August 1991. [A42] B. Hutchinson, Setting Guardband Test Limits to Satisfy MIL-STD-45662A Requirements. Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. Albuquerque NM. August 1991. [A42A] J. Hayes, Calibration and Maintenance of Test and Measuring Equipment. Encyclopedia of Applied Physics. Vol. 3, No. 11. VCH Publishers, Inc. ISBN 1-56081-062-9. New York NY. 1992. [A42B] R. Schumacher, Decision Making and Customizing Decision Limits in Measurement Assurance. Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. pp 301-310. 1992. [A42C] D. Wheeler, Problems with Gauge R&R Studies. Transactions of the ASQ Annual Quality Congress. Vol. 46, No.0, pp 179-185. Nashville TN. May 1992. (reprinted in Quality Digest, January 2011). [A43] D. Deaver, How to Maintain Your Confidence in a World of Declining Test Uncertainty Ratios. Proceedings of the NCSL Workshop and Symposium, National Conference of Standards Laboratories. Albuquerque NM, July 1993. [A43A] R. Williams, C. Hawkins, The Economics of Guardband Placement. Proceedings of the IEEE International Test Conference (ITC). ISBN 0-7803-1430-1. Baltimore MD. October 1993. [A43B] W. Albers, W. Kallenberg, G. Otten. Accurate Test Limits With Estimated Parameters. Technometrics. Vol. 36, No. 1, pp 92-101. February 1994. [A44] D. Deaver, Guardbanding With Confidence. Proceedings of the NCSL Workshop and Symposium, National Conference of Standards Laboratories. Chicago IL, August 1994. [A44A] W. Albers, W. Kallenberg, G. Otten, Setting Test Limits Under Prescribed Consumer Loss. Metrika. Vol. 41, No. 1, pp 163 – 181. December 1994. [A45] H. Castrup, Uncertainty Analysis for Risk Management. Proceedings of the Measurement Science Conference. Anaheim CA. January 1995. [A46] H. Castrup, Analyzing Uncertainty for Risk Management. Proceedings of the ASQC Annual Quality Congress. Vol. 49, No. 0, Cincinnati OH. May 1995. [A47] D. Deaver, Using Guardbands to Justify TURs Less Than 4:1. Proceedings of the ASQC Annual Quality Congress, Vol. 49, No. 0, pp 136-141. Cincinnati, OH. May 1995. [A48] D. Deaver, Managing Calibration Confidence in the Real World. Proceedings of the NCSL Workshop and Symposium, National Conference of Standards Laboratories. Dallas TX. July 1995. [A48A] H. Castrup, Uncertainty Analysis and Parameter Tolerancing. Proceedings of the NCSL Workshop and Symposium, National Conference of Standards Laboratories. Dallas TX. July 1995. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A48B] D. Fitzsimmons, Calibration Family Risk Approach for 4:1 Equivalence – The Meaning of Calibration Uncertainty and the Role of Uncertainty in Standards. Proceedings of the NCSL Workshop and Symposium, National Conference of Standards Laboratories. Dallas TX. July 1995. [A48C] DIN EN 50222:1996-03 (VDE 0847-222), Standard for the Evaluation of Measurement Results Taking Measurement Uncertainty Into Account. (withdrawn). Germany. March 1996. [A49] D. McCarville, D. Montgomery. Optimal Guard Bands for Gauges in Series. Quality Engineering, Vol 9. No. 2, pp 167-177. 1996-1997. [A50] W. Albers, G. Arts, W. Kallenberg, Accurate Test Limits Under Prescribed Consumer Risk. Statistics & Probability Letters. Vol. 34, No. 2, pp 141-149. June 1997. [A51] M. Fecteau, Test Uncertainty Ratio and Intelligent Instrumentation: From 4:1 to SPC to IPC in the Army Measurement System. Proceedings of the National Conference of Standards Laboratories Workshop and Symposium, Atlanta GA. July 1997. [A52] J. Song, The Guidelines for Expressing Measurement Uncertainties and the 4:1 Test Uncertainty Ratio (TUR). Proceedings of the National Conference of Standards Laboratories Workshop and Symposium. Atlanta GA. July 1997. [A53] D. Deaver, Guardbanding and the World of ISO Guide 25 – Is There Only One Way? Proceedings of the NCSL Workshop and Symposium, National Conference of Standards Laboratories. Albuquerque NM, July 1998. [A53A] S. Phillips, W. Estler, M. Levenson, K. Eberhardt, Calculation of Measurement Uncertainty Using Prior Information. [A54] R. Nicholas. Measurement Decision Risk Simplified. Proceedings of the Measurement Science Conference. Anaheim CA. January 1999. [A55] W. Wong. What TUR do You Really Need? Putting Statistical Theory Into Practice. Proceedings of the Measurement Science Conference. Anaheim CA. January 1999. [A56] R. Nicholas. Risk-Based Guardbanding for Unilateral Uncertainties. Proceedings of the NCSL Workshop and Symposium. National Conference of Standards Laboratories. Charlotte NC. July 1999. [A57] W. Schultz, K. Sommer, Uncertainty of Measurement and Error Limits in Legal Metrology. International Organization for Legal Metrology. OIML Bulletin Vol. 40, No. 4, pp 515. October 1999. [A57A] T. Estler, Measurement as Inference: Fundamental Ideas, Section 7.4 – Industrial Inspections III: Accept/Reject Decisions. CIRP Annals – Manufacturing Technology. Vol. 48, No. 2, pp 611 – 631. December 1999. [A58] D. Mader, J. Prins, R. Lampe, The Economic Impact of Measurement Error. Quality Engineering. Vol. 11, No. 4. Pp 563 – 574. 1999. [A59] I. Lira, A Bayesian Approach to the Consumer’s and Producer’s Risk in Measurement. Metrologia. Vol. 36, No. 5, pp 397 – 402. 1999. 2000’s [A60] K. Sommer, M. Kochsiek, W. Schulz, Error Limits and Measurement Uncertainty in Legal Metrology. XVI IMEKO World Congress. Vienna Austria. September 2000. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A61] IEC CISPR/A/291/CDV, Accounting for Measurement Uncertainties When Determining Compliance With a Limit. International Electrotechnical Commission / International Special Committee on Radio Interference. Geneva Switzerland. December 2000. [A62] N. Doganaksoy, Assessment of Impact of Measurement Variability in the Presence of Multiple Sources of Product Variability. Quality Engineering. Vol. 13, No. 2, pp 83 – 89. 2000 [A62A] H. Castrup, Risk Based Control Limits. Proceedings of the Measurement Science Conference. Anaheim CA. January 2001. [A62B] R. Boyles, Gauge Capability for Pass-Fail Inspection. Technometrics. Vol. 43, No. 2, pp 223 – 229. May 2001. [A63] T. Skwirczyṅski, Uncertainty of the Calibrating Instrument, Confidence in the Measurement Process and the Relation Between Them. OIML Bulletin, Vol. XLII, No. 3, pp 510. July 2001. [A63A] G. Arts, W. Albers, W. Kallenberg, Simultaneous Inspection of Several Product Characteristics. Metrika. Vol. 54, No. 1, pp 19 – 41. August 2001. [A64] P. Carbone, D. Macii, D. Petri, Measurement Uncertainty and Metrological Confirmation in Documented Quality Systems. 11th IMEKO TC-4 Symposium – Trends in Electrical Measurement and Instrumentation. Lisbon Portugal. September 2001. [A65] P. Stein, Meeting Specifications – Compensate for Measurement Error in Test Results. American Society for Quality (ASQ) – Quality Progress. Milwaukee WI. March 2002. [A66] R. Burdick, Recent Extensions in Gauge Capability Studies. Quality and Productivity Research Conference. American Statistical Association. University of Arizona. Tempe AZ. June 2002. [A67] P. Stein, Statistical Issues in Measurement, Item #S1010. Special Publication – ASQ Statistics Division. American Society for Quality. Milwaukee WI. Fall 2002. [A68] D. Jackson, Measurement Uncertainty and Risk for Navy Calibrations. Proceedings of the NCSL International Workshop and Symposium. San Diego CA. August 2002. [A68A] I. Lira, Evaluating The Measurement Uncertainty – Fundamentals and Practical Guidance. Section 4.3 – The Consumer’s and Producer’s Risks, Section 4.4 – The Acceptance Interval: Guardbanding. pp 109 – 111. Section 6.7 – Evaluation of the Producer’s and Consumer’s Risks. pp 192 – 200. Institute of Physics – Series in Measurement Science and Technology. ISBN 0 7503 0840 0. IOP Publishing Ltd. Bristol UK and Philadelphia PA. 2002. [A69] P. Carbone, D. Macii, D. Petri, Measurement Uncertainty and Metrological Confirmation in Quality-Oriented Organizations. Measurement. Vol. 34, pp 263-272. 2003 [A70] D. Macii, P. Carbone, D. Petri, Management of Measurement Uncertainty for Effective Statistical Control. IEEE Transactions on Instrumentation and Measurement. Vol. 52, No. 5. October 2003. [A70A] R. Burdick, C. Borror, D. Montgomery, A Review of Methods for Measurement System Capability Analysis. 47th Annual Fall Technical Conference of the Chemical and Process Industries Division and Statistics Division of the American Society for Quality and the Section on Physical Engineering Sciences of the American Statistical Association. El Paso TX. October 2003. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A71] H. Kӓllgren, M. Lauwaars, B. Magnusson, L. Pendrill, P. Taylor, Role of Measurement Uncertainty in Conformity Assessment in Legal Metrology and Trade. Accreditation and Quality Assurance. Vol. 8, No. 11, pp 541-547. November 2003. [A72] J. Sheppard, Accounting for False Indication in a Bayesian Diagnostics Framework. Proceedings of Autotestcon 2003 – IEEE Systems Readiness Technology Conference. Future Sustainment for Military and Aerospace. Cat No. 03CH37447. pp 273 – 279. ISBN 0-78037837-7. September 2003. [A73] P. Carbone, D. Macii, D. Petri, Management of Measurement Uncertainty for Effective Statistical Control. Technical Report No. DIT-04-053. University of Trento. Department of Information and Communication Technology. Trento Italy. May 2004. [A74] D. Shah, The Uncertain State of Uncertainty in Industry. Proceedings of the ASPE, Uncertainty Analysis in Measurement and Design. American Society for Precision Engineering. University Park PA. June 2004. [A75] R. Nicholas, L. Anderson. Guardbanding Using Automated Software. Proceedings of the NCSL International Workshop and Symposium. Salt Lake City UT. July 2004. [A76] L. Rodriguez, A. Arroyo, O. Ramirez, Implementation of a Methodology for Quantifying the Economic Effect of Measurement Errors. Proceedings of the NCSL International Workshop and Symposium. Salt Lake City UT. July 2004. [A77] K. Sommer, M. Kochsiek, Modeling and Uncertainty Evaluation in Calibration and Conformance Testing. Proceedings of the NCSL International Workshop and Symposium. Salt Lake City UT. July 2004. [A78] S. Phillips, Measurement Uncertainty and Traceability Issues: A Standards Activity Update. Proceedings of ASPE Spring Topical Meeting on Uncertainty Analysis in Measurement and Design. July 2004. [A79] AFNOR FD x07-022, Metrology and Applications of Statistics - Use of Measurement Uncertainties: Presentation of Some Cases and Customary Practices. Section 6 – Using the Uncertainty in Conformity Assessment: Evaluation of the Associated Risk. pp 16 – 21. AFNOR Groupe. Association Française de Normalisation (French Association for Standardization). December 2004. [A80] D. Jackson, Measurement Decision Risk and Methods. Proceedings of the Measurement Science Conference. Anaheim CA. January 2005. [A81] K. Bennett, H. Zion, Metrology Concepts: Understanding Test Uncertainty Ratio (TUR). Transcat White Paper, P/N 7-05-879. Rochester NY. May 2005. [A82] R. Burdick, Y. Park, D. Montgomery, C. Borror, Confidence Intervals for Misclassification Rates in a Gauge R&R Study. Journal of Quality Technology. Vol. 37, No. 4, pp 294 – 303 . October 2005. [A83] D. Jackson, Measurement Decision Risk Analysis Methods with Multiple Test Points. Proceedings of the Measurement Science Conference. Anaheim CA. January 2006. [A84] J. Sheppard, M. Kaufman, A Bayesian Approach to Diagnosis and Prognosis Using BuiltIn Test. IEEE Transactions on Instrumentation and Measurement. Vol. 54, No. 3, pp 1003 – 1018. June 2005. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A85] R. Burdick, C. Borror, D. Montgomery, Design and Analysis of Gauge R&R Studies: Making Decisions with Confidence Intervals in Random Mixed ANOVA Models. Section 1.6 – Misclassification Rates. ASA-SIAM Series on Statistics and Applied Mathematics. Society for Industrial and Applied Mathematics. ISBN 978-0-89871-588-0. 2005. [A85A] G. Rossi, F. Crenna, A Probabilistic Approach to Measurement-Based Decisions. Measurement. Vol. 39, No. 2, pp 101-119. February 2006. [A86] WELMEC, Elements for Deciding the Appropriate Level of Confidence in Regulated Measurements. (Accuracy Classes, MPE In-service, Non-conformity, principles of Uncertainty). Section 6 – Measurement Uncertainty and Decision-Making. WELMEC 4.2, Issue 1. European Co-operation in Legal Metrology. Overveen Netherlands. June 2006. [A87] D. Jackson, Measurement Risk Analysis Methods as Applied to Guardbands. Proceedings of the NCSLI International Workshop and Symposium. Nashville TN. August 2006. [A88] R. Nicholas. The Significance on Consumer and Producer Risk for Uncorrected Product and Measurement Biases. NCSL International Workshop and Symposium. Nashville TN. August 2006. [A89] L. Pendrill, Optimised Decision-Making in Conformity Assessment, Proceedings of the NCSLI International Workshop and Symposium. Nashville TN. August 2006. [A89A] D. Wheeler, EMP III, Evaluating the Measurement Process & Using Imperfect Data, Chapter 14 – The Basis for Manufacturing Specifications, Section 14.3 The Bivariate Normal Model, Section 14.4 The Probability of Conforming Product, pp 181 – 193. SPC Press. Knoxville TN. October 2006. [A89B] D. Wheeler, An Honest Gauge R&R Study. ASQ/ASA Fall Technical Conference. Manuscript No. 189 (January 2009 Revision). American Society for Quality / American Statistical Association. Columbus OH. October 2006. [A90] L. Pendrill, Optimised Measurement Uncertainty and Decision-Making when Sampling by Variables or by Attribute. Measurement. Vol. 39, No. 9, pp 829-840. November 2006. [A91] A. Forbes, Measurement Uncertainty and Optimized Conformance Assessment. Measurement. Vol. 39, No. 9, pp 808 – 814. November 2006. [A91A] H. Castrup, S. Castrup, Analytical Metrology Handbook, Part 3 – Analyzing Measurement Decision Risk. First edition. Integrated Sciences Group. Bakersfield CA. 2008. [A92] H. Castrup, Bayesian Risk Analysis. Technical Document. Integrated Sciences Group. Bakersfield CA. April 2007. [A93] Y. Kim, B. Cho, N. Kim, Economic Design of Inspection Procedures Using Guard Band When Measurement Errors are Present. Applied Mathematical Modeling, Vol. 31, No. 5. pp 805-816. May 2007. [A94] L. Pendrill, Optimised Measurement Uncertainty and Decision-Making in Conformity Assessment, NCSLI Measure – The Journal of Measurement Science. Vol. 2, No. 2. June 2007. [A95] M. Dobbert, Understanding Measurement Risk. Proceedings of the NCSL International Workshop and Symposium. St. Paul MN. July 2007. [A96] H. Castrup, Risk Analysis Methods for Complying with Z540.3. Proceedings of the NCSL International Workshop and Symposium. St. Paul MN. July 2007. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A96A] TAF-CNLA-G-04, Method of Stating Test and Calibration Results and Compliance With Specification. Taiwan Accreditation Foundation. Taiwan ROC. July 2007. [A97] OMCL PA/PH/OMCL (05) 49 DEF CORR, Uncertainty of Measurement – Part 1. General OMCL Policy for Implementation of Measurement Uncertainty in Compliance Testing. European Directorate for the Quality of Medicines & Healthcare. December 2007. [A97A] Z. Kosztyán, T. Csizmadia, C. Hegedűs, Z. Kovacs, Treating Measurement Uncertainty in Complete Conformity Control System. Conference: Innovations and Advances in Computer Sciences and Engineering. Vol.1 of Proceedings of the 2008 International Conference on Systems, Computing Sciences and Software Engineering. Bridgeport CT. December 2007. [A98] F. Wang, S. Yang, Applying Bootstrap Method to the Types I-II Errors in the Measurement System. Quality and Reliability Engineering International. Vol. 24, No. 1, pp 83 – 97. February 2008. [A99] D. Caldwell, D. Jackson. Measurement Risk Analysis Definitions as Applied to Z540.3. Proceedings of the Measurement Science Conference. Anaheim CA. March 2008. [A100] R. Nicholas. How to Build Your Own Unilateral Consumer Risk Calculation Tool, Including the Treatment of Biases. Proceedings of the Measurement Science Conference. Anaheim CA. March 2008. [A101] H. Castrup, Applying Measurement Science to Ensure End Item Performance. Proceedings of the Measurement Science Conference. Anaheim CA. March 2008. [A102] R. Walker. Evaluation of Out-of-Tolerance Risk in Measuring and Test Equipment. Proceedings of the NCSL International Workshop and Symposium. Orlando FL. August 2008 [A103] M. Dobbert, A Guard-Band Strategy for Managing False-Accept Risk. Proceedings of the NCSL International Workshop and Symposium. Orlando FL. August 2008. [A103A] M. Cox, G. Rossi, P. Harris, A. Forbes, A Probabilistic Approach to the Analysis of Measurement Processes. Metrologia. Vol. 45, No. 5. August 2008. [A104] Transcat, Risk Management: The Critical Role of Measurement and Calibration. Transcat White Paper, P/N TC918 10-08. Rochester NY. October 2008. [A105] K. Weißensee, O. Kühn, G. Linß, K. Sommer. Risk of Erroneously Deciding Conformity of Measuring Instruments. Accreditation and Quality Assurance. Vol. 13, No. 11, pp 663 – 669. November 2008. [A106] L. Pendrill, Review of Symposium ‘New Developments in Measurement Uncertainty in Chemical Analysis’. Accreditation and Quality Assurance. Vol. 13, No. 11, pp 671 – 674, November 2008. [A107] A. Williams. Compliance With Specifications. Accreditation and Quality Assurance. Vol. 13, pp 617 – 618, November 2008. [A108] A. Williams, Principles of the EURACHEM/CITAC Guide ‘Use of Uncertainty Information in Compliance Assessment’. Accreditation and Quality Assurance. Vol. 13, No. 11, pp 633 – 638. November 2008. [A109] M. Czaske, Usage of the Uncertainty of Measurement by Accredited Calibration Laboratories When Stating Compliance. Accreditation and Quality Assurance. Vol. 13, No. 11, pp 645-651. November 2008. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A110] E. Desimoni, B. Brunetti, A. Clerici, About Considering Both False Negative and FalsePositive Errors When Assessing Compliance and Non-compliance with Reference Values given in Compositional Specifications and Statutory Limits. Accreditation and Quality Assurance. Vol. 13, No. 11, pp 653 – 662. November 2008. [A111] M. Dobbert, A Guard-Band Strategy for Managing False-Accept Risk. NCSLI Measure – The Journal of Measurement Science. Vol. 3, No. 4, pp 44-48. December 2008. [A112] D. Macii, D. Petri, Guidelines to Manage Uncertainty in Conformance Testing Procedures. IEEE Transactions on Instrumentation and Measurement. Vol. 58, No. 1. January 2009. [A112A] I. Lira, Comment on ‘A Probabilistic Approach to the Analysis of Measurement Processes’. Metrologia. Vol. 46, No. 1. January 2009. [A113] M. Dobbert, R. Stern, A Pragmatic Method for Pass/Fail Conformance Reporting that Complies with ANSI Z540.3, ISO 17025, and ILAC-G8. Proceedings of the NCSL International Workshop and Symposium. San Antonio TX. July 2009. [A114] H. Castrup, An Examination of Measurement Decision Risk and Other Measurement Quality Metrics. Proceedings of the NCSL International Workshop and Symposium. San Antonio TX. July 2009. [A115] D. Huang, S. Dwyer, Test Instrument Reliability Perspectives and Practices: Part 1: Interpreted Within System Reliability Framework. Proceedings of the NCSL International Workshop and Symposium. San Antonio TX. July 2009. [A116] S. Khanam, Test Uncertainty Ratio (TUR) and Test Uncertainty. Ph.D Dissertation in Mechanical Engineering. University of North Carolina at Charlotte. Charlotte NC. 2009 [A116A] APLAC TC-004, Method of Stating Test and Calibration Results and Compliance With Specifications. Asia Pacific Laboratory Accreditation Cooperation. October 2009. [A117] E. Desimoni, B. Brunetti, Uncertainty of Measurement and Conformity Assessment: A Review. Analytical and Bioanalytical Chemistry. Vol. 400, No. 6, pp 1729 – 1741. June 2011. [A118] J. Hurll, Assessment of Compliance With Specifications. Cal Lab Magazine. Vol. 15, No. 4, pp 34-37. Oct-Nov-Dec 2009. [A119] R. Fritzsche, Metrology and Calibration (METCAL) Overview. Naval Surface Warfare Center, Corona Division – Measurement Science Department. Corona CA. Undated. 2010’s [A120] D. Deaver, J. Somppi, A Study of and Recommendations for Applying the False Acceptance Risk Specification of Z540.3. Proceedings of the Measurement Science Conference. Anaheim CA. March 2010. [A121] M. Dobbert, R. Stern, A Pragmatic Method for Pass/Fail Conformance Reporting that Complies with ANSI Z540.3, ISO 17025, and ILAC-G8. NCSLI Measure – The Journal of Measurement Science. Vol. 5, No. 1. March 2010. [A122] D. Jackson, Threshold Analysis. Proceedings of the Measurement Science Conference. Anaheim CA. March 2010. [A123] W. Hinrichs, The Impact of Measurement Uncertainty on the Producer’s and User’s Risks, on Classification and Conformity Assessment: An Example Based on Tests on Some 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

Construction Products. Accreditation and Quality Assurance. Vol. 15, No. 5, pp 289-296. May 2010. [A124] M. Yiu. Calibration is Changing. EDN Network. June 2010. [A125] S. Mimbs, Conformance Testing: Measurement Decision Rules. Proceedings of the NCSL International Workshop and Symposium. Providence RI. July 2010. [A126] L. Lewis, Managing Risk Using the Guard Band Method 2 % Bayesian Analysis. Proceedings of the NCSL International Workshop and Symposium. Providence RI. July 2010. [A127] S. Beckert, E. Paladini, L. Valentina, Analysis of the Acceptable Deviation for the Measurement Process Established by the MSA and the VDA Manuals. Proceedings of the World Congress on Engineering. Vol III. London England. July 2010. [A128] J. Salsbury, Uncertainty Management Versus Risk Management in Calibration. NCSLI Measure – The Journal of Measurement Science. Vol. 5, No. 3. September 2010. [A129] J. DeMast, T. Erdmann, W. Van Wieringen, Measurement System Analysis for Binary Inspection. Continuous Versus Dichotomous Measurands. Journal of Quality Technology. American Society for Quality. Vol. 43, No. 2. April 2011. [A130] Y. Zhu, M. Li, C. Young, M. Xie, E. Elsayed, Impact of Measurement Error on Container Inspection Policies at Port-of-Entry. Annals of Operations Research. Vol. 187, No. 1, pp 22-43. July 2011. [A131] D. Jackson Enhanced Equipment Calibration Tradeoff Analysis. Proceedings of the NCSL International Workshop and Symposium. National Harbor MD. August 2011. [A132] H. Alexander, How to Satisfy the Requirements of ISO/IEC 17025:2005 When Reporting Calibration or Test Results and Making a Statement of Compliance with a Metrological Specification from the Perspective of the Accrediting Body. Proceedings of the NCSL International Workshop and Symposium. National Harbor MD. August 2011. [A133] ASTM E2782, Standard Guide for Measurement System Analysis (MSA), Section 9 – Measurement System Performance.; Section 10.7 – Probability of Acceptance Given Measurement Value; Section 10.8 – Probability of Gage Acceptance Given True Measurement. ASTM International. West Conshohocken. 2011 (editorial corrections, May 2014). [A134] S. Rishi, Guard-Banding Methods – An Overview. Proceedings of AdMET, Paper No. UM001. London England. 2012. [A135] C. Grachanen, Are TUR’s a Thing of The Past? The Standard. American Society for Quality (ASQ). Vol. 26, No. 1, pp 4 – 5. March 2012. [A135A] J. Salsbury, E. Morse, Measurement Uncertainty in the Performance Verification of Indicating Measuring Instruments. Precision Engineering. Vol. 36, No. 2, pp 218 – 228. April 2012. [A135B] D. Montgomery, Introduction to Statistical Quality Control, Chapter 8 – Process and Measurement System Capability Analysis. Section 8.7.4 – False Defectives and Passed Defectives. 7th edition. John Wiley & Sons, Inc. June 2012. [A136] M. Dobbert, Understanding Measurement Risk. Agilent Technologies (now Keysight Technologies). White Paper 5991-1265EN. Santa Rosa CA. October 2012.

2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A137] M. Dobbert, A Guard-Band Strategy for Managing False-Accept Risk. Agilent Technologies (now Keysight Technologies). White Paper 5991-1267EN. Santa Rosa CA. October 2012. [A138] M. Weitzel, W. Johnson, Using Target Measurement Uncertainty to Determine Fitness for Purpose. Accreditation and Quality Assurance. Vol. 15, No. 5, pp 491 – 495. October 2012. [A139] T. Wendle, Too Much Calibration? Agilent Technologies (now Keysight Technologies). White Paper 5991-1311EN. Santa Rosa CA. November 2012. [A140] SureCal TN-201203, Interpreting Acceptance Criteria as Reported in SureCal. Northrup Grumman Systems Corporation. December 2012. [A141] ISO/TR 14253-6:2012, Geometrical Product Specifications (GPS) — Inspection by Measurement of Workpieces and Measuring Equipment — Part 6: Generalized Decision Rules for the Acceptance and Rejection of Instruments and Workpieces. International Organization for Standardization. Geneva Switzerland. 2012 [A142] B. Stern, How to Comply with ANSI Z540.3. Webcast. Agilent Technologies (now Keysight Technologies). Santa Rosa CA. February 2013. [A143] W. Estler, D. Hibbert, JCGM 106:2012 – A New Guidance Document on Measurement Uncertainty and Conformity Assessment. International Organization of Legal Metrology. OIML Bulletin Vol. LIV, No. 2. pp 14 – 16. April 2013. [A143A] S. Stamm, A Comparison of Gauge Repeatability and Reproducibility Methods. PhD Dissertation. Indiana State University. Terre Haute IN. May 2013. [A144] M. Bar, C. Elster, S. Heidenreich, C. Matthews, L. Pendrill, L. Wright, Novel Mathematical and Statistical Approaches to Uncertainty Evaluation: Introducing a New EMRP Research Project, Section 4 – Conformity Assessment and Decision Making. Conference Paper, 16th International Congress of Metrology. Paris France. October 2013. [A145] ASTM E29-13, Standard Practice for Using Significant Digits in Test Data to Determine Conformance With Specifications. ASTM International. Philadelphia PA. 2013 [A146] OIML TC3/SC5/p2/N002/2CD, The Role of Measurement Uncertainty in Conformity Assessment Decisions in Legal Metrology. International Organization of Legal Metrology (OIML) Committee Draft 2. February 2014. [A147] H. Mezouara, L. Dlimi, A. Salih, M. Afechcar. Determination of the Optimal Guardbanding to Ensure Acceptable Risk Decision in the Declaration of Conformity. Journal of Engineering Research and Applications. Vol. 4, No. 6, pp 16-20. June 2014. [A148] C. Grachanen, Understanding Test Accuracy and Test Uncertainty Ratios. Quality Progress. American Society for Quality (ASQ). Vol. 47, No. 7, pp 44 – 45. July 2014. [A149] R. Long, Taking My Uncertainty Into Account… Proceedings of the NCSL International Workshop and Symposium. Orlando, FL. July 2014. [A149A] J. Harben, Descriptive Implementation of Prior Knowledge. Proceedings of the NCSL International Workshop and Symposium. Orlando, FL. July 2014. [A150] R. Long, Taking My Uncertainty Into Account… I Have My Uncertainties, Now What? Presentation at the NCSL International Workshop and Symposium. Orlando, FL. July 2014.

2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A150A] N. Finstrom, J. Strief, T. Haddad, E. Maass, K. Hulting, A Comparison of Measurement System Analysis Metrics: Part 1 of 2. iSixSigma. July 2014 [A150B] N. Finstrom, J. Strief, T. Haddad, E. Maass, K. Hulting, A Comparison of Measurement System Analysis Metrics: Part 2 of 2. iSixSigma. July 2014 [A151] L. Pendrill, Using Measurement Uncertainty in Decision-Making and Conformity Assessment. Metrologia. Vol. 51, No. 4, pp 206-218. August 2014. [A151A] C. Hegedűs, Risk-Based Consideration of Measurement Uncertainty in Decisions. Pannon Management Review. Vol.3, No. 3. Pp 73 – 96. September 2014. [A152] C. Burgess, Using the Guard Band to Determine a Risk-Based Specification. Pharmaceutical Technology. Vol. 38, No. 10. October 2014. [A153] ASME B89.7.2-2014, Dimensional Measurement Plan, Appendix D - Probabilities of Pass and Fail Errors. American Society of Mechanical Engineers. New York NY. December 2014. [A153A] C. Hegedűs, Support of Risk-Based Decisions in Conformity and Control Considering Measurement Uncertainty. PhD dissertation. University of Pannonia. Veszprém Hungary. 2014. [A154] C. Shakarji, S. Phillips, Understanding the Ramifications of Important Proposed Changes to Decision Rules in Three Standards Within ISO and ASME. Presentation at the NCSL International Workshop and Symposium. Grapevine TX. July 2015. [A155] J. Sims, Process Accuracy Ratio vs Process Uncertainty Ratio – Risk Mitigation: Calibration and the Customer’s Process. Transcat white-paper. Rochester NY. 2014. [A155A] M. Kuster, Resolving Resolution Uncertainty. Measure – The Journal of Measurement Science. NCSL International. Vol. 10, No. 3, September 2015. [A156] A. Dregelyi-Kiss, Performance Verification of Dimensional Measuring Instruments in Automotive Industry. Proceedings of First International Academy for Quality World Quality Forum (IAQ-WQF). Budapest Hungary. October 2015. [A157] ISO 14253-5:2015, Geometrical Product Specifications (GPS) — Inspection by Measurement of Workpieces and Measuring Equipment — Part 5: Uncertainty in Verification Testing of Indicating Measuring Instruments. International Organization for Standardization. Geneva Switzerland. 2015. [A158] Eurachem/CITAC STMU, Guide on Setting and Using Target Uncertainty in Chemical Measurement. Eurachem and the Cooperation on International Traceability in Analytical Chemistry. ISBN 978-989-98723-7-0. First Edition. 2015. [A159] C. Burgess, P. Curry, D. LeBlond, G. Gratzl, E. Kovacs, G. Martin, R McGregor, P. Netthercote, H. Pappa, J. Weitzel. Fitness for Use: Decision Rules and Target Measurement Uncertainty (Stimuli to the Revision Process). U. S. Pharmacopeial Convention (USP). Vol. 42, No. 2. February 2016. [A160] D. Jackson, Equipment Level Risk Assessment. Proceedings of the Measurement Science Conference. Anaheim CA. March 2016. [A161] R. Stern, J. Harben, Conformance Decision Rules to Support ISO/IEC CD2 17025 Under Revision. Proceedings of the NCSL International Workshop and Symposium. St. Paul MN. July 2016. 2016 NCSL International Workshop & Symposium

Appendix A (Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)

[A162] I. Lin, The Influence of Measurement Uncertainty in Conformity Assessment. NCSL International Workshop and Symposium. Conference Poster Session. St. Paul MN. July 2016. [A163] J. Salsbury, Understanding the Test Measurand and the Profound Impact on Calibration, Verification, and Uncertainty. Proceedings of the NCSL International Workshop and Symposium. St. Paul MN. July 2016.

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

MIL-STD-120 (1950): [2] “2.3.3 In general practice, gage tolerances should not exceed 10 percent of the tolerance of the part to be gaged [10:1 TAR]. 2.3.6 Disputed Rejections. Any part which is so close to either rejection limit as to be improperly rejected [false reject] either as a result of tolerance or wear of the inspection gages, will be reinspected…Any observational errors in the inspection must, however, be in the direction of safety rather than in the direction of danger of acceptance of improper parts [false accept]. 6.2.5 Gages in use must be checked periodically to insure that they have not worn to the extent that they accept defective material or reject acceptable material. 8.3.4.3 When the dimensions of gages are very close to the specified limits… the gage inspector will often be required to exercise considerable judgment in deciding whether or not the gage should be accepted. For example, a plain “go” plug gage is given a plus tolerance. If the plus tolerance should be a little greater than that specified, the plug gage will tend to reject a few more components than a properly made plug gage [false-reject]. On the other hand, if a plain “Go” plug gage should have a diameter less than nominal, in other words, if it is undersize, the gage could pass unsatisfactory components [false-accept]. 8.3.5.2 The measuring instruments used should always be suitable for the tolerance specified on the gage drawings… In general, the accuracy of the measuring instruments should be less than 20 percent [5:1 TAR] of the tolerance on the gage being inspected. A measuring instrument which has an accuracy of 10 percent [10:1 TAR] of the gage tolerance should be used whenever such an instrument is available provided that its use does not involve an excessive expenditure of time. When the acceptability of gage is questionable because it is near the tolerance limit, the gage may be re-inspected by more accurate instruments.”

NAVY: Tech Memo 63-106 (1955): [A5] “The value of a program aimed at improving and ensuring the reliability… is degraded if the quality and proof tests to which… components and systems are subjected, are not in themselves reliable. Considerable effort has been expended on these relationships by Mr. Alan R. Eagle and Dr. Frank E. Grubbs. Their work has resulted in a mathematical expression for the relationship between design [specification] and test tolerance [guard band] and allowable instrumentation error [measurement uncertainty]. …to ensure… a specific testing risk or reliability can be attained… establish a reasonable ratio between A.I.E. [allowable instrumentation error – or measurement uncertainty] and the Design Tolerance [specification of the item being tested] based on known accuracy limitation of the measuring instrument available.”

MIL-C-45662 (1960): [12] “3.1.2 Adequacy of Standards: Standards established by the contractor for this calibration system shall be those authorized or recognized as having capabilities for accuracy, stability, and range required for the intended use. 3.1.5 Calibration Procedures. Written procedures shall be prepared and utilized for calibration of all standards used to control the accuracy of the measuring and test equipment involved in establishing product conformance. The procedure shall require that calibration be performed by the comparison with higher level standard utilizing a ten to one (10:1) accuracy ratio objective. Cross checking between equal or near equal standard shall not be acceptable except for those cases where the state of the art does not provide a higher ratio”.

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

NASA NPC 200-2 (1962): [174] “9.3(b) Within state-of-the art limitations, the standards used for calibration of inspection, measuring, and test equipment shall have a tolerance no greater than 10 % of the allowable tolerance for the equipment being calibrated. 9.4 Evaluation. …evaluations shall determine the accuracy of the inspection, measuring, and test equipment for determining the quality of the articles. Evaluations shall determine the amount of the specified product tolerance that will be occupied by the inspection, measuring, and test equipment tolerances and inaccuracies. “…tests shall provide assurance that the indicated measurements [errors] are those of the article being measured and not of the test equipment itself.” MIL-HDBK-52 (1964): [13] “3.2.1(c) “…assure that measuring and test equipment used in product inspection is calibrated against reference standards having values of 4 to 10 times their accuracy. Also, the reference standards used in the contractor's system should be calibrated against higher level standards having recommended values of 4 to 10 times the accuracy of the reference standards. A value of less than 4 times the accuracy level may be acceptable to the government representative for those requirements which approach the limits of knowledge for given characteristic ."

NASA NHB 5300.2 (1965): [175] “3.3 Selection. The selection of standards and ‘equipment’ will be based on the following criteria: a. The accuracy, stability, and range of standard and ‘equipment’ will be in conformance with end-use requirements. Within state-of-the-art limitations, the standard used for calibration of inspection, measuring and test equipment will have a tolerance no greater than 10 per cent of the allowable tolerance of the ‘equipment’ being calibrated, i.e., I the item to be calibrated has a tolerance of ±1.0 per cent, the maximum allowable tolerance of the standard should be ±0.1 per cent. If the state-of-the-art prevents the attainment of this 10 to 1 ratio, authorization must be secured prior to use.”

NASA NHB-5300.4-1B (1969): [176] Formerly NPC 200-2 “1B903. Article or Material Measurement Process. Random and systematic errors in any article or material measurement process shall not exceed 10 % of the tolerance of the article or material characteristic being measured. Authorization for exemption shall be required from the procuring NASA installation. 1B904. Calibration and Measurement Processes. Random and systematic errors in any calibration measurement process shall not exceed 25 % of the tolerance of the parameter being measured. Authorization for exemption shall be required from the procuring NASA installation”. RDT F3-2T (1969): [203] “3.2 (a) Standards established by the manufacturer for calibrating the measuring and test equipment used to control and verify product quality shall be accurate and stable over the range of calibration necessary to assure the product requirements are met and that any deviation from product requirements are accurately known...(d) The procedure shall require that calibration be performed by comparison with higher accuracy level standards.”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

IEEE STD-498-1975 (1975): [112] “3.1 Adequacy of Reference Standards: Reference standards used for calibrating M&TE shall have an accuracy level, acceptable calibration ranges, and precisions that are equal to or better than those required of M&TE. The accuracies of the M&TE and the reference standard should be chosen such that the equipment being calibrated can be calibrated and maintained within the required tolerances.”

NATO AQAP-6 (1976): [14] “204. All measurements, whether made for purposes of calibration or product characteristic assessment, shall take into account the total error in the measurement process attributable to the standard, or measuring equipment, and as appropriate those contributed by personnel, procedure and environment. 206 …measuring equipment and standards shall be the subject of records which include… a statement of the cumulative effects of errors on the data obtained in the calibration… 214. The cumulative effect of the errors in each successive component of a calibration chain shall be taken into account for each standard or measure equipment calibrated. Corrective action shall be taken when the total error is such as to compromise significantly the ability to make measurements within the designated limits. The basis for the calculation of the cumulative error shall be recorded”.

NATO AQAP-7 (1978): [15] “201 …use measuring devices and techniques of a suitably higher degree of accuracy than the tolerance of the product parameters. 203… ensure that standard and measuring equipment… are of the accuracy, stability, and range appropriate to the intended applications. A significant factor in planning for… measurement.., is the… determination of what measurement equipment will be required, what its accuracy must be… 204 Error is inherent in all measurement processes, whether performed for purposes of calibration, or product characteristic measurement. Valid estimates of the amount of error can be determined and must be taken into account, as appropriate, not only in the use of standards in subsequent calibrations, but also in the use of measuring equipment when making decisions on whether measured values of product characteristics fall within specified limits”.

IEEE STD-498-1980 Section 4.1 (1980): [113] and IEEE STD 498-1985 Section 5.1 (1985): [114] “Adequacy of Reference Standards: Reference standards used for calibrating M&TE shall have an accuracy level, acceptable calibration ranges, and precisions that are equal to or better than those required of M&TE. The accuracies of the M&TE and the reference standard should be chosen such that the equipment being calibrated can be calibrated and maintained within the required tolerances. In general, the inaccuracy of the reference standard shall contribute no more than one fourth of the allowable measuring and test tolerance. However, when the actual inaccuracy of the measuring and test equipment is less than on fourth of the plant equipment tolerance, the requirement of one fourth the tolerance between the reference standard and measuring at test equipment may not be necessary. The rational for deviating from these requirement must be justified and documented.”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

MIL-HDBK-52A (1984): [31] “The standards used for calibrating M&TE shall have capabilities for accuracy, stability, range and resolution required for the intended use. Accuracy ratios may be used for determining adequacy. The accuracy ratio may be high (4:1, 10:1, or higher) or low (3:1, or 2:1). The exact ratio depends on state-of-the-art limitations, and the contactor’s measurement requirements. The Government representative must verify that standards used by the contractor are capable of calibrating the M&TE for the intended use. The accuracy of the standards must at least be equal to the tolerance required (i.e., 1:1) but in most cases should be greater. A 1:1 comparison is permitted only when state-of-the-art limitations preclude a higher accuracy ratio. Normally, when only a 1:1 accuracy ratio can be achieved, any out of tolerance condition of the M&TE will be significant.” ANSI/ASQC M1-1987 (1987): [74] (Reaffirmed in 1996)

“2.5 Accuracy Ratio: The Ratio of the tolerance of the instrument being calibrated to the uncertainty of the standard. 4.3 Calibration ensembles shall have accuracies sufficient for the purpose of the calibration performed with them. 4.3.1 Calibration Uncertainties. The uncertainty of calibration ensembles shall be taken into account by increasing the limits of uncertainty of the calibration or by decreasing the acceptance/rejection limits of tolerance of the calibration by an amount at least equal to the uncertainty of the calibration ensemble, unless such uncertainty is acceptable or negligible with respect to the required uncertainty limits. 4.3.2 Accuracy Ratios. When the use of a fixed accuracy ratio is determined to be acceptable to establish a sufficient level of measurement accuracy between the ensemble to be calibrated and the calibration ensemble, the ratio and rational for its selection shall be clearly specified. Comments: The specification in this standard of a minimum accuracy ratio is considered invalid because the acceptable risk associated with a measurement process may vary with each process, thereby suggesting varying accuracy ratios. Appendix. In the application of the more traditional approach to calibration involving accuracy ratios, one typically find requirements to the effect that any instrument or standard used to calibrate another device must have an accuracy 10 times better than that of the device being calibrated. As an example, a device having a ±0.5 percent tolerance might be calibrated using a standard having an uncertainty of ±0.05 percent or better, and in turn might be calibrated using an item having an uncertainty of ±0.005 percent or better… In such a process, one makes no attempt to determine precisely the actual errors introduced at each step. It is assumed that because of the built-in ‘safety factor’ of 10 at each level, the calibration hierarchy will produce devices of adequate accuracy. In most cases, experience shows that this works quite well, provided devices having the necessary accuracy are available and providing the waste of information in tolerable.”

MIL-STD-45662A (1988): [29] “5.2 Measurement standards used… for calibrating M&TE and other measurement standards shall be traceable and shall have the accuracy, stability, range and resolution required for the intended use. Unless otherwise specified in contract requirements, the collective uncertainty of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each characteristic being calibrated. The… calibration system description may include provisions for deviating for the uncertainty requirements, provided the adequacy of the calibration is not degraded. All deviations must be documented.”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

MIL-HDBK-52B (1989): [177] “5.2 Adequacy of measurement standards. Measurement standards used by the contractor for calibrating M&TE and other measurement standards shall be traceable and shall have the accuracy, stability, range and resolution required for the intended use. Unless otherwise specified in the contract requirements, the collective uncertainty of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each characteristic being calibrated. The contractor's calibration uncertainty requirements provided t h e adequacy of t h e calibration is not degraded. All deviations shall be documented. Concept: The two principal and essential requirements for measurement standards are traceability and accuracy. The traceability requirement is as defined in paragraph 3.5. The accuracy requirement is resolved by comparing the accuracy attributes of the measurement standard (expressed as uncertainty) to the tolerances of the characteristics being calibrated. Since it is recognized that it is not possible to have a perfect measurement standard (i.e., one without any error or uncertainty), some amount of uncertainty is allowable. As stated, the MIL-STD provides for an uncertainty limit of 25% of the tolerance of the characteristic being calibrated unless otherwise specified in the contract. This is commonly called a 4:1 test accuracy ratio. When test accuracy ratios of 4:1 or greater are maintained, the error contributed by the measurement standards is considered negligible. However, when lesser ratios are used due to state-of-the-art or other limitations, the relatively larger error of the measurement standards increases the risks of erroneous acceptance or rejection of the instrument being calibrated. These conditions could ultimately lead to increased acceptance of bad material by the Government or rejection of good material by the contractor . Application: The adequacy of a measurement standard for its intended use is determined by comparing its accuracy, stability, range and resolution to the expected values and tolerances of the instrument’s characteristic being calibrated. The collective uncertainty of the measurement standards used to perform the calibration is typically a combination of the inherent accuracy, including stability and resolution, of each standard comprising the calibration system. For purpose of compliance to paragraph 5.2 of the MIL-STD, collective uncertainty of the measurement standards does not include other possible sources of errors in the use of the standards, such as technician or procedure. Methods for combining uncertainties of the characteristics of more than one measurement standard include simple arithmetic addition, root sum of squares (RSS), use of partial derivatives, distribution analysis, or some combination of these or other methods. The contractor may use any or all applicable methods but should describe and justify the methodology used in his calibration system description. If there are questions concerning the contractor's methods and conclusions in developing collective uncertainty estimates, the Government representative should request assistance from the Metrology Engineering Center of the cognizant DOD agency. To verify the adequacy of a measurement standard or group of measurement standards for its intended use, the Government representative should review the calibration procedure in which the measurement standards are used. Since paragraph5 .5 of the MIL-STD requires that the calibration procedure contain the required parameter, range and accuracy of each measurement standard and the acceptable tolerance of the instrument characteristic being calibrated, comparison of the uncertainty values should reflect a minimum 4:1 test accuracy ratio. Such comparisons should always be in terms of measurement units( e.g., volts, ohms, etc.) and not percentages or parts per million (ppm) unless percentages and ppm are the measurement units (e.g., voltage divider linearity). The reason for using units of measurement versus percentages is demonstrated in the following example. If a ±1% of full scale 0-15 psig working pressure gage were to be calibrated against ±0.1% of full scale 0-100 psig precision pressure gage, it would appear that there is a 10:1 test accuracy ratio if only percentage uncertainties were compared. However, since the percentage uncertainties for both test and precision gages apply to the full-scale values, in terms of measurement units the calibration tolerance of the test gage is ±0.15 psig (±1% of 15 psig) and the measurement uncertainty of the-precision pressure gage is ±0.1 psig (±0.1% of 100 psig). The true test accuracy ratio is then 1.5:1 rather than 10:1 and the precision gage does not meet the minimum test accuracy ratio requirement.”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

MIL-HDBK-52B (1989): Continued. “When a minimum 4:1 test accuracy ratio is not achieved or, stated differently, when the collective uncertainty of the measurement standards exceeds 25% of the acceptable tolerance of the characteristic being calibrated, the contractor should document such deviations and demonstrate how the adequacy of the calibration is not degraded. (The calibration would be degraded if there were and adverse impact to the material or services being delivered to the Government.) Using an example of a measurement standard calibrating M&TE, the following are approaches that maybe taken by the contractor when a minimum 4:1 test accuracy ratio is not achieved: a. The M&TE being calibrated could be derated (test tolerances increased). This option is appropriate when all applications of the instrument are known and the derated tolerances are consistent with the known usage and/or contractual requirements, The M&TE should be labeled as to the specific derating, In effect, this approach may increase the test accuracy ratio to 4:1 or greater for the specific calibration. b. The M&TE test tolerances could be reduced to compensate for the uncertainty of the measurement standards [i.e. guardbanding]. While this would result in an increase of erroneous out-of-tolerance findings for the M&TE during the calibration process, it should not have an adverse impact on the material to be delivered to the Government. This approach should be used sparingly since the increased out-of-tolerance results during the calibration may result in excessive repairs and recalibrations. The contractor's calibration system could also provide for instances where test accuracy ratio is not a significant factor. This could arise when the value and uncertainty of the known measurement standard is essentially transferred to the M&TE unknown measurement standard through a relatively simple intercomparison process. Other typical but more complex methods for obviating the test accuracy ratio requirement are through Measurement Assurance Programs (MAPS) and use of statistical process control (SPC) or statistical quality control in the measurement system. If the contractor uses any of these or similar methods, the Government representative should assure that the deviation from the test accuracy ratio requirement is documented, that the rationale and methodology supporting the deviation are adequate and the adequacy of the calibration is not degraded. If there are questions concerning the acceptability of the proposed deviation, the Government representative should request assistance from the Metrology Engineering Center of the cognizant DOE) agency. To be acceptable for use in the contractor's calibration system, measurement standards must also be traceable as defined in paragraph 3.5 of the MIL-STD, and supported by an appropriate certificate or report of calibration. Paragraph 5.8 of this handbook provides further details concerning traceability requirements for measurement standards. In addition to traceability and accuracy requirements, the Government representative should confirm that the contractor reviews calibration history of the measurement standards to ensure that appropriate reliability in terms of calibration interval is maintained. (See paragraph 5.4 of the MIL-STD.)”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

IEEE STD-498-1990 (1990): [115] “1.2 Applicability. Selection of measuring and test equipment shall be controlled to assure that such items are of the proper type, range, accuracy, resolution, and tolerance to accomplish the function of determining conformance to specified requirements. 5.1 Adequacy of Reference Standards: Reference standards used for calibrating measuring and test equipment shall have calibrated ranges, precisions and accuracies such that the measuring and test equipment and ultimately the facility equipment can be and maintained within the required tolerances. In general, the inaccuracy of the reference standard shall contribute no more than one fourth of the allowable measuring and test tolerance. That is, reference standards utilized to calibrate measuring and test equipment shall have a minimum accuracy four times greater than the measuring and test equipment being calibrated. The accuracy requirement is satisfied with M&TE is calibrated by a reference standard with a minimum 4:1 higher accuracy ratio, and when that reference standard is traceable through a series of calibrations, using reference standards also of 4:1 greater accuracy, back to the National Institute of Standards and Technology (NIST) or other appropriate governing laboratory standard. This is depicted in Fig 1. If it is impractical to maintain these accuracy ratios, the rational for deviating from these requirements shall be justified, documented, and authorized by responsible management”.

ISO 10012-1 (1992): [20] “The error attributable to calibration should be as small as possible. In most areas of measurement, it should be no more than one third and preferably one tenth of the permissible error of the confirmed equipment when in use”.

MIL-PRF-38793A (1993): [178] “3.2.3.2 A measurement system shall consist of all recommended calibration equipment of combinations thereof and have a uncertainty equal to or better than one-fourth of the uncertainty of the test instrument for each parameter tested, i.e. a test accuracy ratio (TAR) (see 6.4.5) equal to or better than four to one. Measurement systems of better TAR may be used for reasons of equipment availability, reduced complexity, or reduced calibration time. If a TAR equal to or better than four-to-one is not feasible because of state-of-theart or other technical considerations, the best TAR available shall be used. In such cases, the actual TAR shall be stated in the procedure. Where several items of equipment are applicable, preference shall be given to the equipment affording the simplest, quickest, and most efficient test method. 6.4.5 Test Accuracy Ratio: The maximum permitted error of the unit to be measured or calibrated, divided by the maximum known error of the measuring or generating device used to perform the measurement. For example, if it is required that a system or equipment output parameter be accurate to 8 % (maximum permitted error) and the known accuracy (maximum known error) of the measuring device used to measure the output parameter is 2 %, then the TAR is 4.”

DOE-STD-1054-93 (1993): [179] “3.4.4.1 M&TE should be calibrated using reference standards (secondary or working) whose calibration has a known valid relationship to nationally recognized standards or accepted values of natural physical constants… The reference standard used should have an accuracy at least four times greater than the device under test. If this accuracy ratio cannot be met, analysis of the error should be estimated to provide a valid uncertainty of the calibration process.”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

DOE 4330.4B (1994): [180] “12.1 … The M&TE selected for use should have the precision necessary to ensure that facility instrumentation and equipment will operate within design accuracy requirements and be durable enough for their intended application. 12.3.2 (b) Calibration Procedures. …The ‘ratio of accuracy’ of the standard to the M&TE being calibrated should be as high as reasonably achievable and consistent with national standards. These procedures should… contain the following: …accuracy of calibration standards to be used; calibration standards to be used…” Calibration Standards: NASA KHB 5330.9A (1994): [181] “2.10 (i) Select calibration standards to attain a minimum calibration accuracy ratio of 4:1. When state of the art limitations prevent attainment of the minimum accuracy ratio, the measurement uncertainty must be reported as part of the calibration data record. Users of equipment calibrated to less than 4:1 accuracy ratios must account for the reported instrument uncertainty when establishing end-item test pass/fail criteria.”

ANSI/NCSL Z540.1-1994 (1994): [35] “10.2 (b) The laboratory shall ensure that the calibration uncertainties are sufficiently small so that the adequacy of the measurement is not affected. Well defined and documented measurement assurance techniques or uncertainty analysis may be used to verify the adequacy of a measurement process. If such techniques or analyses are not used, then the collective uncertainty of the measurement standards shall not exceed 25% of the acceptable tolerance (e.g. manufacturer’s specification) for each characteristic of the measuring and test equipment being calibrated”.

NCSL Z540.1 Handbook (1995): [181] “10.2(b) The standards selected and their associated uncertainties or tolerances should be adequate for the calibration/verification being performed. Example(s): a) An appropriate method or measurement technique is one that meets the needs of the calibration/verification being performed. (You don’t need to use a gage block to calibrate a yardstick and you can’t use a yardstick to calibrate a gage block). b) … When using the standards prescribed by the procedure, it is recommended that the uncertainties and related Test Accuracy Ratio (TAR) be reviewed. c) …As a default alternative to doing an uncertainty analysis, a laboratory may rely on a Test Accuracy Ratio (TAR) of 4:1. A TAR of 4:1 means that the tolerance of the parameter (specification) being tested is equal to or greater than four times the combination of the uncertainties of all the measurement standards employed in the test. If it is determined that the TAR is less than 4:1, then one of the following method may be used: uncertainty analysis as described above, guard-banding, widening the specification, or another appropriate method”.

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

DOE RW-0333P Rev.5 (1995): [182] “12.2.1 (B) Calibration standards shall have a greater accuracy than the required accuracy of the measuring and test equipment being calibrated. 1. If calibration standards with greater accuracy than required of the measuring and test equipment being calibrated do not exists or are unavailable, calibration standards with accuracy equal to the required calibration accuracy may be used if they can be shown to be adequate to the requirements. 2. The basis for the calibration acceptance shall be documented and authorized by responsible management. The level of management authorized to perform this function shall be identified”.

MIL-HDBK-1839 (1996): [183] “5.3.3 Test accuracy ratio (TAR): Unless otherwise specified, the recommended TMDE shall be capable of measuring or generating to a higher accuracy than the measurement parameters being supported. A minimum TAR of 4 to 1 is required. The actual TAR shall be documented. a. If a TAR of 4 to 1 cannot be achieved, the contractor shall analyze the measurement requirements and justify the lesser TAR. b. A minimum TAR of 4 to 1 is required when an actual test is in conducted to characterize performance of operational equipment or to caliber other TMDE. c.

A TAR of 4 to 1 is not required when the TMDE only provides input stimuli which is not used to characterize performance of the operational equipment of other TMDE. In this case, the TAR does not need to be greater that 1 to 1”.

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

MIL-HDBK-1839A (2000): [184] “3.5 Test Uncertainty Ratio: The total uncertainty of the unit to be measured or calibrated divided by the total uncertainty of the measuring or generating device used to perform the measurement. For example, if it is required that a system or equipment output parameter’s uncertainty is 8% and the uncertainty of the measuring device used to measure the output parameter is 2%, then the TUR is 8 to 2 or 4 to 1. Interpretive guidance: The desired TUR is 4 to 1. TUR is the calculated result of dividing the uncertainty or tolerance of a measurement/input requirement by the uncertainty/tolerance of the equipment satisfying the requirement. TURs are generally calculated for each operational system test requirement that is supported by SE [support equipment] and for SE supported (calibrated) by SE. The maximum permitted uncertainty should reflect the actual use requirement of the unit being measured/calibrated for the intended use. Tolerances used in TUR calculations will be in the same measurement units. Uncertainties expressed in logarithmic units such as decibels should be considered. Decibels should first be converted to linear units before computing TUR. For example, suppose the system uncertainty and the measurement uncertainty is 4db and 1db respectively. The db ratio may be 4 to 1 but the linear ratio may be only 2 to 1 or a 3db gain is generally double in value when dealing with power measurements. Caution should be used when uncertainties are stated as percentages because they often are not related (i.e. percentage of full scale versus percentage of reading). Tolerances expressed as ‘minimum’, ‘maximum’, ‘less than’, ‘more than’, ‘greater than’ and ‘less than’, do not allow TURs to be calculated, thus using them is not desired. However, when no other option exists, the design activity should specify the recommended uncertainty of the supporting equipment (see 5.4.3). 5.4.3 Test Uncertainty Ratio: The recommended TMDE shall be capable of measuring or generating to a higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum TUR of 4 to 1 is desired. The actual TUR shall be documented. Interpretive guidance: TUR is a recognized mechanism for establishing the criteria between equipment listed in proximity categories in the [CMRS Calibration and Measurement Requirements Summary]. Actually, use of the TUR is an alternative to performing a more difficult uncertainty analysis as described in ANSI/NCSL Z5402-1997. The TUR does not relate directly to uncertainty analysis. However, it can be linked to the probability of making erroneous test decisions. Another important point, design activities often select SE that is more accurate than required for the application due to availability, future planning, cost considerations, etc. That is perfectly acceptable but in the application, as defined by the CMRS, the specifications should be derated to the 4 to 1 TUR so as not to create unnecessarily accurate and costly support requirements for the SE. Creation of unnecessarily over/under specified problems are minimized when the design activity emphasizes concurrent engineering processes where the logistics support planning and design engineering are time related and maintain a close liaison. The importance of not over specifying a piece of support equipment cannot be over emphasized as substantial unnecessary costs can be incurred by designs using overly stringent specifications. a. If a TUR of 4 to 1, or the specified TUR, cannot be achieved, the design activity shall analyze the measurement requirements and provide documented justification for the lesser TUR. Interpretive guidance: It is highly recommended that if the CMRS preparing activity contact their designated service CMRS review agent to discuss any shortcomings in meeting specified TUR prior to submittal. b. A TUR of 4 to 1 is not required when the TMDE only provides input stimuli which are not used to characterize performance of the operational equipment or other TMDE. In this case, a minimum TUR of 1 to 1 is acceptable. Interpretive guidance: A TUR of 1 to 1 is acceptable between the system and an item of support equipment only when the system requires an input, stimulus, or applied value for establishing test conditions and the system itself does not measure the applied parameters or characterize its own performance. Examples of acceptable 1 to 1 TURs are input/supplied torque applications, electrical stimuli signals, and pressure inputs. A minimum TUR of 1 to 1 only applies between the system and the SE. It does not apply between SE and calibration equipment. A 4 to 1 TUR is desired when the system measures an applied/input parameter and the results determine pass/fail status or the systems performance is characterized.” 2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

ASME B89.7.3.1-2001 (2001): [53] “4.1 Simple Acceptance and Rejection Using an N:1 Decision Rule. This is the most common form of acceptance and rejection used in industry and is the descendant of MIL-STD 45662A. Simple acceptance means that product conformance is verified if the measurement result lies in the specification zone and rejection is verified otherwise (see Fig. 2), provided that the magnitude of the measurement uncertainty interval is no larger than the fraction 1/N of the specification zone. In recent years, as tolerances have become increasingly tighter, the well-known ten-to-one ratio has transitioned to a more commonly used ratio of four-to-one (see MILSTD 45662A) or even three-to-one (see International Standard 10012-1). A 4:1 decision rule means the uncertainty interval associated with the measurement result should be no larger than one-fourth of the allowable product variation, which requires the expanded uncertainty, U, to be no larger than one-eighth of the specification zone. Once the uncertainty requirement is satisfied, then the product is accepted if the measurement result lies within the specification zone and rejected otherwise”.

ISO 17025:2005 (2005): [27] 1st edition published 1999

“5.10.3.1 Test Reports should include …where relevant, a statement of compliance/non-compliance with requirements or specifications. …uncertainty is needed in test reports when…it affects compliance to a specification limit. 5.10.4.2 When statements of compliance are made, the uncertainty of measurement shall be taken into account.”

NASA NSTS 5300.4(1D-2) w/ Change No. 9 (2006): [185] Note: Hyperlink is to older version (Chg No. 2 - 2000)

1D507 “4. Article or Material Measurement Processes. The expanded uncertainty in any article or material measurement process shall not exceed 10 percent of the tolerance of the article or material characteristic being measured… 5. Calibration Measurement Process. ANSI/NCSL Z540.1-1994 (R2002), ‘Calibration Laboratories and Measuring and Test Equipment – General Requirements’ shall be the standard for calibration processes. Per this standard, uncertainty analysis may be used to verify the adequacy of a calibration Measurement processes… If this technique is not used, the expanded uncertainty in any calibration measurement process shall not exceed 25 percent of the tolerance of the parameter being measured”.

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

DOE RW-0333P Rev.21 (2009): [186] “12.2.1 (B) Calibration standards shall have a greater accuracy than the required accuracy of the measuring and test equipment being calibrated. 1. If calibration standards with greater accuracy than required of the measuring and test equipment being calibrated do not exists or are unavailable, calibration standards with accuracy equal to the required calibration accuracy may be used if they can be shown to be adequate to the requirements. 2. The basis for the calibration acceptance shall be documented and authorized by responsible management. The level of management authorized to perform this function shall be identified. (C) Calibration standards used for the calibration of M&TE shall have an accuracy of at least four times the required accuracy of the equipment being calibrated or, when this is not possible, shall have an accuracy that ensures the equipment being calibrated will be within required tolerance.”

MIL-STD-1839D (2010): [187] “3.1.13 Test Accuracy Ratio. A simple comparison between the accuracy of the Unit Under Test (UUT) and the accuracy of the calibration standard. However, this ratio does not consider other potential sources of error in the calibration process. For example, if it is required that an equipment output parameter’s accuracy is ±8% and the accuracy of the single measuring device used to measure the output parameter is ±2%, the TAR is 8 to 2, or 4 to 1. 3.1.16 Test Uncertainty Ratio. The comparison between the accuracy of the Unit Under Test and the estimated calibration uncertainty is known as a Test Uncertainty Ratio (TUR). This ratio is more reliable because is accounts for possible sources of error in the calibration process that the TAR does not. This value also takes into account issues such as temperature, humidity, Type A (statistical analysis including actual and random measurement results), and Type B (anything not a Type A) uncertainties. 5.4.3 Test Uncertainty Ratio (TUR). The recommended TMDE shall be capable of measuring or generating to a higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum TUR of 4 to 1 is desired. The actual TUR shall be documented. If a TUR of 4 to 1, or the specified TUR, cannot be achieved, the design activity shall analyze the measurement requirements and provided documented justification for the lesser TUR. A TUR of 4 to 1 is not required when the TMDE only provides input stimuli which are not used to characterize performance of the operational equipment of other TMDE. In this case, a minimum TUR of 1 to 1 is acceptable.”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

Air Force T.O. 00-20-14 (2011): [188] “1.4.28 TAR/TUR. Test Accuracy Ratio/Test Uncertainty Ratio. The ratio of the uncertainty (accuracy) of the TI [Test Instrument or Unit-Under-Test] to the uncertainty (accuracy) of the standard (for example 4:1). TAR and TUR are equivalent in the Air Force Metrology and Calibration Program, as the term Accuracy has been used for the (more correct) term Uncertainty in Air Force Calibration procedures. 3.1.7 AFMETCAL shall ensure: …The calibration uncertainties are sufficiently small so that the adequacy of the measurement is not affected. Well-defined and documented measurement assurance techniques or uncertainty analyses may be used to verify the adequacy of a measurement process. If such techniques or analyses are not used, then the collective uncertainty of the measurement standards shall not exceed 25 percent (4:1) of the acceptable tolerance for each characteristic of the measuring and test equipment being calibrated or verified” L-A-B LABAG-502 Accreditation Guidance. (2011): [189] “3.9. TAR – Test Accuracy Ratio, a ratio of the collective uncertainties of the equipment used to the accuracy specification(s) of the unit under test. 3.9.1. Test Accuracy Ratio (incorrect), a ratio of the collective accuracies of the equipment used to the accuracy specification(s) of the unit under test. (This incorrect definition was commonly used due to the use of the term “Accuracy Ratio”) 3.10. TUR – Test Uncertainty Ratio, a ratio of the collective uncertainties of the equipment used to the measurement uncertainties of the unit under test. A technical review of the calibration certificate’s content will evaluate the following; * the measurement uncertainty is sufficiently low for your application, * the data’s impact on the laboratory’s work (data beyond prescribed limits such as manufacturer’s specifications or calibration tolerances, requires an investigation to determine if there is non-conforming work). * ISO/IEC 17025:2005 Section 5.10.4.1c states that calibration certificates shall include “the uncertainty of measurement and/or a statement of compliance with an identified metrological specification or clause thereof”. * a calibration certificate may make a statement of compliance and contain a statement referring to a TAR or TUR of 4:1 or greater. Neither a TAR nor a TUR is acceptable as a measurement uncertainty for a unit under test. NOTE: If a laboratory can calculate a proper TUR, then a measurement uncertainty for the unit under test has been calculated and should be available upon request”.

A2LA P102 Policy on Measurement Traceability (2015): [190] “It is often the case that a calibration certificate will contain the statement “in tolerance”, or words to that effect, along with a statement to the effect that the measurement uncertainty does not exceed a certain fraction of the tolerance. Such fractions are often called “test uncertainty ratios”, TURs for short. Uncertainty statements phrased in terms of TURs are implicit statements of the uncertainty: knowing the tolerance ratio allows one to determine the largest possible value of the measurement uncertainty. Implicit statements of uncertainty are acceptable on accredited calibration certificates as long as the measurement uncertainty and the measurement results are also provided”.

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

A2LA R205 Specific Requirements: Calibration Laboratory Accreditation Program (2015): [191] “6.5 Statements of Compliance Laboratories are permitted to issue certificates with a statement of compliance (e.g., conformance to a specification) relating to the metrological aspects of specifications. In such cases the laboratory shall ensure that: 1) The specification is a national or international standard or one that has been agreed to or defined by the customer; 2) The measurement needed to determine conformance are within the accredited scope of the laboratory; 3) When parameters are verified to be within specified tolerance, the associated uncertainty of the measurement result is properly taken into account with respect to the tolerance by a documented procedure or policy implemented by the laboratory that defines the decision rules used by the laboratory for declaring in or out of tolerance conditions; 4) The laboratory shall ensure the decision rule used meets the needs of the customer; 5) When parameters are verified to be within specified tolerance, the calibration laboratory shall determine the uncertainty and take that uncertainty into account when issuing the statement of compliance. In addition, no claim of compliance shall be made when the measurement uncertainty is larger than the tolerance being evaluated. An exception can be made only in cases where the laboratory indicates in the contract with the client that the calibration results will be reported without factoring in the effect of uncertainty on the assessment of compliance, and the client agrees to the contract. In this case, the uncertainty can be excluded when making that statement of compliance on the calibration certificate. In effect, both parties share the risk that the results may or may not meet the specification since the uncertainty was not included when the results were determined. In these cases, the measurement uncertainty shall still be determined and shall be reported on the calibration certificate. 6) In accordance with ILAC P14:01/2013 ILAC Policy for Uncertainty in Calibration section 6.1, the calibration laboratory shall retain documentary evidence of the measured quantity value and the uncertainty of measurement as specified in ISO/IEC 17025 clauses 5.10.4.2 and 4.13, and shall provide such evidence upon request. 7) The certificate relates only to metrological quantities and states which clauses of the specification are verified to have been met”.

IAS AC204 CL/009 (2016): [192] “5.4 As applicable, assessments may include review of the 4:1 Test Uncertainty Ratio (TUR) requirement for each technical demonstration”.

ANAB (previously ACLASS) CL 2043 – Z540.3 Checklist (2016): [193] “In cases where it is not practicable to estimate PFA, does the laboratory ensure that the Test Uncertainty Ratio (TUR) is equal to or greater than 4:1?”

2016 NCSL International Workshop & Symposium

Appendix B (Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)

ANAB (previously ACLASS) MA 2002: ISO/IEC 17025 Accreditation Requirements for Calibration Laboratories (2016): [194] “In some instances, a calibration certificate may contain the statement “in tolerance,” along with a statement that the measurement uncertainty does not exceed a certain fraction of the tolerance. These fractions are often called test accuracy ratios (TARs) or test uncertainty ratios (TURs). Uncertainty statements phrased in terms of TARs or TURs are not acceptable for demonstrating measurement traceability or as an alternative to calculated MU [measurement uncertainty].”

IAS CL/014 Policy Guide on Calibration, Traceability, and Measurement Uncertainty for Calibration Laboratories. (2016): [195] “Calibration certificates and/or reports held by or issued by IAS accredited calibration laboratories must meet the requirements of ANSI/ISO/IEC Standard 17025:2005, Clause 5.10 (sub-clauses 5.10.2 and 5.10.4 are required, other sub-clauses as appropriate). Calibration certificates must include appropriate statements of uncertainty… All calibration certificates issued under the laboratory’s accreditation, whether or not indicating compliance to the 4:1 rule or other accepted metrological specification, must include the IAS logo and the uncertainty of the calibration... The American National Standard for calibration, ANSI/NCSL Z540-1-1994, known as Z540-1, has been formally retired effective in July 2007, although the standard continues to be used. The standard was replaced by ANSI/NCSL Z540.3-2006, known as Z540.3. There are significant differences between the two documents. The two most obvious differences are the alignment of Z540.3 to be consistent with the requirements of ANSI/ISO/IEC ISO 17025:2005 in Section 5.3… Z540-1 allowed the use of a Test Accuracy Ratio (TAR), with a minimum ratio of 4:1. Z540.3 allows the use of a Test Uncertainty Ratio (TUR), again with a minimum of 4:1. This provision is known as the 4:1 Rule. The laboratory may claim compliance to the 4:1 rule, providing the laboratory maintains evidence, including information regarding calibration of the laboratory’s measuring equipment and measurement uncertainty calculations, that it can consistently maintain the 4:1 ratio for any calibration where compliance with the 4:1 rule is stated. The evidence must be available for review by IAS upon request. If the 4:1 cannot be maintained, the laboratory must report either specific uncertainties or the actual TUR for any affected readings, as appropriate for the specific type of calibration… It is understood that some laboratory may have a need to be verified as compliant to the requirements of ANSI/NCSL Z540.3-2006, either solely for Section 5.3, or for the entire Standard. This will require additional assessment time and the laboratory must provide additional documentation as required by the Standard”.

2016 NCSL International Workshop & Symposium

Appendix C

2016 NCSL International Workshop & Symposium

Appendix C

2016 NCSL International Workshop & Symposium

Appendix C

2016 NCSL International Workshop & Symposium

Appendix C

2016 NCSL International Workshop & Symposium

Appendix D

FDA Good Guidance Practices (excerpts from 21 CFR §10.115) [79] “(a) What are good guidance practices? Good guidance practices (GGP's) are FDA's policies and procedures for developing, issuing, and using guidance documents. (b) What is a guidance document? (1) Guidance documents are documents prepared for FDA staff, applicants/sponsors, and the public that describe the agency's interpretation of or policy on a regulatory issue. (2) Guidance documents include, but are not limited to, documents that relate to: The design, production, labeling, promotion, manufacturing, and testing of regulated products; the processing, content, and evaluation or approval of submissions; and inspection and enforcement policies... (d) Are you or FDA required to follow a guidance document? (1) No. Guidance documents do not establish legally enforceable rights or responsibilities. They do not legally bind the public or FDA... (2) You may choose to use an approach other than the one set forth in a guidance document... (e) Can FDA use means other than a guidance document to communicate new agency policy or a new regulatory approach to a broad public audience? The agency may not use documents or other means of communication that are excluded from the definition of guidance document to informally communicate new or different regulatory expectations to a broad public audience for the first time. These GGP's must be followed whenever regulatory expectations that are not readily apparent from the statute or regulations are first communicated to a broad public audience. (f) How can you participate in the development and issuance of guidance documents? (1) You can provide input on guidance documents that FDA is developing under the procedures described in paragraph (g) of this section. (2) You can suggest areas for guidance document development. Your suggestions should address why a guidance document is necessary. (3) You can submit drafts of proposed guidance documents for FDA to consider. When you do so, you should mark the document ‘Guidance Document Submission’ and submit it to Division of Dockets Management (HFA-305), 5630 Fishers Lane, rm. 1061, Rockville, MD 20852...”

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

Pay Close Attention To Equipment Calibration Certificates, FDA Says By Shawn M. Schmitt [Editor's note: This article originally ran in the medical device industry publication The Silver Sheet in June 2010. The Silver Sheet was incorporated into The Gray Sheet in April 2015. Please visit TheGraySheet.com for in-depth regulatory and quality news, analysis, podcasts and features.] Too many device firms aren’t double-checking the quality of finished products after determining that manufacturing equipment was out of calibration, an FDA official says. “A large majority of manufacturers contract out their calibration activities. Because of this, what we typically see is that companies are blindly accepting calibration certificates from test houses,” FDA/GMP expert Kim Trautman told “The Silver Sheet.” Test houses are third parties used by firms to ensure that equipment used in the manufacture or the inspection and testing of their devices is properly calibrated and maintained. Calibration certificates are sent to firms describing the condition of the equipment as it was received by the test house, as well as its condition when it was returned to the manufacturer. “I have seen situations where a test house has sent equipment back to a manufacturer and the certificate said, ‘Yes, this piece of equipment was way out of calibration. It was XYZ, and we have now recalculated it,’” Trautman said. “So my question to the manufacturer is, what did you do with that information? Because you now know that there is finished product that was sold during this period of time when the instrument [used to manufacture it] was way out of calibration,” she said. At that point, a firm should investigate to ensure that the equipment in question did not have an adverse effect on devices already manufactured, Trautman said. In addition to equipment condition, test-house certificates also specify the calibration method that was used. Those test methods typically tie to national and international standards from the International Organization for Standardization (ISO), the American National Standards Institute (ANSI), the American Association for Laboratory Accreditation (A2LA), or another accrediting body. FDA requires equipment testing and calibration under Quality System Regulation Sec. 820.72, “Inspection, Measuring and Test Equipment,” as well as Sec. 820.70(g), “Production and Process Controls; Equipment.” Manufacturing equipment, as well as instruments used for inspection and testing, can range from a simple ruler, to widely used voltage meters, to complex custom-built test equipment that contains software – all of which “require accuracy and precision,” Trautman said. “All equipment has to be appropriate for its purpose, and it has to be maintained, adjusted and cleaned,” she said. Companies may calibrate equipment themselves; however, most firms use a combination of in- and out-ofhouse testing.

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

Firms also may have test-house representatives calibrate equipment at their manufacturing facilities if the equipment is too large to move or if the firms don’t have the expertise to conduct calibration activities on their own. Tracie Capozzio, director of quality assurance & regulatory compliance for NeuroWave Systems in Cleveland Heights, Ohio, says regulatory agencies, including FDA, are checking more thoroughly to see if manufacturers are examining whether finished devices may have been tainted by out-of-calibration equipment. During an agency inspection, “they’re usually looking for pieces of equipment that have come back out of calibration, and what you’ve done when they’ve come back out of calibration – whether or not you have evaluated the equipment that was used and how you went about identifying which product was processed with that equipment, and what the impact was,” Capozzio said. NeuroWave’s “procedures are written so that any product that was made during the time period between the last calibration, which was good, and the current calibration – if it was received at the calibration site out of calibration – is reviewed for any adverse impacts,” she said. Regulatory bodies, including FDA, are “focused a bit more now on inspection and test equipment than in the past, and you’re starting to see more regulatory activity related to equipment,” Capozzio said. FDA: Many Firms Don’t Look At Certificates Another problem FDA sees is that many companies are not adequately reviewing test-house certificates, and some do not look at the certificates at all. “Ninety percent of the time when there are problems [with manufacturing equipment] it’s because someone hasn’t even paid attention to the certificates that the calibration test house has given them,” Trautman said. “The manufacturers that get into these problems are those that just take the certificates, file them and don’t really look at them and utilize the information that the test house is giving them,” she said. “Kim Trautman is probably correct that 90 percent of companies file away the certificates, assuming that everything is good, but you can’t assume that,” NeuroWave’s Capozzio told “The Silver Sheet.” “It’s very easy to get a certificate from a certification house and just assume that they’re the experts, and when they give you a certificate, that means the equipment is good,” she said. “That assumption isn’t 100-percent false, though, because when the test house gives the equipment back and they give you a certificate, the equipment should be good at that point,” Capozzio said. “However, you have to be cognizant enough to know what the equipment was like when the test house received it.” NeuroWave recently received a CE mark for its first product, the NeuroSENSE Monitor, which monitors patient brain activity levels to help guide anesthesia and sedation in operating rooms and intensive care units. Although the device is not yet on the market, NeuroWave has produced valuation units. It uses several pieces of manufacturing equipment that it must calibrate, including signal generators, high-voltage probes, digital calipers, oscilloscopes and multimeters. Most of the firm’s equipment is sent to test houses for calibration, although some, such as electrostatic discharge equipment and an in-house built biosignal playback unit, is tested by NeuroWave itself.

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

The manufacturer’s biosignal playback unit “is not something we can send out for calibration because there is no standard known to a test house that it could be calibrated against,” Capozzio said. “In-house built equipment often requires special or unique maintenance and calibration instructions that can only be developed and implemented by the manufacturer of the equipment,” she said. At NeuroWave, “the test-house certificates that come back to us with equipment are reviewed, and after review, the person who is in charge of the calibration and maintenance program is responsible for looking at the different specifications that are critical for our product,” she said. Next, the certificates are initialed, indicating that they were reviewed for accuracy. “That is definitely one of the areas that regulatory agencies such as FDA look at during an inspection,” Capozzio said. “So we have it built into our system to do that every time the certificates come back in.” Improper Review Of Certificates Means Test-House Mistakes May Be Missed The key risk in not reviewing calibration certificates, of course, is that the manufacturer won’t notice that a test house has made a mistake – and that devices that have already gone out the door are out of spec. “You may find that you’re getting nonconformances kicked back, and your failure investigation shows that somehow this testing is either not being done right or not being done with the right equipment,” FDA’s Trautman said. “The investigation might lead you back to a piece of test equipment, and then possibly it might lead you back to a calibration test house not doing something right,” she said. Test-house errors indeed happen, admits Pete Sweetnam, VP of test house RS Calibration Services in Pleasanton, Calif. “Mistakes can take place, especially when you’re moving equipment back and forth,” he told “The Silver Sheet.” “We try to do the majority of our calibrations through a pick-up and drop-off system to avoid UPS and FedEx if we can. Even during the physical handling and movement of equipment, things can get damaged or become out of adjustment again. “There are certain pieces of equipment where there is a high tendency for that to happen, so we recommend that it not be moved, and that’s why we’ll go onsite and do the work,” added Sweetnam, whose test house is accredited to ISO, ANSI, A2LA and other organizations. Although FDA ideally wants manufacturers to adequately review calibration certificates, Trautman acknowledges that calibration errors may still slip through. “You can never prevent everything bad from happening,” she said. However, having a robust failure investigation program in place may mitigate any problems brought on by equipment improperly calibrated by a test house and not immediately detected when the equipment is accepted at the company, Trautman said. “In a circumstance where there is a product failure and a manufacturer can get down to a root cause [and determine that equipment was calibrated incorrectly], we’re not going to ding the firm during an investigation

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

for a calibration violation or a purchasing control violation, provided they have a robust failure system in place that they were following,” she said. “What’s more important to FDA is, what are you going to do about it now?” Trautman continued. “What are you going to do to fix the issue? Are you going to switch test houses? Are you going to increase the frequency of calibration? But we’re not going to say during an investigation, ‘Oh, this test house is terrible.’” CAPA Not Always Necessary In Cases Of Improper Calibration If a manufacturer finds that a piece of equipment was improperly calibrated by a test house, it should not automatically open a corrective and preventive action (CAPA), Trautman says. “CAPA is an escalating activity, but clearly there needs to be some elevation to at least explore or evaluate a correction,” she said. “Are you going to go back to your test house? Are you going to talk to this supplier about what happened?” Calibration test houses must be controlled under QSR Sec. 820.50, “Purchasing Controls,” because they supply a service. “If it’s a one-off and you can handle it through a nonconformance investigation, then that is OK,” Trautman said. “It needs to be handled, but how the manufacturer handles it is important, whether they do it via a nonconforming report or a particular correction.” However, “if it is elevated to a CAPA, then it says to me that something a little bit bigger systemically needs to happen with feedback to this particular supplier.” When NeuroWave notices trouble with poorly calibrated equipment, it first opens a nonconformance report. “It would depend on how broad the problem was and how many pieces of equipment may be affected,” Capozzio said. “If it’s isolated to one piece of equipment, we may not move it into a CAPA because our actions through our nonconformance process are very similar to what we do through CAPA,” she explained. “Unless it’s systemic in nature, we may not advance it to a CAPA unless there is some reason to do so.” Test-House Errors Are Manufacturer’s Responsibility Further, Trautman reminds manufacturers that the buck stops with them if a test house makes an error on a calibration certificate. “If that certificate doesn’t contain a piece of information that is required by the Quality System Regulation, the manufacturer can’t use the excuse that ‘This is all the calibration test house people gave me,’” she said. “It’s the manufacturer’s responsibility to make sure that they have the control, through supplier controls and contracts, to get the information that is necessary for that finished device manufacturer to meet regulatory requirements,” Trautman said. If firms run into recurring problems, they will have to decide whether they should use a different test house. “If I’m a manufacturer and it’s a problem that the test house continues to give me documentation that doesn’t meet my requirements, then yes, I’m going to probably look into switching test houses,” Trautman said. 2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

“It depends on the scope of the issue and how willing that test house is to rectify whatever issues I’m having with them,” she said. “After all, there are other test houses that can do a fair majority of these types of calibrations.” How Often Should Equipment Be Calibrated? There are several factors that help determine how often manufacturing equipment and instruments used for inspection and testing should be calibrated. “When companies buy manufacturing equipment, they come with instruction manuals that talk about the recommended frequency of calibration,” FDA’s Trautman said. “Most manufacturers will start off with that recommendation unless they have historical data that tells them they can lengthen or shorten” the time between calibrations. “Suppose this is the first time that I’m using particular scales for weighing, and the operation manual recommends that the scales be calibrated every nine months,” she said. “As a manufacturer, I’m going to start off my calibration of this particular piece of equipment to have it calibrated every nine months. “Then one of three things can happen,” Trautman continued. “One, that frequency may be perfect. Two, because I use the scale a lot, I may need to shorten the calibration time to six months. Or three, because I don’t use the scale all that often, I probably can lengthen the calibration time to one year. But I have to justify those changes in my calibration procedures.” According to QSR Sec. 820.72(b), calibration procedures must include “specific directions and limits for accuracy and precision.” Further, Sec. 820.72(b)(2) notes that calibration records should include “equipment identification, calibration dates, the individual performing each calibration and the next calibration date.” “The obvious question for FDA investigators is, how is the frequency determined?” Trautman said. “Ninetynine percent of manufacturers will just pull out an operations manual that came along with the equipment and say, ‘This is what was suggested.’ “If it’s not the same timeframe as in the procedures, then the investigator might ask why,” she added. “If there is a documented specification as to why the calibration of a particular piece of equipment can be less frequent or more frequent, then the manufacturer is probably fine as long as that has been documented.” Karen St. Onge, director of corporate quality assurance at NxStage Medical in Lawrence, Mass., cautions firms that at a minimum they should follow the equipment’s manufacturer-recommended calibration intervals. She said companies should consider what the equipment is used for when determining when it should be tested. “You may consider a much shorter calibration interval depending on the level of risk or the intended use of that piece of equipment,” says St. Onge, whose firm manufactures home hemodialysis devices and other products. For example, if a piece of equipment “is being used for acceptance criteria – it’s measuring some criterion that is being used to accept the product – then you should not have a calibration cycle longer than 12 months, and in some cases it could be even shorter,” she said.

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

“You have to assess,” St. Onge said. “If after a year you go in and look at that same piece of equipment and find out that that attribute is out of calibration, then you have to look at a year’s worth of product that you accepted and determine whether or not there is any impact [to devices] due to that piece of equipment being out of calibration. “The bottom line is that it’s not a one-size-fits-all situation,” she continued. “There is a business risk involved. You have to look at all equipment individually when determining the appropriate calibration interval.” St. Onge recommends that device manufacturers determine one month in advance which equipment needs to be calibrated. “You need to run things in advance to make sure that it gets calibrated before its due date. So at the beginning of June we are looking at what is due next month in July,” St. Onge told “The Silver Sheet.” “We are already starting to pull equipment that is due for testing in July. We’re going through and determining what we need to do to get it calibrated.” For equipment that is no longer used, St. Onge suggests that companies conduct and document one last calibration. “If the process changes and you are no longer going to use a specific piece of equipment, part of the closure to put that piece of equipment out to pasture is to do a final calibration,” she said. When that happens, “you have closed the book on that piece of equipment, knowing that the entire time it was in use during your process that it was within its calibration,” St. Onge said. That is “important, because if it were to come back as being out of calibration, you will need to do an assessment of what impact, if any, that had on how you were using that equipment,” she said. According to Trautman, “a red flag is raised when a manufacturer has a calibration system that is set up to calibrate every single thing all the time, once a year, because that’s rather arbitrary. “Maybe all of their instruments really do have a one-year recommended calibration time,” she added. “However, I see too many programs that have everything calibrated once a year, and then you sometimes need to go back and wonder, what did the operation manual for this instrument say? Or, do they even still have the manual?” FDA Investigators Check Calibration Stickers Most calibration citations found on FDA-483 inspection forms come about because manufacturers do not calibrate equipment when they said they would. Firms “will have a calibration schedule set up and there will be a sticker on an instrument, and the investigator will go in and pick up the instrument and they’ll find that it was supposed to be recalibrated last month, but it wasn’t,” FDA’s Trautman said. “The sticker will say, ‘Calibration due on X date,’ and the investigator will walk in and find that it is past that date,” she said. “There is really no excuse for these problems, because this isn’t tough technology. These aren’t things that are hard to handle. “This is just about manufacturers being diligent and following through, and keeping track, and keeping on top of their equipment,” Trautman noted.

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

Even the tiniest pieces of equipment should be labeled with a calibration sticker, NxStage’s St. Onge said. “If you have a set of pin gauges, then you may not be able to necessarily label each pin gauge, but they generally would come in a box, so you would label the box that they’re in with an equipment number,” she said. “Any equipment that requires calibration generally must bear a calibration sticker.” St. Onge urges manufacturers to check periodically that calibration stickers on equipment are kept up to date. “My experience has been that in walking through a manufacturing area or an engineering area where equipment is in use, most auditors or investigators will look at different equipment that is in use and make note of their identification numbers,” which are included on calibration stickers, she said. Identification numbers make it possible for an investigator to more easily find calibration documentation and records related to a particular piece of equipment. “They will always look to see if equipment is marked as ‘calibration due,’ and whether it is within its calibration date,” St. Onge said. “Often they will go back and ask to see some examples of the calibration that was performed, whether it’s in the form of a certificate from an outside test house or documentation that something was calibrated in-house. “They will look at the results and look at the certificate and make sure that the certificate has appropriate information on it and that the standards that were used are noted and are identified as traceable to national or international standards,” as required by QSR Sec. 820.72(b)(1), “Calibration Standards,” she added. NIST is the National Institute of Standards and Technology, an agency of the U.S. Department of Commerce. NIST is the regulatory overseer of calibration test houses that works to ensure that measuring and test standards are reliable. Instruction Manual Storage Instruction manuals must be kept as part of calibration records. “Everyone has a manual for a refrigerator, or a manual from a washer and dryer in some drawer or some file someplace,” FDA’s Trautman said. “Just like all of us at home, manufacturers have that kind of setup for all of their equipment as well.” That is how it is performed at Hitachi Chemical Diagnostics, a company that manufactures tests to measure a patient’s reaction to allergens. Instruction manuals “are not part of our standard operating procedures, because they would make the SOPs too big. So they are kept in a separate file,” says Emi Zychlinsky, VP of research and development for the Mountain View, Calif., firm. “We basically just have a file where we just go by equipment name, serial number and part number,” she said. “This is not a big deal for us because we don’t have that much equipment.” The company has “manufacturing equipment that we use to manufacture our reagents, and then we have manufacturing equipment we use for bottling and mixing reagents,” Zychlinsky told “The Silver Sheet.” The firm also has equipment for general manufacturing purposes, such as centrifuges, cold rooms, pipettes and balances.

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

After initial purchase of the equipment, Hitachi Chemical rarely looks at the instruction manuals. “If we have a problem, we usually call the manufacturer,” Zychlinsky said. NeuroWave handles its instruction manuals in a similar fashion. “Each piece of equipment has an equipment folder, and the equipment folder is going to have all of the calibration certificates, all of the maintenance protocols and maintenance test results, and the user manuals or any instructions that came with it,” Capozzio said. She notes that equipment folders also contain new equipment forms, which are filled out when equipment is purchased. “The purpose for the new equipment form is to identify the equipment and its unique identification number, and to set the calibration and maintenance frequency, as well as the methods we’ll use for the calibration and maintenance – whether it will be sent out to a calibration house or whether it’s something that we would do internally,” Capozzio said. Unique identification numbers are created by NeuroWave as a way to track its equipment. Manufacturers also may mine information from instruction manuals and incorporate that information into calibration documentation or records. “Maybe a particular equipment-maker is prolific, or the manual has six different languages, and the manufacturer doesn’t want to keep an inch-thick manual,” Trautman said. If that is the case, “then manufacturers can extract the important information from that operation manual into some other documentation to say, ‘The original owner’s manual says this,’ so it’s clear where that information came from,” she said. “We have done that at our company,” Zychlinsky said. Most instruction manuals “come in 10 languages now. So if we need to, we extract from the manual’s recommendations and put them into an SOP. But we don’t do that for every piece of equipment. It just depends.” Keep Sharp Eye On Calibration Test Houses – Device Manufacturers Aren’t Their Only Customers Calibration test houses, which are widely used by manufacturers, offer convenience when it comes to testing equipment. However, firms must keep a sharp eye on them just as they would any other supplier. “Most of the time manufacturers send equipment to a calibration test house or they have a calibration test house come in. I’m not saying that a firm has to. They don’t,” FDA’s Trautman said. “There are ways that manufacturers can calibrate their own equipment, but a lot firms, because they have to tie [calibration] to a national or international standard, send their equipment to test houses that already have all of those national and international standards set up,” she said. For example, “let’s consider weights,” Trautman said. “There are weights that tell you that something is exactly 10 grams. So instead of a manufacturer going out and buying a whole set of calibrated weights to calibrate particular measuring equipment, they send it to a test house instead.”

2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

Most test houses calibrate equipment for other manufacturing sectors in addition to the medical device industry. At RS Calibration Services, about 40 device manufacturers are clients, which accounts for about 30 percent of the firm’s business. Other industries served by the company include biomedical, pharmaceutical and micromachining, among others. “We work in what we call ‘the FDA environment,’” RS Calibration VP Sweetnam said. “A very high percentage of our customers are being monitored and controlled by the FDA, so our role is to make sure that all of our customers’ equipment and documentation, as it pertains to calibration, is in compliance with FDA requirements.” Another test house, Accutek Testing Laboratory in Cincinnati, derives approximately 55 percent of its business from device manufacturers. However, it also services the aerospace and defense sectors, company President John McCoy told “The Silver Sheet.” Because they are only one client out of many, it’s important for a device manufacturer to ensure that it receives proper equipment calibrations and documentation that meet FDA regulatory requirements. “Calibration test houses do calibrations for a whole wide range of industry sectors, some regulated, some clearly not,” Trautman said. “We are obviously a regulated sector, and in the Quality System Regulation there are some very specific requirements for calibration and specific requirements for what a calibration record has to have. “There are some very particular requirements that every calibration test house might not put on a certificate that goes back to the manufacturer,” she added. “So this is where, under purchasing and supplier controls, firms need to establish quality requirements and documentation requirements up front with the supplier of this service, since the manufacturer knows the type of information that is required in their calibration records.” Hitachi Chemical’s Zychlinsky said her firm does some in-house calibration, but much of the work is outsourced to test houses, including Rainin, Alpha Omega Instruments and USHIO. All three of those companies not only calibrate instruments, but manufacture equipment as well. Whether or not Hitachi Chemical sends equipment to test houses for calibration “depends on the capabilities we have here,” Zychlinsky said. “If we have enough engineering capabilities to do it here, we might choose to do that, but if not, we go to the outside. We’re a small company, so going outside a lot of times is better.” When it comes to selecting a test house to place on its approved supplier list, Hitachi Chemical “looks for certification, and we look to make sure that they comply with the Quality System Regulation and with the requirements of ISO 13485,” Zychlinsky said. ISO 13485 is the international quality system standard for medical device manufacturing used in Canada, Australia and the European Union, among others. Its requirements for device manufacturers are similar – but not identical – to the QSR. Researching Your Calibration Test House FDA recommends that manufacturers research test houses as part of their supplier control activities to ensure that equipment will be properly calibrated. 2016 NCSL International Workshop & Symposium

Appendix E (S. Schmitt – Silver Sheet Article; June 2010)

“There are criteria and accreditations and so forth for test houses, so firms should ask them these questions: Are you accredited? What standards do you use?” FDA’s Trautman said. “That is part of the assessment of the supplier,” she said. After receiving that information, a device manufacturer “should then be able to say, ‘OK, this test house is certified by ISO or ANSI and is traceable to NIST, so there is some assurance.’” Trautman notes, however, that most manufacturers don’t typically audit test houses in the way they would a component supplier. “But that’s not to say that they couldn’t audit them,” she said. Manufacturers “absolutely can if they want to. However, because inspectional test houses have other governing bodies that are looking at them, most firms rely on national and international accreditations. “Most manufacturers will use accreditations as their assurances,” Trautman continued. “Firms are going to want a test house that is at least tied to NIST.” NeuroWave’s test houses are all accredited through ISO, which gives the company a higher level of confidence that they have the quality management systems in place to ensure that equipment will be properly calibrated. “Because they’re ISO certified it doesn’t require an onsite visit, unless our risk evaluation determines otherwise,” Capozzio said. “But we do send out what we call a ‘request for information,’ which is very similar to a self-assessment. It goes through and asks them a lot of the general questions we would look for during an onsite audit. “However, if we have not done a physical visit to the calibration house, that doesn’t mean we wouldn’t in the future,” she added. “But at this point we haven’t.” NxStage’s St. Onge recommends that firms evaluate a test house in the same manner that they would any supplier that could impact the quality of finished device products. This means that test houses should be included on a company’s approved supplier list. “I have physically audited test houses even though they were accredited and had applicable ISO certifications,” she said. “However, I don’t audit the manufacturer of a piece of equipment,” St. Onge said. “If I’m going to use the original manufacturer [of the equipment] to calibrate it, I’m going to assume that they’re going to calibrate it correctly since they’re the manufacturer.” Hitachi Chemical also is cautious. “We don’t go solely by a test house’s accreditation,” Zychlinsky said. “We evaluate and monitor service suppliers exactly the same way as we do with component suppliers. “We qualify them first, and then we monitor them on a regular basis,” she noted. “And then we audit them again as per our procedures.” *Follow Shawn M. Schmitt @shawnmschmitt

Used with permission. © Informa Business Intelligence, Inc., an Informa company. All rights reserved.

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Medical Device Quality Systems Manual – A Small Entity Compliance Guide Chapter 7 – Equipment and Calibration* First Edition, Dec 1996 Note: Withdrawn by the FDA on 12/12/2013

U.S. Department of Health and Human Services (HHS) Food and Drug Administration (FDA) Centers for Devices and Radiological Health (CDRH) Division of Small Manufacturers Assistance (DSMA) Office of Health and Industry Programs Andrew Lowery, Judy Strojny, Joseph Puleo

HHS Publication 97-4179 (1996) Supersedes: 1991: Medical Device Good Manufacturing Practices Manual. Fifth Edition (HHS Publication FDA 91-4179) U.S. Department of Health and Human Services. Public Health Service. Food and Drug Administration. Center for Devices and Radiological Health. Division of Small Manufacturers Assistance. Office of Training and Assistance. Andrew Lowery, Joseph Puleo. ISBN 0-16-035844-2. August, 1991. NOTE: Refer to Chapter 5: Equipment and Calibration

1987: Device Good Manufacturing Practices Manual. Fourth Edition (HHS Publication FDA 87-4179) U.S. Department of Health and Human Services. Public Health Service. Food and Drug Administration. Center for Devices and Radiological Health. Division of Small Manufacturers Assistance. Office of Training and Assistance. Andrew Lowery, Richard J. Rivera. November, 1987. NOTE: Refer to Chapter 5: Equipment and Calibration

1984: Device Good Manufacturing Practices Manual. Third Edition (HHS Publication FDA 85-4179) U.S. Department of Health and Human Services. Public Health Service. Food and Drug Administration. Center for Devices and Radiological Health. Division of Small Manufacturers Assistance. Office of Training and Assistance. Andrew Lowery, Richard J. Rivera. November, 1984. NOTE: Refer to Chapter 4: Buildings, Equipment and Calibration

Note: No information is readily available on the First and Second editions of the GMP Manual, which presumably preceded the 1984 Third edition. However, the following publications are available:

1982: Device Good Manufacturing Practices – A Workshop Manual U.S. Department of Health and Human Services. Public Health Service. Food and Drug Administration. Bureau of Medical Devices. Office of Small Manufacturers Assistance. Andrew Lowery. Fred Hooten. October, 1982. NOTE: Refer to Section 4: Buildings, Equipment and Calibration

1979: Device Good Manufacturing Practices – A Quality Audit Program for Industry U.S. Department of Health, Education, and Welfare. Public Health Service. Food and Drug Administration. Bureau of Medical Devices. Division of Compliance Programs. September, 1979. NOTE: Refer to pages 88 through 92 for calibration guidance

1978: Device Good Manufacturing Practices – An Industry Self-Inspection Program U.S. Department of Health, Education, and Welfare. Public Health Service. Food and Drug Administration. Bureau of Medical Devices. Division of Compliance Programs. February, 1978. NOTE: Refer to pages 30 through 34 for calibration guidance

Note: FDA GMP/QSR guidance documents, subsequent to FDA 97-4179 (1997), do not include specific chapters on Equipment Calibration. Such documents include:  Guide to Inspections of Quality Systems – Quality System Inspection Techniques (QSIT)  Compliance Program Guidance Manual for Inspection of Medical Device Manufacturers (CP 7382.845)  Compliance Policy Guides (CPG) for devices (Sub Chapter 300) Note: The information contained in this Appendix is a work of the United States Government, Department of Health and Human Service, and is therefore not subject to copyright under the provisions of Title 17 of United States Code, Chapter1, Section 105 of The Copyright Act. Appreciation is extended to those who authored the content of the work as identified above. *Note: Sections of Chapter 7 which precede “Measuring Equipment and Calibration” have been omitted in this Appendix.

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Chapter 7: MEASURING EQUIPMENT CALIBRATION The QS regulation is intended to help assure that devices will be safe, effective, and in compliance with the FD&C Act. To support this goal, each medical device manufacturer should develop and implement a quality system that assures, with a high degree of confidence, that all finished devices meet the company's device master record specifications. These specifications should, in turn, reflect the company quality claims. Section 501(c) of the FD&C Act states a device shall be deemed to be adulterated if its strength differs from, or its purity or quality falls below, that which it purports (claims). Such assurance is obtained by many activities including the measurement of component, device, and process parameters during design and production. These measurements shall be made with appropriate and calibrated equipment as required by 820.72. Each manufacturer should assure that production equipment and quality assurance measurement equipment, including mechanical, electronic, automated, chemical, or other equipment, are: • suitable for the intended use in the design, manufacture, and testing of components, in-process devices and finished devices; • capable of producing valid results; • operated by trained employees; and • properly calibrated versus a suitable standard. To succeed, the quality system shall include a calibration program that is at least as stringent as that required by the QS regulation (820.72). The intent of the GMP calibration requirements is to assure adequate and continuous performance of measurement equipment with respect to accuracy, precision, etc. The calibration program implemented by a company may be as simple or as sophisticated as required for the measurements to be made. Some instruments need only be checked to see that their performance is within specified limits, while others may require extensive calibration to a specification. Manufacturers should determine which measurements are necessary to assure that finished devices meet approved device master record specifications, and assure these measuring instruments are included in a calibration program. Measurement equipment should be identified by label, tag, color code, etc., when located in the same areas as instruments that are not part of the calibration system. Identification can assure that proper equipment is employed to verify and determine compliance to specification of a device component, in-process device, or finished device. Sometimes equipment used only for monitoring a parameter need not be calibrated but should be identified (e.g., for monitoring). A monitoring function might be to indicate if a voltage or other parameter exists, but the exact value is not important. Calibration Requirements The QS regulation requires in section 820.72(b) that equipment be calibrated according to written procedures that include specific directions and limits for accuracy and precision. Figure 5.1 illustrates bias, precision, and accuracy.

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Precision has no unit of measure and only indicates a relative degree of repeatability, i.e., how closely the values within a series of replicate measurements agree with each other. Repeatability is the result of resolution and stability. Bias is a measure of how closely the mean value in a series of replicate measurements approaches the true value. The mean value is that number attained by dividing the sum of the individual values in a series by the total number of individual values. Accuracy is the measure of an instrument's capability to approach a true or absolute value. Accuracy is a function of precision and bias. Because different manufacturers have different accuracy requirements, each manufacturer should decide the level of accuracy required for each measurement and provide equipment to achieve that accuracy.

Figure 5.1 Bias, Precision and Accuracy

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Proper and periodic calibration will assure that the selected equipment continues to have the desired accuracy. GMP calibration requirements are: • routine calibration according to written procedures; • documentation of the calibration of each piece of equipment requiring calibration; • specification of accuracy and precision limits; • training of calibration personnel; • use of standards traceable to the National Institute of Standards and Technology (NIST), other recognizable standards, or when necessary, in-house standards; and • provisions for remedial action to evaluate whether there was any adverse effect on the device’s quality. Remedial action includes recalibration and evaluation of the impact of out-of-tolerance measurements: • on the device design or process validation parameters or data; • on the quality of existing components, in-process, or finished devices; and • appropriate corrective action. Equipment Selection The manufacturer should establish and maintain procedures to ensure that purchased and otherwise received equipment and associated supplies conform to specified requirements (820.50). The purchase of stable and accurate measuring equipment can reduce the frequency of calibration and increase confidence in the company's metrology program. Where economically feasible, equipment with more accuracy than needed for various measurements can be used longer without recalibration than equipment that marginally meets the desired accuracy requirements. Delicate instruments, however, that are "pushing the state-of-the-art" should not be used for routine measurements unless no other approach is feasible. Procedures There are a number of sources of information from which calibration procedures can be developed. Instrumentation manufacturers often include calibration instructions with their instruction manuals. Although these instructions alone are not adequate to meet the QS requirements for a calibration procedure, they usually can be used for the actual calibration process. In some cases, voluntary standards exist such as those by the American Society for Testing and Materials (ASTM), the American National Standards Institute (ANSI), and the Institute of Electrical and Electronic Engineers (IEEE). Information contained in calibration procedures should be adequate to enable qualified personnel to properly perform the calibrations. An example of a calibration procedure for mechanical measuring tools appears at the end of this chapter. A typical equipment calibration procedure includes: • purpose and scope; • frequency of calibration; • equipment and standards required; • limits for accuracy and precision; 2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

• preliminary examinations and operations; • calibration process description; • remedial action for product; and • documentation requirements. Management of Metrology Managers and administrators should understand the scope, significance, and complexity of a metrology program in order to effectively administer it. The selection and training of competent calibration personnel is an important consideration in establishing an effective metrology program. Personnel involved in calibration should ideally possess the following qualities: • technical education and experience in the area of job assignment; • basic knowledge of metrology and calibration concepts; • an understanding of basic principles of measurement disciplines, data processing steps, and acceptance requirements; • knowledge of the overall calibration program; • ability to follow instructions regarding the maintenance and use of measurement equipment and standards; and • mental attitude which results in safe, careful, and exacting execution of his or her duties. Calibration Records Calibration of each piece of equipment shall be documented to include: • equipment identification, • the calibration date, • the calibrator, and • the date the next calibration is due. Many manufacturers use a system where each device has a decal or tag which contains the date of calibration, by whom calibrated, and date the next calibration is due. Examples of such decals are shown on the next page. These decals are examples of the types commonly used to identify the status of measuring instruments and tools. They are available as catalog items or a manufacturer may use its own artwork to purchase decals with specialized wording. Typical calibration decals have a write-on surface. A tough paper or cloth stock and a pressure sensitive adhesive are used for easy application and removal of the decal. “Due” is the blank for the date when recalibration is due.

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Calibration Identification Number or its equivalent is usually the minimum information that may be on equipment. This information allows the manufacturer to read by finding the associated calibration record\card\file. Measuring equipment that is not calibrated or otherwise unsuitable for use should be placed in a quarantine area or labeled with a "calibration void" decal.

A seal or protective cover for exposed, recessed calibration controls on instruments. The calibration control cannot be adjusted without breaking the seal or removing the instrument case.

A decal to be applied to measurement or monitoring instruments not intended for use in determining conformance to device master record specifications with respect to testing, manufacturing, environmental control, etc.

Calibration information is entered onto cards or forms, one for each piece of equipment, or entered into a computerized data system. Most data systems include the calibration date, by whom calibrated, date recalibration is due, the reason for the calibration, comments, address of the manufacturer and calibration laboratory, equipment specifications, serial number, use, etc. An example of a typical card used to record calibration information follows.

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Schedules Measuring instruments should be calibrated at periodic intervals established on the basis of stability, purpose, and degree of usage of the equipment. Intervals between calibrations should be shortened as required to assure prescribed accuracy as evidenced by the results of preceding calibrations. Intervals should be lengthened only when the results of previous calibrations indicate that such action will not adversely affect the accuracy of the system, i.e., the quality of the finished product.

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

A manufacturer should use a suitable method to remind employees that recalibration is due. For small manufacturers, calibration decals on the measuring equipment may be sufficient because recalibration can be tracked by scanning the decals for the recalibration date. For other manufacturers, a computerized system, calibration cycle cards, tickler file, or the like may be used. Calibration cycle cards are maintained in a 12-month (12-section) tickler file. There is one card per item of measuring equipment. The cards in the section of the file for the current month are pulled and all of the equipment listed is calibrated. For example, in a 6-month calibration cycle, when an instrument is calibrated in May, the card is moved from the May section to the November section of the file. When the file is checked in November, the cycle card will be there to remind the manufacturer that calibration is due. The process is repeated until an event such as instrument wearout occurs and the respective cycle card is removed from the file. Cycle cards are used where a manufacturer has many instruments to be calibrated. It would be rather difficult to keep track of the calibration of a large number of instruments by reviewing calibration record cards or scanning the decal on each instrument. It is easier to use a cycle card file. A cycle card file or equivalent also should be used if the calibration records are filed by type of instrument or manufacturer rather than due date. A typical cycle card follows. The "calibration card number" blank refers to the calibration record card for the same item of equipment.

Standards Where practical, the QS regulation requires that standards used to calibrate equipment be traceable to the National Institute of Standards and Technology (NIST), or other recognized national or international standards. Traceability also can be achieved through a contract calibration laboratory which in turn uses NIST services. The meaning of traceability to NIST is not always self-evident. Two general methods commonly used to establish and maintain traceability to NIST are: • NIST calibration of standards or instruments: When this method is used, private standards are physically sent to NIST for calibration and returned. • Standard Reference Materials (SRM's): NIST provides reference materials to be used in a user's calibration program. These SRM's are widely used in the chemical, biological, medical, and environmental fields.

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Information can be obtained from the "Catalog of NIST Standard Reference Materials,” available free from the National Institute of Standards and Technology, Office of Standard Reference Materials, Gaithersburg, MD 20899, phone: (301)975-2016. When in-house standards are used, they should be fully described in the device master record or quality system record. Independent or in-house standards should be given appropriate care and maintenance and should be used according to a written procedure as is required for other calibration activities. FDA recommends that at least two in-house standards be maintained – one for routine use and one for a back-up. Calibration Environment As appropriate, environmental controls should be established and monitored to assure that measuring instruments are calibrated and used in an environment that will not adversely effect [sic] the accuracy required. Consideration should be given to the effects of temperature, humidity, vibration, and cleanliness when purchasing, using, calibrating, and storing instruments. AUDIT OF CALIBRATION SYSTEM The calibration program shall be included in the quality system audits required by the QS regulation. These audits should determine the continuing adequacy of the calibration program and assess compliance with the program. Many manufacturers use contract calibration laboratories to calibrate their measurement and test equipment. If this is the case, FDA views the contract laboratory as an extension of the manufacturer's GMP program or quality system. Normally FDA does not inspect contract laboratory facilities, but it does expect the manufacturer to assess the contract lab to verify that proper procedures are being used. Generally, the manufacturer of the finished device is responsible for assuring the device is manufactured under an acceptable quality system. When a medical device manufacturer uses a contract calibration laboratory, FDA expects the manufacturer to have evidence that the equipment was calibrated according to the GMP requirements. The device manufacturer can do this by: • requiring and receiving certification that the equipment was calibrated under controlled conditions using traceable standards; • maintaining an adequate calibration schedule; • maintaining records of calibration; or • periodically auditing the contractor to assure appropriate and adequate GMP procedures are being followed. For example, the contractor should have: • written calibration procedures; • records of calibration; • trained calibration personnel; and • standards traceable to NIST or other independent reproducible standards. Certification notes and data should include accuracy of equipment when received by the lab to facilitate remedial action by the finished device manufacturer, if necessary. Certification should also include accuracy after calibration, standards used, and environmental conditions under which the equipment was calibrated. The certification should be signed and dated by a responsible employee of the contract lab. 2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

If in-house standards are used by a contractor to calibrate device-related measuring equipment, these standards shall be documented, used, and maintained the same as other standards. INTEGRATING MEASUREMENTS INTO THE QA SYSTEM Proper and controlled calibration can contribute to overall quality by assuring that device design and process parameters are accurately measured and that unacceptable items are not accepted, and acceptable items are not rejected as a result of measurements. If the appropriate product-quality parameters are not checked, however, calibrated equipment will have little impact on assuring quality. A good quality system shall include calibration activities. However, proper calibration will be of little use unless the applications of the measurement equipment are properly developed and qualified during the preproduction development of inspection test methods and procedures. As stated, effectiveness depends on the participation and influence of QA and production management at the preproduction stage. Calibration of equipment cannot correct poor design of products nor can it compensate for poor applications of equipment and techniques. It is the continued use of a complete, integrated quality system, which assures that safe and effective devices are produced. EXHIBITS Examples of calibration cards, decals, and cycle cards were presented above in the text. Examples of a device cleaning procedure and a calibration procedure follow. Manufacturers may use these as presented if they match the manufacturers operations; or may modify them to meet specific requirements. P.C. Board Cleaning [Omitted herein] Calibration Procedures for Mechanical Measuring Tools This is a calibration procedure for mechanical measuring tools. In actual use, the initial accuracy of each tool is checked using the procedure and is recorded. Thereafter, each tool is recalibrated (checked) versus the initial accuracy. Of course, the initial accuracy should meet or exceed the requirements of the measurements to be made with the tool. Precision is checked by making several measurements at various points on the tool's measuring face (surface).

2016 NCSL International Workshop & Symposium

Appendix F (FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)

Sheet 1 of 1

TITLE: Calibration Procedures for Mechanical Measuring Tools No.________Rev.___________ ECN Notes __________________________________________________________________ ______________________________________________________________________________ Drafted by______________________________App.________________Date____________ PURPOSE: This procedure establishes a standard method for the calibration and maintenance of mechanical measuring tools such as micrometers, calipers, etc. SCOPE: All measuring tools used to set specifications or measure conformance to specifications, such as micrometers, calipers, etc., will be included in the calibration program. Each tool will be assigned a number and checked every six months for accuracy. If you suspect a tool is damaged or out of calibration, it should be removed from service and brought to the Quality Control Lab (QC) for checking. To enter a tool in the program, take it to QC where a number will be assigned and initial accuracy checked and recorded. PROCEDURE: 1. Each measuring tool shall be kept clean and maintained in a protective container. As needed, all threads and slides shall be lubricated with a fine tool oil to assure free movement. 2. The calibration shall be done by a comparison to standard gage blocks traceable to the National Institute of Standards and Technology standard with an accuracy 3 to 10 times greater than that of the measuring tool. 3. The comparisons shall be made at different points along the measuring range of the tool. The gage blocks used shall be picked at random to assure that the measuring tool is not checked at the same points on each calibration cycle. When a measurement is made, move the gage blocks from one side of the tool's measuring face to the other on an X/Y axis to assure no wear or taper exists on the measuring faces. 4. Measurement tools not intended for testing or manufacturing do not require calibration in accordance with the QS regulation. These tools should be kept out of manufacturing or labeled to avoid inadvertent use. Otherwise, they should be entered in this calibration program. 5. After calibration, the date of calibration and the next due date of calibration shall be recorded on the Calibration Form No._______. Any adjustments and/or repairs to be recorded. The form is placed in the tickler file according to the next calibration date. 6. If a tool is found to be out of calibration, the QC lab will immediately pass the out-of-calibration information to the appropriate supervisor in the department where the tool is used. The Department and QC management will take appropriate remedial action for affected in-process or finished devices.

2016 NCSL International Workshop & Symposium

View publication stats