1-22INTRODUCTION (1000) A more complete exposition3 of the above quality-control checks has been published. 7. References 1. ROSSUM, J.R. 1975. Checking the accuracy of water analyses through the use of conductivity. J. Amer. Water Works Assoc. 67:204. 2. FRIEDMAN, L.C. & D.E. ERDMANN. 1982. Quality Assurance Practices for Analyses of Water and Fluvial Sediments. Tech. Water Resources Inc., Book 5, Chapter A6. U.S. Government Printing Off., Washington, D.C. 3. OPPENHEIMER, J. & A.D. EATON. 1986. Quality control and mineral analysis. In Proc. Water Quality Technology Conference (Houston, Texas, December 8-11, 1985). American Water Works Assoc., Denver, Colo. 1040 METHOD DEVELOPMENT AND EVALUATION 1040 A. Introduction Although standard methods are available from many nationally recognized sources, there may be occasions when they cannot be used or when no standard method exists for a particular constituent or characteristic. Therefore, method development may be required. Method development is the set of experimental procedures devised for measuring a known amount of a constituent in various matrices, in the case of chemical analyses; or a known characteristic (e.g., biological or toxicological) of various matrices. 1040 B. Method Validation Whether an entirely new method is developed by accepted research procedures or an existing method is modified to meet special requirements, validation by a three-step process is required: determination of single-operator precision and bias, analysis of independently prepared unknown samples, and determination of method ruggedness. TABLE 1040:I. PRECISION AND BIAS FOR A SINGLE CONCENTRATION IN A SINGLE MATRIX Result mg/L 1.23 1.21 1.30 1.59 1.57 1.21 1.53 1.25 Sum Difference (Ϫ1.30) Ϫ0.07 Ϫ0.09 0.0 0.29 0.27 Ϫ0.09 0.23 Ϫ0.05 0.49 Squared Difference 0.0049 0.0081 0.0 0.0841 0.0729 0.0081 0.0529 0.0025 0.2335 1. Single-Operator Characteristics This part of the validation procedure requires determining the method detection level (MDL) as in Section 1030; the bias of the method, i.e., the systematic error of the method; and the precision obtainable by a single operator, i.e., the random error introduced in using the method. To make these determinations, analyze at least 7 but preferably 10 or more portions of a standard at each of several concentrations in each matrix that may be used. Use one concentration at, or slightly above, the MDL and one relatively high so that the range of concentrations for which the method is applicable can be specified. The use of several concentrations to determine bias and precision will reveal the form of the relationship between these method characteristics and the concentration of the substance, the characteristic toxicity of the substance, or the biological factor of interest. This relationship may be constant, linear, or curvilinear and is a significant characteristic of the method that should be explained clearly. Table 1040:I shows calculation of precision and bias for a single concentration in a single matrix from eight replicate analyses of a standard with a known concentration of 1.30 mg/L. The bias is 0.49/8 ϭ 0.06 mg/L and the precision is the square root of 0.2335/(8Ϫ1) ϭ ͌0.03336, or 0.18 mg/L (note that this is similar to the calculation for standard deviation). 2. Analysis of Unknown Samples This step in the method validation procedure requires analysis of independently prepared standards where the value is unknown to the analyst. Analyze each unknown in replicate by following the standard operating procedure for the method. The mean amount recovered should be within three standard deviations (s) of the mean value of the standard but preferably within 2 s. Obtain the unknowns from other personnel in the analyst’s laboratory using either purchased analytical-grade reagents or Pa. 5. it may be prudent to test the method for equivalency to standard methods.R. Calculate all seven pairs to get seven differences.S. Washington. Official Analytical Chemists. the result will be t.1M 95°C 10 min no 6. is the final validation step.METHOD DEVELOPMENT AND EVALUATION (1040)/Method Validation 1-23 standards available from National Institute of Standards and Technology (NIST). reference. NATRELLA. The standard deviation is a realistic estimate of the precision of the method. Washington. It is especially important to determine this characteristic of a method if it is to be proposed as a standard or reference method. J.3 3.C. ASTM STP 891. ENVIRONMENTAL PROTECTION AGENCY. If the range of concentration is very broad.H. & E.. W. denote the nominal factors by capital letters A through G and the variations by the corresponding lower-case letters. To illustrate. National Bureau of Standards Handbook 91. apply the following statistical steps:2 1. 4. 1975. If combination 2 is analyzed. 4. unless none exist.0 Variation 12 min 10 g 1.C. & E. A properly conducted ruggedness test will point out those procedural steps in which rigor is critical and those in which some leeway is permissible. Hansen.G. STEINER. Equivalency Testing Source: YOUDEN. 1994. Then set up a table of the factors as in Table 1040:III. D. For example. 1983. Select an appropriate sample size based on an estimate of the standard deviation. Philadelphia. Guidelines for Establishing Method Equivalency to Standard Methods. Assoc. Rep. Statistical Manual of AOAC. M. L. 1963. and equivalent methods for water analysis is available. American Soc. WILLIAMS.5 . 3. 8th Symp. YOUDEN. Nev. test more concentrations. Las Vegas. and so on until all eight combinations have been analyzed. To make the determination. Guidelines establishing test procedures for the analysis of pollutants under the Clean Water Act. If available for the particular constituent. Once an initial set of analyses (five or more) has been made at each chosen concentration.S. Testing & Materials.C. This requires analysis of TABLE 1040:II. suppose the effect of changing the factors in Table 1040:II is to be determined. FACTOR MATRIX FOR METHOD RUGGEDNESS DETERMINATION Combinations Factor value 1 A B C D E F G s 2 A B c D e f g t 3 A b C d E f g u 4 A b c d e F G v 5 a B C d e F g w 6 a B c d E f G x 7 a b C D e f G y 8 a b c D E F g z A test of the ruggedness. 2.C. find the four results where the factor was nominal (all caps) and the four where it was varied (all lower case) and compare the averages of the two groups. D. 1985. If there is no outstanding difference. the result will be s. Test the distribution of data for normality and transform the data if necessary (Section 1010B). Bahner & D. which can then be ranked to reveal those with a significant effect on the results. Experimental Statistics. An explanation of each of these steps with additional techniques and examples has been published. A listing of standard. To determine the effect of varying a factor. stability of the result produced when steps in the method are varied. Final rule. 1975. Federal Register 59:20: 4504. Statistical Manual of AOAC.J. VARIATIONS IN FACTORS FOR METHOD RUGGEDNESS DETERMINATION Factor Mixing time Portion size Acid concentration Heat to Hold heat for Stirring pH adjust Nominal 10 min 5g 1M 100°C 5 min yes 6. STEINER.. W. not interactions. Assoc.e. use results (s ϩ u ϩ w ϩ y)/4 and (t ϩ v ϩ x ϩ z)/4.4 Because the number of analyses can be very large. a minimum of three concentrations by the alternate and by the standard method. The Association of Official Analytical Chemists1 has suggested a method for this test in which eight separate analyses can be used to determine the effect of varying seven different steps in an analytical procedure. performance evaluation samples from EPA-Cincinnati are particularly useful. U. H. Official Analytical Chemists. Method Ruggedness A or a B or b C or c D or d E or e F or f G or g Result TABLE 1040:III. to compare the effect of changing C to c. 600/X83-037. U.J. the calculations become complex and familiarity with basic statistics is necessary. ENVIRONMENTAL PROTECTION AGENCY. 40 CFR Part 136.5 5. If combination 1 is analyzed. References 1. i. This design tests main effects. Washington. eds. 3. Test the variances of the two methods using the F-ratio statistic. R. Harmonization of Biological Testing Methodology: A Performance Based Approach in Aquatic Toxicology and Hazard Assessment. After a new method has been validated by the procedures listed above. Environmental Monitoring Systems Lab. 4. Test the average values of the two methods using a Student-t statistic. 2. D. calculate the average and standard deviation of the eight results s through z. 7 ϭ 0. SAMPLE COLLABORATIVE TEST RESULTS Deviation Laboratory 1 Result mg/L 32.3Ϯ0.5 mg/L) is 4.7 33.8 From Known 2.0Ϯ0. and b is the relative standard deviation TABLE 1040:IV.. where y is the relative standard deviation. As noted in Table 1040:IV. use all 15 results to calculate a grand average and standard deviation.8 0 Ϫ0. Therefore.7 mg/L) with instructions to analyze in triplicate using the procedure provided. x is the concentration.3. The difference between the average of each laboratory and the grand average reveals any significant bias.5%. 1. Number of Replicates Send each of five laboratories four concentrations of a compound (4. e.4 ⌺ ϭ Ϫ0. 3.6 30. use appropriate grades of reagent water as long as this is stipulated in the resulting statement of method characteristics. and 32. involve at least six analysts with not more than two from each laboratory. 11. Calculate the average and standard deviation for each laboratory.1 ⌺ ϭ 1. For all four unknowns in this test. analyze at least two replicates of each concentration per laboratory.6 32. If it is not constant.g.8 32. Collaborative Testing Once a new or modified method has been developed and validated it is appropriate to determine whether the method should be made a standard method.4. and concentration range.6 33. If this is not feasible.3. conduct the test in each medium for which the method was developed. so the average deviation (bias) was 1.2 36.9 (⌺x)/n ϭ 33 s ϭ 1. Illustrative Collaborative Test Test at least the following variables: Laboratory—Involve at least three different laboratories.2 3 where: r ϭ number of replicates and P ϭ the product of several variables. In planning for a collaborative test.5 Experimental xϮs 34.7 or r ϭ 3.7 35.3. The difference between the grand average and the known value is the method bias. if three levels of a substance are to be analyzed by single operators in six laboratories on a single apparatus.26. different laboratories use the standard operating procedure to analyze a select number of samples to determine the method’s bias and precision as would occur in normal practice. These may include the laboratory.1-24 INTRODUCTION (1000) 1040 C. Tabulate results as shown in Table 1040:IV below (the results for only one concentration are shown). and the number of replicates required.4 32.3 Ϫ1.0 Ϫ 32. operator.3 32.1 In this test.7 Calculate the number of replicates after the number of variables to be tested has been determined by using the formula: r Ͼ 1 ϩ (30/P ) 2 0. use more levels spread uniformly over the operating range.8 33. Because there are no obviously aberrant values (use the method in Section 1010B to reject outliers). then P is calculated as follows: P ϭ 3 ϫ 1 ϫ 6 ϫ 1 ϭ 18 5 .7Ϯ1. although more are desirable to provide a better estimate of the standard deviation.6Ϯ0. which is the method precision. apparatus.4 33. The relative standard deviation of the grand average (1.3/5 ϭ 0.6 31.2Ϯ1.0 From Grand Average 1.9%. Operators—To determine overall precision.6.9 32. 2. 33. the number of variables to be tested. test three levels covering the range of the method. the percentage results indicated increasing bias and decreasing precision as the concentration decreased. m is the slope of the line.3 mg/L or 0. and the s for each laboratory is the single-operator precision. As an example.3 Ϫ0. use all the data. The procedure to convert a method to standard status is the collaborative test. which itself is the result of many sources of variation. such as that shown for Laboratories 1 and 3.0 33. 23.5 0.5 33. the number of levels to be tested. Variables and the number of replicates is r Ͼ 1 ϩ (30/18) Ͼ 2. consider the following factors: a precisely written standard operating procedure. the sum of the deviations from the known value for the laboratories was 1.6 32. to describe the method in a formal statement. rounded to 0. the variables that affect it must be tested. If matrix effects are suspected.4 32. 4 The minimum number of replicates is two. Because method precision is estimated by the standard deviation.3 0.6 30.6 Ϫ1. Levels—If the method development has indicated that the relative standard deviation is constant. Apparatus—Because model and manufacturer differences can be sources of error. the precision would be given by a straight line with the formula y ϭ mx ϩ b. which is the same as the difference between the grand average and the known value. 04 17. Assoc.4 4.J.043 0.604 68.082 0.5 at concentration ϭ 0.5 10. or the identical and less ambiguous term milligram-equivalents per liter.81 79.47 13. The unit equivalents per million (epm).03 48.053 0.052 0.6 23.055 0.024 0.9 0.072 0. make a correction if the results are expressed as parts per million (ppm) or percent by weight: ppm by weight ϭ mg/L sp gr % by weight ϭ mg/L 10 000 ϫ sp gr In such cases.111 0. Reference 1.016 0. See Section 7020D for expression of radioactivity results.07 81. Record only the significant figures. W.049 0.020 0.3 11.277 0.81 32.009 0.010 0.EXPRESSION OF RESULTS (1050)/Units 1-25 AND TABLE 1040:V.026 0.99 33. 4.62 1. Statistical Manual of the AOAC.994 3.016 0.15 27.144 0. Express concentrations greater than 10 000 mg/L in percent. state specific gravity.8 12.6 43.992 0. If concentrations generally are less than 1 mg/L.6 1.92 18.062 0. & E.941 12. 1050 EXPRESSION OF RESULTS 1050 A.70 Ion (Anion) BO2 BrϪ ClϪ CO32Ϫ CrO42Ϫ FϪ HCO3Ϫ HPO42Ϫ H2PO4Ϫ HSϪ HSO3Ϫ HSO4Ϫ IϪ NO2Ϫ NO3Ϫ OHϪ PO43Ϫ S2Ϫ SiO32Ϫ SO32Ϫ SO42Ϫ Ϫ me/L ϭ mg/Lϫ 0.07 126.01 62.10 6. or milliequivalents per liter (me/L).025 0.058 0.007 0. However.022 0.H. Units This text uses the International System of Units (SI) and chemical and physical results are expressed in milligrams per liter (mg/L). Washington.03 38. Official Analytical Chemists. Cations and anions are listed separately in alphabetical order.012 0. The values found from the collaborative test are shown in Table 1040:V.01 31.9 46. These results indicate that the method is acceptable. D.017 0. concentrations of less than about 10 mg/L require greater care in analysis.45 30.33 31.008 39. In solid samples and liquid wastes of high specific gravity.036 0.023 0.77 27.057 0.73 22.031 0. YOUDEN.04 40.00 17.66 20.035 0.031 0.1 mg/L.030 2 5 56 90 70 47 81 72 1 58 1 29 40 81 50 44 653 83 59 mg/L ϭ me/Lϫ 8.020 36 52 21 33 24 64 39 84 31 24 33 30 880 74 13 80 59 37 29 98 82 mg/L ϭ me/Lϫ 42.00 61.021 0. CONVERSION FACTORS* (Milligrams per Liter—Milliequivalents per Liter) Ion (Cation) Al B3ϩ Ba2ϩ Ca2ϩ Cr3ϩ Cu2ϩ Fe2ϩ Fe3ϩ Hϩ Kϩ Liϩ Mg2ϩ Mn2ϩ Mn4ϩ Naϩ NH4ϩ Pb2ϩ Sr2ϩ Zn2ϩ 3ϩ me/L ϭ mg/Lϫ 0.010 0.03 * Factors are based on ion charge and not on redox reactions that may be possible for certain of these ions.012 0. .8 33 BIAS Bias % 11. it may be more convenient to express results in micrograms per liter (g/L).90 35.C.7 Amount Found mg/L 4.04 103. Use g/L when concentrations are less than 0.5 5. if the result is given as milligrams per liter.07 97.4 32.014 0. can be valuable for making water TABLE 1050:I. 1% being equal to 10 000 mg/L when the specific gravity is 1.00 19.66 16.2 23.033 0.028 0.02 47.99 96. 1975.2 5.9 CV (% Standard Deviation) 12.00.99 18. STEINER.00 58. METHOD PRECISION Known Amount mg/L 4.030 0.