From the IG work papers in 1998, resulting from Eric Larson's August 13, 1999, FOIA request, with the IG's redactions noted. The IOG withheld 1,088 pages of draft reports. ------------------------------------------------------------------------- Office of Inspector General Review of ATF's Branch Registration Procedures A-CH-98-001 Record of Discussion Participants: [deleted], Chief, NFA Branch, Headquarters, Washington, D.C. [deleted} [deleted], ATF, Specialist [deleted], Chief, Explosives and Firearms Division [deleted] Specialist [deleted] OST, ISD, ATF [deleted] OST, ISD, Consultant [deleted] OST, ISD, ATF Treasury OIG, Chicago - Gary Wilk [deleted] Ms. B. Rickey, Mr. B. Bronstrup Date: June 17, 1998 Time: 9:00 AM EST Location: 740 15th Street, OIG Headquarters, Washington, DC Subject: Draft report: Congress; review and discussion of database error analysis [deleted] asked for an explanation of of the analysis results obtained by the OIG audit of the physical and electronic records maintained by ATF and known asa the NFRTR. [deleted] further, added that [deleted] reason for asking was that the results obtained by the OIG audit were disappointing at best and could have serious consequences for the ATF firearms registry mission. This auditor offered that perhaps ATF would prefer to identify a term other than "critical" as the identifier for the errors identified by this audit report. [deleted] stated that the term "critical" had been misapplied in the data analysis. [deleted] believed that when the audit team visited [deleted] to define what elements actually constituted "critical" [deleted} understanding appears to have been different than the auditors. This auditor explained that our definition had come from our understanding of the discussion. However, if ATF could at this time better define what ATF considered "critical" it would be very helpful. [agreed] that the name of the weapon owner and the weapon serial number were "critical." [deleted] did not agree that the address, date the document was received, the date of birth of the applicant, and weapon description were "critical." Prepared by: G. Wilk Date: 6/30/98 Reviewed by: RKB Date: 8-17-98 Page 1 of 3 F37 ------------------------------------------------------------------------- page 2 Office of Inspector General Review of ATF's Branch Registration Procedures A-CH-98-001 This auditor explained that the address was "critical" since that was the basis for identifying the location of the weapon, the date received was important due to the 4467 registration amnesty issue, the date of birth to detect underage registrations (one example was detected), and the weapon description due to the H&R Handy Gun issue as well as the fact that the database contained a "field" that required an input. This auditor offered that a consideration could be made that would draw a distinction between "mission critical" and data fields that were of importance the audit [sic] and therefore very important but perhaps not as "critical" to the mission of ATF." [deleted] accepted that if the auditor could draw that specific distinction it would be more consistent with [deleted] perception of the database and the needs of ATF as a whole. A second area that was of concern to the audit and the auditee was the reconciliation of the database when the data was transitioned from the legacy system to the client server system. [deleted] offered registry "print out" information that illustrated the discrepancy between the two systems of 407 records. This audit has been provided this information at other times during the audit never with a satisfactory explanation and therefore discounted. [deleted] went on to explain that ATF had developed an explanation that "rationalized" the errors as occurring during the period of time immediately after the data transition. [deleted] further explained that they identified the cause, operator error, and subsequent error correction attempts that created the variance in record count. This auditor explained that a specific explanation for each error type would required [sic] as foot note disclosure. In addition this auditor stated that if the explanation were provided that ATF would have to provide a certification regarding the transition date and sources of the errors. It was also suggested that ATF institutionalize the method use to produce the reconciliation data sets and archive those data sests such that ATF would be able to reproduce the results in the future should that prove to be necessary. This auditor made other editorial comments and the nature and extent of the documentation that ATF might consider relevant to this area. In addition, this auditor pointed out that not only should the total number of records be reconciled but each year and registration type be reconciled. [deleted] stated that [deleted] intended to reconcile each number in the database. [deleted] further stated that [deleted] also intended to eliminate the need for the "other" category or greatly reduce the number of entries made by improved awareness of more correct data entry. Prepared by: G. Wilk Date: / /98 Reviewed by: Date: Page 2 of 3 ------------------------------------------------------------------------- page 3 Office of Inspector General Review of ATF's Branch Registration Procedures A-CH-98-001 This auditor produced an electronic copy (diskette) of the detailed analysis work performed by this auditor. The purpose of the copy was to enable ATF to better analyze the results obtained by the audit team during the examination of the database. Conclusion The NFA Branch representatives appeared to obtain an improved appreciation of the specific requirements that produced the outcomes of the audit. In addition it appears that ATF is more closely approaching a more defensible database reconciliation at this time as well as obtaining a permanent reference for their use in the future by the establishment of the archived bseline dataset. Prepared by: G. Wilk Date: / /98 Reviewed by: Date: Page 3 of 3 ------------------------------------------------------------------------- Office of Inspector General Review of ATF's Branch Registration Procedures A-CH-98-001 Summary of Sample Selection and Sampling Error Calculations Purpose: To determine if the NFA database contains a significant amount of errors as alleged. Scope: Limited to Forms 4467, Letter and Other for the period from 1934 through 1997 Source: 1. NFA electronic (computer) database random sample listing extracted from ATF Headquarters, NFA Branch, in Washington, D.C. 2. Original documentation sampled from ATF Tracing Center in Falling Waters, WV. Results: Of the 528 records and documents reviewed: W/P's Vols. H through K -- We discovered a total of 395 errors or omissions of which 176 were Critical to the NFA mission and the remaining 219 were Administrative. -- A random sampling methodology was implemented for the computer to original documentation analysis. Random "alphabet" and random date methodologies were implemented for the original documentation to computer records analysis. The "alphabet" method was required because the original documentation had been stored in either alphabetical or chronological sequence depending on when the documentation was processed, pre or post 1985. -- We were unable to adequately identify 14,301 Unknown records contained within the category "Other." These records have subsequently been tentatively identified as 9,621 miscoded Form 6 and 4,680 unknown (database conversion errors). -- The total number of records contained within the database at the cutoff for our audit had not been accurately determined. Conversion of the computer database from an existing system to a new software system haad been accomplished in October of 1997. However, a record count reconciliation was not performed and extraneous records were introduced and not detected nor eliminated. ATF Software mangement Branch has been working to reconcile and rectify coding errors in the NFA database. Prepared by: G. Wilk Date: 4/23/98 Reviewed by: RKB Date: 5-7-98 Page 1 of 2 H-0 ------------------------------------------------------------------------- page 2 Office of Inspector General Review of ATF's Branch Registration Procedures A-CH-98-001 W/P's Vols. H through K -- Form 4467 was a critical indicator for our audit. We determined, based on our discovery sample, that the combined error rate for original documentation and the computer database was 18.4 percent. -- We were able to determine that the error rate was in excess, with 95 percent confidence, +/- 7 percent, of the NFA Branch specified error rate limit of (+/-) 5 percent. Based on our Discovery error estimate we did not implement the full statistical sampling plan. Conclusion: The NFA database -- National firearms Registration and Firearms Record [sic] (NFRTR) does not contain less than the 5 percent error rate limit for Critical data established by the Chief, Fireamrs [sic] and Explosives Division, ATF. Note to File: For Purpose, Scope, Source, Conclusion on W/P's H-1 through H-143, please refer to this workpaper (H-0). Prepared by: G. Wilk Date: 4/23/98 Reviewed by: RKB Date: 5-7-98 Page 2 of 2 -------------------------------------------------------------------------