Bias in ATF Statistical Practice

Philip F. Lee, 4/19/1999 (rev. 11/11/03)

The ATF's Youth Crime Gun Interdiction Initiative (YCGII) program "Crime Gun Trace Analysis Report" (CGTAR) for 1998 contains statistically biased calculations which make firearm trafficking appear to be a more prevalent than it is. When determining what cases to investigate, the ATF takes as an indicator that a gun was trafficked (i.e., a straw purchase was made) if the time between the first retail sale and the time the gun was "recovered" by authorities is three years or less. Such guns are called short time-to-crime guns. But short time-to-crime guns may not be trafficked and using the number of short time-to-crime guns to estimate levels of trafficking can over-estimate real levels of trafficking. Since the ATF investigates trafficking, it is strange that they let people gain this false impression which they could correct by reporting results of their investigations. Since they do not make those reports, we estimate the amount of this bias and explain how the ATF methodology produces it.

This 1998 ATF report limits traces to guns bought in retail since 1990 and examines gun traces submitted during a one year period beginning in August. The CGTAR report contains a lengthy discussion about statistics of the time-to-crime data for traced guns, but makes no mention of bias. The ATF is careful to note "Newly purchased firearms can be stolen, for example; such a crime gun, if traced soon after the theft, would show a short time-to-crime" (see page 8 of the CGTAR). So they know not all such guns are trafficked and their data are biased. However, the naive media seems not so aware and the ATF reports such data with no effort to determine what proportion of these guns were stolen rather than trafficked. We estimate how much bias could be in the data.

By ignoring guns made prior to 1990, the ATF produces a larger fraction for short time-to-crime guns than is in the population of traced firearms. To illustrate, if the ATF were to limit tracing of guns to those sold since 1994 instead of 1990, virtually 100% of the guns traced and reported in the 1998 CGTAR would show a time-to-crime of under three years. The media would scream that "trafficking must be stopped because it comprised 100% of crime guns", but that would not be true. We know guns are stolen for use in crime. This example illustrates that bias can be produced by ignoring older guns with the extreme cut-off date of 1994, now we estimate potential biasing for the ATF data sets with their 1990 and other limits.

One estimate for biasing is to use the hypothetical case that no guns are trafficked and all are stolen. To simplify calculations we assume guns are sold at a constant rate since 1/1/1990, stolen at random, trace requests are made uniformly throughout the reporting period (1 Aug. 1997 to 31 July 1998 for 1998 CGTAR), and only stolen guns are recovered in crime for tracing by the ATF. Now these assumptions give a simple model that allows a calculation of what proportion of guns recovered would show a time-to-crime of less than three years. There is no suggestion that this model reflects reality (obviously some trafficking does happen). The intent is to illustrate the possible biases any data analysis with this measure and with the limit to tracing guns made after 1/1/1990.

The percentages of guns showing time-to-crime less than three years are calculated for the 1998 and the 1997 report periods. Since, in our hypothetical case, no guns are trafficked, this short time-to-crime percentage does represent bias only. Similar bias is calculated for tracing in other (earlier) years to illustrate how bias would change in time. The table below shows the percent of short-time-to-crime guns assuming guns are “recovered” uniformly during the corresponding August to July reporting period for the trace year.

Trace Year	Percent of Guns Showing Short-time-to-Crime
1993	98.0
1994	73.9
1995	59.3
1996	49.4
1997	42.3
1998	37.2
1999	33.1

The actual CGTAR for the 1997 year only covered 10 months and not the full year so the actual bias would be slightly different from our model calculation. These calculations show that short time-to-crime guns making up even 30% or 40% of the trace population does not mean large scale trafficking is happening. The potential bias is so high that use of that data in a statistical sense to indicate trafficking is misleading.

The ATF's 1998 CGTAR states that new funds will permit tracing back to 1985 for the 1999 report. By similar calculations, the 1999 bias percentage would change from 33.1 (for the 1990 cut-off) to 21.3 (for a 1985 cut-off). By switching cut-off date used for 1998 traces (bias of 37.2%) to 1985 for the 1999 traces (bias of 21.3%), the naive could be fooled into believing great progress is being made against gun trafficking by the apparent reduction of short time-to-crime guns as a proportion of the whole.

It is fair to ask why the ATF does not report results of investigations into these short time-to-crime guns. Investigations as to actual trafficking would provide better evidence than biased statistics.

Philip F. Lee has a PhD in Mathematics.