First spike latency has been suggested as a source of the information required for fast discrimination tasks. However, the accuracy of such a mechanism has not been analyzed rigorously. Here, we investigate the utility of first spike latency for encoding information about the location of a sound source, based on the responses of inferior colliculus (IC) neurons in the guinea pig to interaural phase differences (IPDs). First spike latencies of many cells in the guinea pig IC show unimodal tuning to stimulus IPD. We investigated the discrimination accuracy of a simple latency code that estimates stimulus IPD from the preferred IPD of the single cell that fired first. Surprisingly, despite being based on only a single spike, the accuracy of the latency code is comparable to that of a conventional rate code computed over the entire response. We show that spontaneous firing limits the capacity of the latency code to accumulate information from large neural populations. This detrimental effect can be overcome by generalizing the latency code to estimate the stimulus IPD from the preferred IPDs of the population of cells that fired the first n spikes. In addition, we show that a good estimate of the neural response time to the stimulus, which can be obtained from the responses of the cells whose response latency is invariant to stimulus identity, limits the detrimental effect of spontaneous firing. Thus, a latency code may provide great improvement in response speed at a small cost to the accuracy of the decision.