The ability to detect unexpected or deviant events in natural scenes is critical for survival. In the auditory system, neurons from the midbrain to cortex adapt quickly to repeated stimuli but this adaptation does not fully generalize to other rare stimuli, a phenomenon called stimulus-specific adaptation (SSA). Most studies of SSA were conducted with pure tones of different frequencies, and it is by now well-established that SSA to tone frequency is strong and robust in auditory cortex. Here we tested SSA in the auditory cortex to the ear of stimulation using broadband noise. We show that cortical neurons adapt specifically to the ear of stimulation, and that the contrast between the responses to stimulation of the same ear when rare and when common depends on the binaural interaction class of the neurons.