In spite of a large number of studies at different levels of the auditory system, there is still no satisfactory physiological account for the perception of speech. We argue here that many of the problems in creating such an account are common to other processing tasks of the auditory system, such as the extraction of pitch and sound localization. These difficulties are related to the global aspects of these percepts, which depend on integrating information over large bandwidth and on temporal context. We describe a possible framework for a solution – the Reverse Hierarchy Theory, recently formulated to account for difficulties of similar character in the relationships between visual psychophysics and the physiology of the visual system. We show that the concepts of Reverse Hierarchy Theory can be easily mapped into the auditory system, and that it generates natural explanations to some perplexing features of auditory perception of global structures such as pitch, space and speech. We argue that in the context of Reverse Hierarchy Theory, primary auditory cortex seems to occupy a pivotal role in that it generates auditory objects over which higher-order processing tasks can be performed.