Contextual regularities help us analyze visual scenes and form judgments on their constituents. The present study investigates the effect of context violation on scene processing using event-related potentials (ERPs). We compared ERPs evoked by congruent vs. incongruent visual scenes (e.g., a man playing a violin vs. a man “playing” a broomstick), when the scene and object are presented simultaneously, so subjects cannot form previous expectations about the object’s identity. As expected, an ongoing anterior negativity emerged around 270 ms post-stimulus presentation, lasting for about 330 ms. This negativity, resembling the N300/N400 effect previously associated with semantic integration, was followed by a later and broadly distributed negativity between 650 and 850 ms, possibly related to late processes of semantic evaluation and response preparation. The results confirm that contextual congruity affects scene processing starting from ∼300 ms or earlier, and that this early electrophysiological congruity effect indeed reflects context violation processing, rather than indexing a mismatch between expected vs. actual events, or between prepared vs. correct responses. They also suggest that contextual information may affect object model selection processes, and influence later stages of semantic knowledge activation and/or decision-making.