The human primary motor cortex (M1) is robustly activated during visually guided hand movements. M1 multivoxel patterns of functional MRI activation are more correlated during repeated hand movements to the same targets than to greatly differing ones, and therefore potentially contain information about movement direction. It is unclear, however, whether direction specificity is due to the motor command, as implicitly assumed, or to the visual aspects of the task, such as the target location and the direction of the cursor’s trajectory. To disambiguate the visual and motor components, different visual-to-motor transformations were applied during an fMRI scan, in which participants made visually guided hand movements in various directions. The first run was the “baseline” (i.e., visual and motor mappings were matched); in the second run (“rotation”), the cursor movement was rotated by 45° with respect to the joystick movement. As expected, positive correlations were seen between the M1 multivoxel patterns evoked by the baseline run and by the rotation run, when the two movements were matched in their movement direction but the visual aspects differed. Importantly, similar correlations were observed when the visual elements were matched but the direction of hand movement differed. This indicates that M1 is sensitive to both motor and visual components of the task. However, repeated observation of the cursor movement without concurrent joystick control did not elicit significant activation in M1 or any correlated patterns of activation. Thus, visual aspects of movement are encoded in M1 only when they are coupled with motor consequences.