Why is there camera sway when using a 360 controller?

When I hook up a 360 controller to Unity, the buttons and analog sticks all work fine except for the fact that there is a little (sometimes a lot) of camera sway immediately upon playing, with no buttons having been pressed. I've tinkered with every sensitivity setting I could find, to no avail.