Unity ignoring joysticks of specified number

I am building a 2-4 person multiplayer game with Unity.

I hooked up a controller, and used both the input manager (Input.GetButton("xbox A")) and direct calls to the Input KeyCodes (Input.GetKey(KeyCode.JoystickButton0)).

In either case, as long as I don't specify a joystick number, the game responds to the player input correctly. The result of course, is that all players on screen will accept all input form the controllers, which is not what anyone wants.

So I tried using requesting joystick input from specific joysticks (i.e. "joystick 1 button 0" in the input manager, and KeyCode.Joystick1Button0). But now the game just doesn't receive the input. The same code that used to make a character jump and print to console now doesn't because its not getting joystick 1 button 0 input.

But of course thats what I'm giving it. As soon as I stop specifying the controller number it works fine.

Is there something I'm missing here? This behavior is happening with both a PS3 controller and Xbox360 controller on my Mac.