Collider/object size effecting the behavior of a spring joint

I have a FPS controller that swings from objects by attaching a spring anchor to objects when i click on them.

I found that when experimenting with the spring joint, with smaller objects i was getting strange spring behavior (object was jittering and bouncing around rather than swinging like a pendulum.) When i increased the size of the object, i got more motion like what i was expecting...

The problem is, i thought that the physics engine was tuned so that 1 unit = 1 meter. I don't want to use abnormally large objects just to get this mechanic to work. Why is it behaving this way? is there a way i can set it up so i can use the smaller objects, but still get the behavior i want?