Weird behaviour with hinge joint

I am trying to create a kind of platformer game where you play a little bug flying around in a tree.

My player object is currently a simple box with a box collider. It is a rigidbody which is controlled using AddForce (see below).

Naturally this tree has branches at multiple locations which are supposed to react to the player (or other physics objects) touching it. These branches are assembled from smaller sprites, each of which has its own MeshCollider. A branch has one rigidbody component at the top level together with a springy hinge joint to make it react to other physics objects.

This works so far, except in some special cases. When two (or more) weights are touching my branch and my player object touches a collider close to the branch (not the branch or the weights themselves!!), it behaves as if the weights somehow lose their mass.

I've recorded a video to illustrate the problem: http://youtu.be/AXbuSEA3XQc

You can see the branch and the two spherical weights on top and my player controlled object at the bottom. As you can see the player never touches the branch. The player touches some other objects in between, all of which are non-physical and marked as static. This does not occur when only one (or no) weight is used. Everytime I touch the big static branch below my physical branch, the physical branch begins to return to its inital position, presumably because of the hinge spring trying to return to its initial position. As soon as I move my player object away the branch starts behaving normal again.

I've tried to reduce this to simple boxes without anything else but still get the same behaviour. The only time force is applied anywhere in the project is in my player control script, which more or less applies user input as a force directly to the player object.

 void FixedUpdate () {
     Vector3 force = new Vector2(Input.GetAxis("Horizontal"), Input.GetAxis("Vertical"));
     rigidbody.AddForce (force * Time.fixedDeltaTime * forceFactor);
 }

At no point is there any force applied to any other objects from code.

I tried playing around with all the physics settings but still got the same result.

I also tried replacing my hinge joints with equivalent configurable joints but they still behaved the same.

I am out of ideas, any help would be appreciated!