Fixed Joint Rigidbody Colliding

I just recently created a projectile shooter by getting help from Unity3D forums and documents but I got a new problem that I havnt been able to find a solution to.

I'm shooting a prefab(a arrow but it's just a cylinder at the moment) and I've added a "joint" script to them also meaning if I hit something, they stick to that object and so far it looks good except the arrows stick to each other like, if I shoot more than one arrow at the same spot, it builds up on each other...

I would like to make the arrows pass through each other and not colliding but I have no idea how to do that as I tried to turn of the colliders of the prefab with no luck as the rigidbody wouldnt work if collider was off.

Im just posting a bit of the script here incase anyone wants to correct something

     void ReleaseArrow()
     {
         Rigidbody boltinstance;
         boltinstance = Instantiate(BoltPrefab, barrelEnd.position, barrelEnd.rotation) as Rigidbody;
         boltinstance.AddForce(barrelEnd.forward * Speed);
     }

Joint script

     void OnCollisionEnter(Collision c)
     {
         var joint = gameObject.AddComponent<FixedJoint>(); joint.connectedBody = c.rigidbody;    
 
     }