rigidBody collision with wall stops vertical movement.

I am using the code below to move a group of GameObjects. The script is on an empty GameObject which also has a RigidBody component. One of the child GameObjects has a BoxCollider component.

The problem

When the gameObject is above ground (in free fall) and collides with a wall, the gameObject gets stuck if the user is moving towards the wall making the object "cling" to the wall at the collision point. I want it to continue falling. How do I stop it from halting on the side of a wall?

The code

 void FixedUpdate()
     {
         velocity = rigidbody.velocity;
         velocity.x = Input.GetAxisRaw("Horizontal") * moveSpeed;
         velocity.z = Input.GetAxisRaw("Vertical") * moveSpeed;
         rigidbody.velocity = velocity;
     }