Tilemap collider - character passes through wall

I have a character that moves tile-by-tile with the following simple controller:

 public class PlayerController : MonoBehaviour {

     private Vector2 speedX = new Vector2(1, 0);
     private Vector2 speedY = new Vector2(0, 1);
     private Vector2 startPosition;

     private Rigidbody2D rb;

     // Use this for initialization
     void Start ()
     {
         rb = GetComponent<Rigidbody2D>();
     }
 
     // Update is called once per frame
     void Update ()
     {
         startPosition = transform.position;
         if (Input.GetKeyDown("left"))
         {
             rb.position = (startPosition - speedX);
         }
         else if (Input.GetKeyDown("right"))
         {
             rb.position = (startPosition + speedX);
         }
         else if (Input.GetKeyDown("up"))
         {
             rb.position = (startPosition + speedY);
         }
         else if (Input.GetKeyDown("down"))
         {
             rb.position = (startPosition - speedY);
         }
     }
 }

Player has Rigidbody 2D (Dynamic) and Circle Collider (isTrigger is off and same behavior with Box Collider). The wall Tilemap layer has Tilemap Collider 2D, Rigidbody 2D (Static), and Composite Collider 2D (again, isTrigger is off on the colliders).

The problem is that the player bounces off the wall a few times but then slips through the wall eventually. Even worse, sometimes when it passes through, it is offset by a fraction of a tile (usually 0.5) and can then pass freely through everything!

My guess is that the tile offset occurs because the "bounce" off the wall is not restricted to exclusively X or Y movement (no idea how to implement that), but I have no clue why the passing-through occurs. I tried fiddling with the Game's Time setting thinking it was because I was hitting the button too fast, but it makes no difference and the player passes through walls regardless of the speed that I repeat the movement.