Sprite Becoming Blurry

Hello, I know this gets asked quite a bit, but I still have not been able to find a solution to this problem.

I have a sprite (not a power of 2). Using the 4.6 Image, I add the sprite to the image and it looks great:

In my script, I am messing with these images to create a pseudo-inventory system. However, when I switch the sprite back to the SAME image, the image becomes blurry (The top two images on the left):

I have tried setting the sprite as generate mip maps vs not generate mip maps, filter mode point vs bi-linear, format true colour vs 16 bit vs compressed, none of it is happening. No matter what combination of those three settings I use, the image is still blurry when swapped out via code.

The image is 1024 x 1024 so it is a power of 2, the max size is set to 1024.

I am at a loss for how to solve this problem.

The code I am using to swap the image is a simple image.sprite = (sprite loaded from resource)

 public void AddItem(int itemID)
     {
         if(inventoryItem[itemID] != null)
         {
             Debug.Log ("Item exists!");
 
             for(int i = 0; i < inventoryOpen.Length; i++)
             {
                 if(inventoryOpen[i] == true)
                 {
                     inventoryOpen[i] = false;
                     inventorySlots[i].sprite = inventoryImages[itemID];
                     inventoryItem[i] = itemID;
                     Debug.Log ("Added item! Slot: " + i + " Slot Name? " + inventorySlots[i].name + " Item ID: " + itemID);
                     break;
                 }
             }
         }
     }

I really appreciate any help or thoughts anyone has on this problem.