Do LOD's with different textures affect performance

Was wondering if you can all help. I am in the process of creating many LOD's for a project I am working on, but due to the difference in poly count and UV mapping, my LOD's would require different UV maps for each LOD. My question is will this hurt performance? i.e. does unity keep the old LOD texture in memory or does it remove it....