Unity re-imports everything when switching Github branches

Hi, I'm using Github to version control my Unity project and I haven't figured out how to prevent Unity to recompile/reimport all the assets when I switch branches. It takes a good 30min now each time I switch branch or change computer. If I understand correctly meta files should prevent Unity from doing that. Is there anything I'm missing? (In the Editor Settings I have my Version Control mode set to Visible Meta File.)