Pixel Perfect and Forced Aspect ratio

I'm currently working on a pixel art game. But something has been driving me crazy, and that is getting pixel perfect and aspect ratio right. I've been looking everywhere, but can't find what I'm looking for. A few

I want to display a level with the resolution of 640x480 (4:3) and should be the same on multiple resolutions. A few examples are TowerFall and Shovel Knight. So the first thing I did was looking for pixel perfect camera's, this was my result:

The pixels are nice and perfect but this is not how people like to play a game. So I decided to look at forcing the aspect ratio (with letteboxing/pillarboxing). It does kind of get the result I want, but the big problem is that the pixels are not square anymore:

So I really don't know what to do. Is it possible to get the result I'm looking for, am I on the right track, Or am I doing it totally wrong?