Making floating specks for space movement

Hello all -

I am making a game where the main player ship flies through space at a relatively slow pace. I have a space-y skybox (thats slightly broken, but, a separate issue) so it looks like the ship is in space. I am not yet worrying about shaders to make the ship look like its in space, that can come later. For now, I am looking for a way to create dots in 3d space so that it looks kinda like space junk. I don't want them to pass as stars, since it wouldn't make sense to have stars floating around and moving as the ship moves. However, what I want to do is make all these dots around everywhere so as the player ship moves (it is stationary in the middle of the screen and camera rotates around it) and the camera rotates, the specs move around so one can see that the ship is moving.

My current thought for this is using a oneshot particle generator to make dots EVERYWHERE along the path of the ship, but I'm worried that with a lot of these particles, performance is going to suffer. Is this a workable method? I did some tests and even though the view shows lots of dots all around, I can't seem to see any when I actually run my game. I want them to vary in size and color (from white dots to gray, etc) so it looks almost natural as just specks floating in space.

Any suggestions?