How to fix wobbling float when away from spawn?

You wont like this answer but in general it boils down to: "You simply don't go that far"

Why is that? Because Unity uses float precision to handle positions. If you want to know why that happens what you see you should learn how floats work. The short version is: The larger your number the less space your variable has to calculate precise decimals. So the best thing in your case would be to either scale everything down by a factor of 10 so that everything is basically closer to the origin.

OR: (and that is probably the better and proper solution) Change your frame of reference. Instead of moving your plane, move the world. This way you always stay close to the origin and can (in theory) have an infinitly large world.