Reducing server-side application size

Hi. I'm trying some Unity networking with a Server-Client architecture. The server is run on a headless (no graphics) host, with the "-batchmode" and "-nographics" arguments. The thing is, I noticed the server application size grow as I'm developing my client (because I'm adding assets and stuff).

Is there a way I can create a server application which doesn't contain all the textures, sounds, etc. inside (because it'll never use them anyways) so that my deployments are easier/faster?

Thanks for your time! :D