How to organise SDK/Framework/Gameplay code separation? (for it to work with continuous integration)

We're currently in the process of migrating our codebase from a monolithic "all source code in the project" to a more sensible Framework/Game approach.

We're also thinking about our build pipeline, and how this could be handled without the need to use Unity Editor, in a push for a more controlled way of deploying Framework's releases and handling SDKs for our projects. We aim to get our framework deployed as binaries, to simplify packaging. This point creates a whole new world of interesting problems in itself.

The current state of our code is roughly the following set of folders

 [3rd party]
 [Game]
 [Framework]
 [Plugins] (unity specific folder)

We're trying to break that in something like that

 [--Framework unity project--]
 [3rd Party & SDK]
 [Framework]
 [Plugins]
 
   |
   | build and deploy
   v
 
 [-- Game unity project--]
 [Game]
 [Plugins] (contains built 3rd party, built Framework, and Plugins)

Using the AsmDef system allows a welcome flexibility in organising code, but as specified in the doc, it's not a build system, and we're aware of that.

We were then thinking about setting everything framework related in an isolated VS solution, so we can control its build independently. But then comes the inevitable puzzle of 3rd party code and unity dependencies (as in "referencing unity dll in the framework").

 [-- Visual Studio Solution --]
 [Framework] (references unity without breaking, in an install path agnostic way)
 [3rd Party & SDK]
 
   |
   | build and deploy
   v
 
 [-- Game unity project--]
 [Game]
 [Plugins] (contains built 3rd party, built Framework, and other Plugins)

That is not really an easy solution, due to the reliance on Unity's modules, and adding them as reference will be a challenge (because not everyone will install unity at the same path). At this point, we're open for suggestions and advice.

Do you know documentation regarding that sort of setup available to you, or guidelines and recommendations on how to setup that in the best way for Unity projects? With the various profiles of Unity users, there should be some public doc available on the subject, but we've failed to find something to help us move forward without relying on trial and error.