First problem – economical. Short version - Specialization is OP.
Right now the way to build a settlement is to focus it entirely on one aspect.
That is too primitive of a system.
Kick specialization in the [censored] - Introduce diminishing returns on worker efficiency.
I'd say about -10% per head after first.
One worker on a work type 100%
Two – 190%
Three – 270%
Four – 340%
Five – 400%
You get the picture.
The point is to create a system where you need to actually think about how you're building up your settlement. Should mix very nicely with tile-specific unique features if you ever implement any.

Second Problem – military. Short version – cannon fodder is OP.
Even vanilla rim-world's value to efficiency rates are tilted.
They work for the colony, because every colonist counts, and every loss counts as well.
But with empire armies... 10% better armor and 10% better guns aren't worth having 25% less troops on the field.
And then there are mods. When their item values come into play it all turns into a circus.
My solution? Forget about absolute freedom of troop design and about being able to use moded items.
It sounds cool, but it's impossible to balance, and when it's not balanced it's not fun, no matter how cool it looks on paper.
Stick to standardized military units. Or at least re-balance your costs from the ground up and keep the mods out of it.
Also standardized units would make it easier to simulate empire's tech progression.
For example, right now, the moment I research guns in my colony I can immediately equip my entire empire with guns even though I can't even produce enough for my colony because components are scarce.
That feels very hollow.
If I needed some kind of military upgrade or a building to unlock the next tier of standardized units that'd actually feel like my empire IS advancing technologically and guns DID become available there.

P.S. One thing you might wanna code in sooner than everything else is “local spending”. Basicly a system that allows us to re-route the production of the settlement right back to the settlement without it traveling to us and back.
Example:
Village needs 1000s for upkeep. It produces [stuff] for total worth of 2000s. Empress Jane the Plain wants to tithe it for some [stuff] but also wants it to cover it's costs and wants to build something in there that's worth 1500s. She then sets local spending to 1750s. She gets 250s worth of [stuff] tithed, 1000 worth of upkeep covered, and once two tax periods have passed the village accumulates a local 1500s worth of funds and can now build that something.
It's convenient, it doesn't bloat the wealth of the capital (for raiders and such) and, if you ever code in ambushes of your caravans, it meshes in well with that too.

Just wanted to point out that it'd make sense to have that sooner rather than later.

Well, that is once you're done with ploughing through all the “fun” the 1.3 update has dumped on you.

Stay sane, we're with you. Mostly in spirit though ;P.