It would be nice if you had a slider for more of less bandits.
Add more bandits, it will improve the late game.
If you talk to villagers, they can share rumors of where bandits might be.