It's a pathing calculations / job assignements issue that won't be solved I'm afraid. After a certain population level, the game's AI simply has too many things to calculate to get it all done before the next "round" of calculations is supposed to start.
Using a no-smoke mod and turning down the shadow quality help a bit with graphical stuttering and game smoothness, but at some point you'll always slow down, purely based on population numbers. Your setup should let you reach pretty high pop though, especially if you don't mind playing at X5 speed or lower. (gives the AI more time to wrap things up before the next "round")

Thanks! I wish this game would develope more it has a lot of potential.

Sadly it was published as a "finished" games, with only plans to fix bugs, add a modkit and port to Linux/Mac. The dev said he wanted to move to another game after that, with the experience gained from making Banished.
If that slow-lag-death of towns was a bug, I think we'd be lucky as he'd probably could be convinced to do something about it. I'm afraid it's just the way his game engine/AI is made that wasn't really optimal, and redoing that would just be too much work for pretty much no profit to him. Let's hope whatever he makes next is better planned and doesn't have that issue. :)