I happen to play with all of my authored mods enabled, plus one or two others. So when I write this keep in mind I have Construction Flavoured Garrisons + Increased Tower Rate of Fire enabled. The combination is harsh even on Normal difficulty.

I attacked a "undefended" Castle level town that had a Sword School, Naginata Dojo and Bow Dojo. The resulting garrison spawn was two Ashigaru plus six Samurai units - a mix of bow, sword and spear - a Retainer and finally an Onna Bushi, backed by no less than four archery towers (I use a castle mod too). I was attacking with almost a full stack of ashigaru.

I must have tried that assault five or six times, with each attempt taking 30-45 minutes to play out. Each time I lost. Any time i'd approach a manned section of wall i'd get murdered by Bow Dojo Samurai. If I tried to sucker the bowmen to one wall and scale another i'd get murdered by towers. If I did manage to get up, the Samurai swarmed over and destroyed me. On a few rare occasions I managed to capture and hold towers, but my men were eventually overwhelmed and the tower ownership flipped back without inflicting enough casualties, whereupon I was down one, sometimes two units without inflicting any lasting harm on the defenders.

In summary, once a town grows large enough, and has enough recruitment buildings in it, the garrison by itself is usually enough to resist everything short of a full stack of Samurai. The fact that it's backed by improved towers makes it a nightmare to break into a high level castle without siege weaponry. I end up laying siege and luring the defenders into attacking me, where I can leverage the terrain and the defenders don't have the unbreakable morale they have inside a fortification.

You can still cheese your way through with autoresolve, sadly.