ogres spawn when your keep is walled off; although if your entire island is enclosed that should do it. Check for gaps around rivers, docks and fishing huts -- if the peasants can get through with supplies, it probably also counts as a place the vikings can get through your walls. Of course there are sea gates and normal gates so it's still possible to wall off the island and maintain access; but those are the main "gaps" that come to mind.

Note that diagonals can be tricky and inconsistent -- it's best to fill in all corner sections just to be sure.

Also note that if you've used any kinds of natural rocks (unusable, wolf dens, or even stone/iron deposits) as part of your wall, then they might count as a gap even though the vikings can't get through them. I've used that in the past to "cheese" walling in my keep without triggering ogres; I thought it was patched long ago but I could be wrong?

Finally, if you've done a "perfect" wall right out to the edge of the shallow water using piers, then there's simply nowhere for vikings to land and hence ogres won't spawn if they have nowhere to disembark. You may need to designate a landing area and build the most powerful kill-box possible around it -- I guarantee you that if you have an otherwise impenetrable island, the second you open up that kill-box you'll get a horde of ogres and catapults that will challenge even the most terrifying defences.