The escape button should *ALWAYS* exit out of current mode.
This is not a feature that should be put off. Instead of exiting the destroy mode, the tiny little button that I have to click was not clicked. No. Instead, I destroyed my farming settlement.
When the outcome is one of "what I wanted" or "catastrophic failure that requires game restart several hours in because there's no saving" you have a fundamental bug.