The trees can only be removed / selected to be destroyed once they are entirely built. It's one of the limitations we have to accept in order to be able to build them anywhere.

The reason why they're not being built is that you might have selected some specific frut trees that require some fruits to build. Like decorative chestnut trees require some chestnuts as building materials.
Try remembering which tree you chose, check in the building tab if it looks like they need something specific, and order that from the next food traders that pass through your town.

Once built, if you select them you'll have a small icon in the top right of their info window. Either a little "destroy" button, or an "upgrade" one. If it's the Upgrade button, they will need to be upgraded to become impassable, which also turns them into a regular "building" you can destroy with the regular destruction tool.