Nope, that recruitment building is useless to you except for the 1600 you make out of destroying it which you have no reason not to do; it's even a viable strategy (maybe not the best, but it is a thing you can do) to go around enemy settlements that are bothering you can deconstruct their buildings or just the recruitment building so that the enemy may retake it but won't be able to pump a unit out for at least 6 turns. Do that to 4 settlements close together and that 4 units a turn the enemy will not send your way or use to garrison their towns, and that's 24 units that will never get in your way again. (Meanwhile you are of course building up a new army and/or marching into other settlements.