The Library folder is hidden. to open it, click the desktop to make sure you’re in the Finder, hold down the Option key, then choose Go > Library.

After getting there open up the steam folder.

After getting in the steam folder go to the SteamApps

After that open up the Common folder.

Then you should see the folders for all of your games, open the Garry's Mod one.

Then open up the garrysmod folder.

After you get there open up the download folder and you should see a folder called servers.

You're now a few clicks away, once you've opened up the servers folder delete everything thats in it and it will remove all of the addons downloaded from the servers

DO NOT DELETE THE FOLDER

Do not delete it or you won't be able to download server addons anymore which means you probably won't be able to join a server anymore, simply select all of the items in the folder and drag them to the trash and then empty the trash, after this we are done and you have deleted all of the server addons and now have a bigger and cleaner hard drive.