Generally the best idea is to not add or remove any mods in the middle of a save, unless the mod page specifies that you can do one of those.
Generally, if a mod is very simple and doesn't add any new features, only expanding upon existing functions, then it would work.
The safe mods include texture-only mods, which are pretty much always fine, except Vanilla Expanded texture mods, which are often more complicated.

If you want to be extra sure, I would say as long as a mod doesn't have a folder labelled "Assembly" or "Source", then it's probably fine.