I send a second guy to the "pope" to improve relations with him towards my empire. It didn't add a third +20 and I am not sure it even increases speed. So I called the last sent guy back.

Turns out the game is incapable of discerning between lasting agents and the recall action. The moment I called the second one back, relations began to detoriate by -4 (Foreign Agents) despite the first one still being there... doing the exact same thing.

You clearly missed a "remaining agents doing X" check and used a simple on an off switch in diplomacy for agents. Not amusing... and bad for my current relations.