I think it's due to not technically having a retinue member at that second. Problem is pressing back did nothing and it just froze, unable to press anything. Had to force close the game