There should an end to the mission when you captured all of the Katana Fleet but you may have ran into a bug.

If you capture the last enemy ship it will cause the battle to never end and the only way to resolve is to reload a save

Thanks for the help you two, now i dont wraste like 5h trying to solve it myself

It was my first playthrough and I didn't realize I had to reload if that happened, I just thought the next missions would still happen after. Nope! I've now conquered every system and since I never technically finished the katana fleet thing, the missions that allow entry into two of the systems never happened, and as such I cannot finish capturing every system.