Taking a colored pearl out of the inventory instantly bugs the map, at least when the slug senses remix option is enabled :/
not sure if it helps

Slug Senses doesn't effect the Inventory mod for me, whether I put a colored pearl in or take it out. I have found that after I disabled Key Item Tracking, it did clear up this issue and a couple others. If you have a map mod, like MapOtions, you might also want to make sure it goes below Inventory in the load order. Can you try these things out and let me know if they work?

you could remap the inevntory hotkey, i heard that if you tap map twice it switches from inventory to map, not sure if this is true