Mining settlements, those villagers can mine the gems. You may need the gem mining card though, not sure because I left the gems alone until well after I had gotten the card.

Thanks for your answer, I have to apologize because a few days after I posted this, I had enough spare resources to mess around that area, the chest were not under the colored rocks it turns out, it was ahead of it, as in closer to the viewer, because that chest was literally in the middle of the mountain but on the ground level - it was very deep - I see the particle effects above those colored rocks and thought the chest was under them, spending more worship to widen my meteor stikes crator, I finally saw and accessed the chest.

I didn't know those were gems! Initially thought they were, built a mining settlement next to some, nothing happened, yes I problably don't have the gem mining card.