What VR headset do you have?
If its a WMR (Windows Mixed Reality) headset its tracked by your head so your hands in front of you it wont really work. its hard to explain.

Galaksee I have the same problem using the Rift S.
Grabbing with the left and pushing down left while looking right up, I guess the Rift loses the tracking? I am using the level editor right now to find out about the distance to the wall because I also ged "pushed away" from the wall and stuck to it.

I did some tests and for me, there is a latency difference between the right and left hand with Oculus Rift S controller and fresh batteries. If I let go on the right hand there is about half a second latency - on the left hand it is not that much.

I recorded a video:

I use the grab on both controlls and let go on both controls. Decpide letting go off both hand simultaniously, the right hand reacts slower. Pushing of walls is not possible. :(

https://youtu.be/kwYFAj-Uryw

Switch to the public beta on Oculus Home and potentially also switch to the SteamVR beta, I've seen this latency and it's not my game but somewhere in the drivers/firmware from what I can gather.
Switching to the beta on Home fixed it for me personally.