Holy shit no way

:3

while going through the code of Godot I also found this. Gave me a laugh
github.com/godotengine/...

Am tired now.
As a last resort I try changing the gravity and setting the scale to 0.
Now it works....
I am angy... But hey it work now...

Change again, now it ignores the wall, change something again, now it falls to the floor when colliding, change again, now it glitches again.
Some trial and error later, I look at the source code of Godot. Another while later I see that I am not doing anything wrong, it should just work.