I was using area2d in both cases (ball and pad), so not sure

Something similar happened to me when building and arkanoid. Probably the ball is reacting to the other side of the pad. I would try changing the collision shape so that the back side is further away. It