The human motor system can adapt to perturbations by updating existing internal models of motor control or by creating context-specific motor memories or strategies that can be flexibly activated or deactivated. Using a virtual target-directed ball rolling task, we investigate if motor learning is context specific when perturbations are applied to either the throw direction of a ball, or the acceleration of the ball post-release. Additionally, by altering the visual slant of the task surface in an immersive virtual-reality environment, we determine if informative visual cues that predict visual perturbations to expected action consequences affect the propensity for context-specific motor learning. Our findings reveal that perturbations resembling post-throw accelerations enable flexible, context-specific motor adaptation regardless of the presence of the visual slant cues. Perturbations in the throw directions, conversely, lead to the updating of existing internal models. Additionally, we find fast implicit visual-context-specific changes in performance during early learning and context switching. Our findings underscore the role of visual properties of both perturbations and environments in flexible and generalizable motor learning. Data and analysis scripts available at https://osf.io/a5nv3/.