I know there's a lot of discourse about Whitebeard ditching Xebec rather than realizing he was being controlled, but I feel like it's not unreasonable given he didn't especially like the guy. He followed him out of duty, was insulted and attacked, and took his chance to leave.