I think they're more afraid of his fanatical supporters. Look at the threats of violence against the Georgian Republicans who didn't cave to Trump in 2020. Trump only needs to allude to violence to get his fanatics working on his behalf