I think you should go for it tbh (unfollow everyone except your close friends). You’ll feel much better when your feed + mind are decluttered. If people are attacking you for it, then they’re only validating your reasons to unfollow them.