I'm a little bit south of you, and they swarm ours! Spiky orange and black caterpillars. They can strip a plant, but passionflower is resilient. :)

And great for butterflies! Especially gulf fritillaries.

I planted some a few years ago; still surprised (happily) where they pop up.