Very yes to all of the above. The temperature in your house isn't something that should rely on potentially inconsistent methods without solid fallbacks and Google doesn't seem to realize that.

Activating it in Nest should disable it in Home and vice versa, so I would test that out again. You could also try turning your wifi off and see if that helps. Android will still use wifi scanning for location when the wifi switch is off, and that tends to reduce drift on my 7a.

Do you have location history on? If so, does your timeline show you moving? Any change if you enable Home/Away Assist in the Nest app instead? Which Pixel? I changed from a 6 to 7a and have experienced way more drift on the 7a.