Tell them if you have to go back to the office, you quit.

Nah, never say “or I’ll quit”, you just say you won’t go to an office. End of statement. You don’t make any sort of threat or give them an ultimatum, just set your boundaries. Make them fire you, if you’re valuable and/or well liked odd are they won’t and you just get to keep working remote.