I switch when the other person switches first, and treat it as their call to initiate rather than mine. Safer default than guessing.
My language has one 'you' and here there are two, with a whole etiquette about which to use with whom and when to switch. I've been too formal with friends (cold) and too casual with an official (rude) and cringed both times. How did you learn to feel the line rather than just memorise the rule?