Honestly? It stopped bothering ME before it stopped happening. Once I decided my accent was evidence I'd learned a whole second language, not a flaw, the self-consciousness mostly dissolved. Locals almost never care as much as we fear.
My grammar is getting decent but the moment I open my mouth, my accent marks me as foreign and I brace for the switch to English. Some days I don't mind; other days it genuinely knocks my confidence. For those further along — does the self-consciousness about your accent ever fully go away, or do you just stop caring?