Now I find it harder to focus and the anxiety ends up taking over the yoga rather than the other way around.

I have a tendency to recursive anxiety, though for me it mostly happens with climbing rather than yoga -- I'll start worrying about whether I'm going to enjoy it and whether I'm going to start feeling anxious/depressed, which then pretty much guarantees that I'll start feeling anxious/depressed.

I don't know if this is similar to what you're dealing with, but it's a pain.