I'm suprised no one has mentioned this. :V
I can relate to this. Scarlet Meister and Scarlet Gensokyo are no joke, and Remilia's final nonspell is probably among my top 5 for frustrating patterns, even if it isn't her hardest attack.
Koishi has destroyed way too many of my SA Extra runs with her spellcard Suppression "Super-Ego". On paper, it does seem pretty easy, but I just can't do it no matter how many times I try, which makes her one of my most feared ex-bosses, even if just for that pattern.
Honorable mention to Yumemi Okazaki, for being the only touhou boss that completely annihilates me, even on lower difficulties.