But I don't really know why people feel this way.
Well, there are several often cited opinions on the matter. Let me play a devil's advocate here.
1. "Survival play is easy; the games don't push you hard enough."
This is true. By standards of most arcade shooters, doujin games, including Touhou (even on Lunatic), are easy to survive due to the enormous amounts of bombs, extends, and panicbomb time they offer. Cases in point: every no-vertical, no-horizontal, and no-movement replay around. Scoreplay largely alleviates this issue since you have to play aggressively to score well.
Ironically, Cave are going the opposite way nowadays: they're gradually adding autobomb as a permanent feature of their recent releases, and Futari Arrange mode has a long-lasting bullet-repelling shield a-la GigaWing.
2. "Stages are boring and drawn-out."
To a large extent, this is true. If you play a couple credits of, say, DoDonPachi, you'll notice that the stages are pretty much a non-stop rush of enemies that come in way more varied orders and waves. There are no long pauses where you can just relax or stand in one spot for several seconds, you are forced to constantly move all around the screen just to not get overwhelmed. SA and UFO are somewhat better than earlier games in this respect.
3. "Scoring systems are obtuse and feel like a tacked-on gimmick."
This is very hard to debate, especially if the game is question is Perfect Cherry Blossom, where score is comprised of a million factors that you need to account for on the fly to score competently. Personally, I consider it a great advantage of Touhou games, because, even if the scoring systems are gimmicky and unintuitive, they at least make sense. The exceptions are Subterranean Animism, which has hilariously broken Lunatic, and the entirety of MoF, which is just mediocre item chaining and killing bosses at just the right time. It's a good thing ZUN has gone for a less-punishing system with UFO, where dying once doesn't fuck you up badly for the rest of the game, as opposed to SA and especially MoF.
I won't touch graphic and music design because it's too subjective to even consider debating. For instance, nintendonut888 hates animated bullet sprites because he finds them distracting. :D
On the other hand, you shouldn't be too concerned with this kind of criticism because every game company gets criticized a lot for their game design choices. I could list a good deal of common qualms with Seibu Kaihatsu's, Cave's, Raizing's, Treasure's, Taito's, Konami's and Psikyo's shmups, and these pretty much constitute the most popular shmup developers ever. Not sure if you know this, but games like Ikaruga and Radiant Silvergun get bashed on, like, a daily basis.