I stated all this before, but I forget where. Apologies if you've read this theory already.
Rinnosuke actually visits what's probably (but not definitively) the Hakurei Shrine of the outside world in CoLA. Sadly, he doesn't get to actually SEE anything besides the fact that it's a shrine.
It should be noted that while he was there, he actually heard a lot of noise, as if the place was occupied. At that moment, Yukari kicked him out and booted him back to Gensokyo, and proceeded to make up some lies (blaming everything on the IPod and telling Rinnosuke it was all just a dream) in order to get Rinnosuke to forget about it (it works).
It should be noted that Akyu states "It is said on that the Hakurei Shrine on the outside is abandoned and desolate" or something like that. The fact that Akyu starts the sentence with "it is said" indicates there's a good chance that whoever is telling her this information (likely Yukari, since Yukari is the main suspect with the power to actually VISIT the outside in the first place) could possibly be lying.
My own personal guess from these tidbits is that the Hakurei clan is alive and well on the outside world, and that Yukari takes a new one at a young age (baby? Child? An age too young to remember anything, probably) into Gensokyo whenever they need a new maiden there. And for whatever reason, she doesn't want anyone to know this, and thus lies to everyone that the Hakurei Shrine on the outside world is actually abandoned and desolate.
(of note in PCB is that Yukari indicates that while this was the first time Reimu may have seen her, she has clearly been aware of Reimu since long before then)
If the above is true, then Yukari might be getting tired of always having to spirit away a new maiden, seeing as how she actually tries to trick Reimu into becoming immortal in IN's extra stage by consumming the elixir. Either that or she just likes Reimu a lot compared to previous maidens. Or maybe both.
Another side tidbit: ZUN often times refers to himself in Touhou works as Kinnosuke Hakurei, which means "Head of the Hakurei Clan" or "Chief Priest of the Hakurei Clan" or something like that. He describes himself as a composer in BoiJR, in character. It's possible that the racket Rinnosuke heard was him trying to test his compositions (although if that's the case, the testing must have been going pretty badly since the sound was pretty horrible)
