You should watch this interesting analysis about the various physical impossibilities of the hotel in The Shining. There’s an argument to be made that these inconsistencies are the result of simple set design flaws, but even in that case the point of the analysis is that (intentional or not) the spatial anomolies present in the film afford its horror. That said, I think that some of them were probably intentional. I was reminded of the impossible camera behavior in What Lies Beneath, which is absolutely intentional and serves a similar purpose.
As you watch this, you should consider the lakeside hotel in Silent Hill 2. This is the location in which the game’s story makes its key reveal, and in addition to being a sort of similar hotel, it also exhibits some very odd spatial behavior.
Silent Hill 2 actually got me thinking about screwing with the layout of physical space as a way to scare the player, but so far no other game that I can recall has picked up on this approach. There are games with portals and such (Ghost Hunter has a really nice one!), and games in which two areas appear similar but in fact diverge (e.g. the otherworld vs the real world in all the Silent Hill games). One of the best aspects of Silent Hill 4 is the apartment door that imprisons the protagonist: objects slipped under the door come out differently on the other side, and no amount of pounding can be heard by passers-by. But I can’t think of a game that actually tries to be scary by using physically impossible level layouts.
The only game I can think of (besides Silent Hill 2) that really did this is Marathon 2, an early (and awesome) first-person shooter (the predecessor to Halo). Marathon comes from the era of ray casting engines, in which physical space was defined by 2D polygons that were connected at the edges. Polygons can overlap without physically intersecting; it’s only when the edges of one area are explicitly connected to the edges of another that the two areas become a single space. This allowed for “5D space”–areas that physically overlap but yet existed as independent spaces. Marathon supports hallways that pass through each other but do not intersect. This was, as you might expect, throughly confusing, though the Marathon series made no attempt to use it seriously (it was more of a funny “look what we can do” feature used only in multiplayer maps). But it seems to me that the idea might be applicable to horror.