Well, the Quicktime problems with many videos is a result of the fact that, last I knew, QT7 doesn't support anything beyond Baseline Profile + an implementation of 1 or 2 B-frames. I'd wager that a lot of videos coming out are probably making use of Main or High Profile due to the better compression options available there. I personally use High Profile (or the equivalent thereof; I don't really pay attention to profile restrictions too much) 99% of the time, for example. So unless there's some other Quicktime-based solution to output or play more Profiles, there you go.
What Quicktime (and therefore FCP) outputs is at the simple-to-decode end of the AVC scale, so most players capable of h.264 playback are able to play them with few problems. Another factor might very well be hardware resources, but as for why VLC drops frames and mplayer doesn't, my only guess is that VLC isn't kept as up-to-date as mplayer is (or has other problems in general, which has pretty much been my diagnosis all along).
Basically, think of it sort of the same way as QT6 supporting MPEG-4 Part 2's Simple Profile, but not Advanced Simple Profile (which is what DivX and XviD are implementations of).
Another thing that might influence it, relative to the climate right now, is that as far as I know, neither HD-DVD nor Blu-ray's specifications for h.264 are public knowledge yet, and there's not really been a real solidification of guides for encoding h.264 yet either, unlike what you see for XviD (what I mean is that the different XviD guides I see tend to agree on certain points, whereas the relatively few h.264 guides I've seen don't necessarily agree too much on what many see as integral features). This might be due to the fact that x264 is still technically in early development, even if it has been deemed stable enough to use, and been around since early 2005. Overwhelmingly, like XviD (and maybe early versions of DivX), the solidifying is probably going to come as a result of the online encoding community, which tends to gravitate toward the free option, and thus h.264 guides are probably going to coalesce around x264 usage and Quicktime, etc. will be more of an afterthought just as it was with ASP.