If the video source is starting to run low on available scenes or by using the entire song you start to dilute your video concept, it's probably a good idea to edit it down. And there are some songs that are
, or can get, repetitive after a certain point and should probably be edited.
4 minutes is what I consider the average length of AMVs. But tastes vary, and for some people certain music and/or videos will start to be less interesting after a certain length.
Ultimately, it's really dependent on how well edited and how focused is the video concept whether the length becomes an issue.