Large-scale visualization of MEP attributes

This forum is for the general discussion of Anime Music Videos.

Large-scale visualization of MEP attributes

Postby trythil » Wed Oct 03, 2007 11:40 pm

I've started up a side project (one out of, uh, 56) to do some ad-hoc analyses of MEPs. Those in the AMV IRC channel have already been spammed to death by these images, so I thought I might as well push this out to a larger community.

I've generated a few graphs of MEPs. These graphs link editors to MEPs, and are laid out using a force-directed layout algorithm (specifically, the Fruchterman-Reingold algorithm).

For this first iteration I did only a direct application of the algorithm, making no attempt to influence the layout with AMV-specific data, e.g. I did not try to pin project founders / coordinators closer to their projects and I did not try to group similar projects (RoS series, DDR series, NES/RVG, etc.). Nevertheless, the nature of the algorithm does expose some interesting insights into the collection of MEPs on the .org, and I'll discuss a few of them. You might (and, I hope, will) see other patterns. I'll discuss future work after I've presented the current work.

I broke the graphs up into three subsets: all MEPs with more than 2 editors, all MEPs having 2 < editor count <10>= 10 editors.

NOTE - These are big, big PNGs. I know Firefox and Safari can handle them, given sufficient memory. (For Firefox, you will want to turn off Firefox's automatic image resizing; this can be done by typing about:config in the address bar and setting browser.enable_automatic_image_resizing to false.) Nevertheless, you may want to download them and view them in a program explicitly designed to handle large images.

A note on reading these graphs: Vertices and edges are coded by brightness and color. The color is assigned by an object's ID, and is meaningless beyond that, except perhaps as a guide to assist in following edges. The brightness of an edge or vertex is directly proportional to the degree of the node:

- The brightness of a member node (and the brightness of its edges) is linear in the number of MEPs in which that member has participated.
- The brightness of a video node is linear in the number of editors participating in that project.

I tried some other scaling schemes, such as logarithmic scaling, but I liked the the way linear scaling worked out, so.

I am placing these graph images in the public domain. Do whatever you want with them.

Here's the graphs:

MEPs with more than two editors:
Image

The middle section is quite unreadable, but by scrolling around you can see some neat large-scale patterns. First off, you can see a bunch of small production studios, as well as some that connect to larger projects by one or two people (Studio Gaijin, consisting of Kazz, Spyral, and Silvercat, is like this); secondly, there's a pretty obvious "mainstream" and then some much smaller, but still prolific, sub-communities. It's interesting that said "mainstream" and those smaller communities appear to be disconnected from each other.

The nature of the layout algorithm (i.e. in a nutshell, it attempts to minimize edge lengths, make edge lengths as uniform as possible, and distribute nodes as evenly as possible) tends to places MEPs with similar editor sets next to each other; this is how you get e.g. the DDR projects all stacked next to each other, with Kusoyaro pretty close to them all. It also tends to put editors who have done similar projects next to each other, and also tends to put editors who tend to do MEPs (in godix's terms, "MEP whores") in the center. You can check out the center of the "mainstream" segment to confirm this.

==

MEPs having (2, 10) editors:

Image

This one takes out some of the noise caused by the mega-MEPs, and brings to light some more patterns. My favorite one is the Random Destination Studios cluster in the middle-right, mostly because it's one of the few I can recognize; I'm sure that there's other studio clusters present as well. It'd be interesting to see if (and how) those clusters interact.

Also, I think this graph also makes a case for AtomX actually being in RDS, despite his constant claims to the contrary. :P

==

MEPs with 10 or more editors:

Image

Another view of the big MEPs.

==

So, where to go from here? Here's a few things I want to do:

- Address problems in the data: Not all the multi-editor projects cataloged on the .org are present here, simply because the collaboration data hasn't been entered for all MEPs. I'm pretty sure that I'm missing a bunch of More Than Toast videos, for example.

- Use more domain-specific data to shape the graph: One thing I want to do is use hit (and, if possible, download) counts to influence the graph. It could, for example, provide some insight into the most popular MEPs, the editors that tend to join those MEPs, and so forth. There's other things that can be done with the data from this site.

- Try different layout algorithms: Force-directed layout algorithms are pretty handy for this kind of stuff, but I've got my mind open to other ideas. One thing I'd like to do is to use splines or polylines instead of straight lines for the graph edges; that'd allow me to clean up the layout significantly. Unfortunately, I've not been able to get the solvers I've tried (graphviz and aiSee) to route spline edges with any sort of reasonable efficiency. I know it's a computationally-intensive problem, but I'm hoping there's something out there that will handle it.

==

Technical details: I used a pair of Ruby scripts to generate these graphs. The first Ruby script parsed video information pages using the Hpricot HTML parser (oh what I would give for a good Web service API), and stored the data in a MySQL database via Active Record. The second Ruby script actually did the graphing, controlling Graphviz via the ruby-graphviz bindings.

If anyone would like a copy of these scripts, the DOT files fed to Graphviz, and/or the dataset I used, just let me know in this thread and I'll post them. I'm not posting them now because I'm not particularly happy with the code (actually, some of it is outright embarrassing) and I'd like to clean it up before publishing it.
This is a block of text that can be added to posts you make. There is a 512 character limit.
trythil
is
 
Joined: 23 Jul 2002
Location: Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch

Re: Large-scale visualization of MEP attributes

Postby Brad » Wed Oct 03, 2007 11:53 pm

trythil wrote:Also, I think this graph also makes a case for AtomX actually being in RDS, despite his constant claims to the contrary. :P


Despite the fact that CDVV people and other random people are around me too :P
Image
Ask Brad an After Effects Question! - Forum - Site
User avatar
Brad
 
Joined: 20 Dec 2000
Location: Chicago, IL

Re: Large-scale visualization of MEP attributes

Postby Kitsuner » Wed Oct 03, 2007 11:57 pm

AtomX wrote:
trythil wrote:Also, I think this graph also makes a case for AtomX actually being in RDS, despite his constant claims to the contrary. :P


Despite the fact that CDVV people and other random people are around me too :P

So it actually makes a case for AtomX just being a slut. :amv:
OtakuGray wrote:Sometimes anime can branch out to a younger audience and this is one of those times where you wish children would just go die.
Stirspeare wrote:<Stirspeare> Lopez: Vanquish my virginity and flood me with kit. ["Ladies..."]
User avatar
Kitsuner
Maximum Hotness
 
Joined: 16 Feb 2003
Location: Chicago, IL
Status: Top Breeder

Postby downwithpants » Wed Oct 03, 2007 11:58 pm

you have too much free time
maskandlayer()|My Guide to WMM 2.x
a-m-v.org Last.fm|<a href="http://www.frappr.com/animemusicvideosdotorg">Animemusicvideos.org Frappr</a>|<a href="http://tinyurl.com/2lryta"> Editors and fans against the misattribution of AMVs</a>
User avatar
downwithpants
BIG PICTURE person
 
Joined: 03 Dec 2002
Location: storrs, ct
Status: out of service

Re: Large-scale visualization of MEP attributes

Postby trythil » Thu Oct 04, 2007 12:01 am

AtomX wrote:
trythil wrote:Also, I think this graph also makes a case for AtomX actually being in RDS, despite his constant claims to the contrary. :P


Despite the fact that CDVV people and other random people are around me too :P



Well then I guess you're just a whore :P

==

One thing I forgot to add, and I'm not sure how I forgot this because it's precisely what I'm working on now:

Future work: interactive tracing -- I want to provide a way for people to trace paths through the graph, e.g. by highlighting certain nodes and edges. I'd like to do this with SVG and JavaScript, mostly because that's The Web 2.0 Way; but something tells me that most SVG viewers would explode with a graphic of this complexity. It's more likely I'll end up using a Java-based graph inspector / viewer or some such. (Or maybe I'll do something totally crazy and code a Flex or Silverlight application. Heh heh heh. :P )
This is a block of text that can be added to posts you make. There is a 512 character limit.
trythil
is
 
Joined: 23 Jul 2002
Location: Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch

Postby Kariudo » Thu Oct 04, 2007 12:08 am

downwithpants wrote:you have too much free time

x2
ImageImage
Image
User avatar
Kariudo
Twilight prince
 
Joined: 15 Jul 2005
Location: Los taquitos unidos
Status: 1924 bots banned and counting!

Postby NS » Thu Oct 04, 2007 12:34 am

Ho. Ly. Shit.
ImageImage
User avatar
NS
I like pants
 
Joined: 08 Jul 2006
Location: Minneapolis, Minnesota
Status: Pants

Postby CrackTheSky » Thu Oct 04, 2007 1:21 am

David, shouldn't you be cleaning your fridge?
User avatar
CrackTheSky
 
Joined: 27 Aug 2006
Location: Chicago
Status: Probably not editing.

Postby trythil » Thu Oct 04, 2007 1:46 am

downwithpants wrote:you have too much free time


The scripts themselves took about a half-hour to write :P The layout took significantly longer, but I ran those jobs while I was at work or at home sleeping. :P

CrackTheSky wrote:David, shouldn't you be cleaning your fridge?


I did that during graph layout :P
This is a block of text that can be added to posts you make. There is a 512 character limit.
trythil
is
 
Joined: 23 Jul 2002
Location: Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch

Postby JaddziaDax » Thu Oct 04, 2007 1:58 am

interesting... yay im next to jubby on one of those graphs :)
User avatar
JaddziaDax
Crazy Cat Lady!
 
Joined: 16 Mar 2004
Location: somewhere i think O.o
Status: I has a TRU Arceus

Postby trythil » Thu Oct 04, 2007 2:17 am

Oh, right. Here's a hypothesis I meant to throw in my original post: the MEP scene is highly inbred.
This is a block of text that can be added to posts you make. There is a 512 character limit.
trythil
is
 
Joined: 23 Jul 2002
Location: Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch

Postby ngsilver » Thu Oct 04, 2007 3:47 am

I like how I'm nowhere near the rest of my studio, and somehow right next to sailor earth. (in the big version)

This is quite an interesting representation of everything.
ImageImage
ImageImageImage
User avatar
ngsilver
The Hookah Guy
 
Joined: 28 Jun 2003
Location: Detroit area
Status: Calm Your Miniature Colorful Horses

Postby Sephiroth » Thu Oct 04, 2007 5:37 am

i watch over the children project with an iron fist of death.
User avatar
Sephiroth
 
Joined: 19 Dec 2000
Location: California

Postby Mr Pilkington » Thu Oct 04, 2007 6:09 am

Why must I get continually blamed for CBR Got Back?


They stole my footage and my name. :x
Am I that much of an AMV media icon that they needed me on the project to sell it to the public? I kind of thought the other way. I figured since I haven't made a decent video in years that people avoided my shit like bio-engineered crazed, flesh eating, locusts on steroids.
Image
<Orwell> WTF are they doing upstairs. Almost sounds like construction
<[Kristyrat]> Orwell, you see / <[Kristyrat]> when a man really likes a woman / <[Kristyrat]> or has a bottle of chloroform
<inthesto> He takes her stuff / <inthesto> And then poops in her vagina
User avatar
Mr Pilkington
 
Joined: 09 Apr 2002
Status: Stay outa my shed

Postby Sephiroth » Thu Oct 04, 2007 7:12 am

Because you are a super smart raptor from jurassic park 2. NO one can keep you locked in a closet as long as you have a newspaper.
Mr Pilkington wrote:They stole my footage and my name. :x
Am I that much of an AMV media icon that they needed me on the project to sell it to the public? I kind of thought the other way. I figured since I haven't made a decent video in years that people avoided my shit like bio-engineered crazed, flesh eating, locusts on steroids.
User avatar
Sephiroth
 
Joined: 19 Dec 2000
Location: California

Next

Return to General AMV

Who is online

Users browsing this forum: No registered users and 2 guests