Suggestions to save space, time and money

[Shadow Wolf]

With as such a big project of AMV I know it costs a lot of money to maintain. One of the reasons for the 2 week waiting period was so people wouldn't sign up multiple times, use twice as much space, bandwidth and hopefully instead decide to stick with one account. I understand it’s a huge undertaking and can be overtaxing especially if you only have a few people administrating, perhaps on volunteer who don't get paid.

One thing I've noticed is a lot of wasted space. It comes in two formats. Either someone is waiting for their video to be deleted or the topic to be removed or you have someone making a post about a project they are working on. When I do a search for videos one of the most aggravating things is the waste of time and space when you see something like, "I've decided to work on Project X but at this time I'm out of clips. Keep looking here in the future." or "I have this idea but am looking for clips.". It takes up space, bandwidth and costs money and they need to be removed.

There are a couple ways to do this to speed things up, allow members to delete their own work. Perhaps its a limiting in the programming that’s being used but I never understood why someone couldn't just delete their own video and topic. This would reduce probably 40-50% of the PMs/Mod Requests and free up more time for them to clean up other things.

The second idea would be to either have an application process or ask some people your familiar with and trust to see if they would volunteer to help out. I know there are plenty of people including myself who would be perfectly happy volunteer to go through and remove Video posts that are "Projects In Work".

When you’re fully caught up then things will be much easier to manage and handle. The longer things take, the more you fall behind and the more money it costs in the long run. Getting caught up on just deleting excess video requests would probably clear up at least at most a GB or more. Then remove excess "Work in Projects" clears up a few MBs of space on whatever mySQL or database system you’re using. Not to mention helps cut down on bandwidth because when people do video searches it isn't being wasted showing things that aren't even there.

[downwithpants]

the limiting resource on the donut is bandwidth, not space i believe. the faster we take stuff down, the faster people will put new stuff up, and the less bandwidth we will have for downloads.

[Shadow Wolf]

It uses bandwith when I do a search for videos. Lets say I do a search for Linkin Park and it will do a search pulling up 2000 or so videos. Probably about 10% of those are dead links, etc which translates to a few pages of wasted bandwith. Believe me it adds up faster than one thinks. Even with there being more space and ability for people to upload it frees up more bandwith because there isn't wasted links using bandwith up during searches.

[Scintilla]

It uses bandwith when I do a search for videos. Lets say I do a search for Linkin Park and it will do a search pulling up 2000 or so videos. Probably about 10% of those are dead links, etc which translates to a few pages of wasted bandwith.

It shouldn't matter whether or not the video's links are valid.

The point of this site, first and foremost, is to be an AMV catalogue that lists as close as possible to all the AMVs in existence.

[Shadow Wolf]

It shouldn't matter whether or not the video's links are valid.

The point of this site, first and foremost, is to be an AMV catalogue that lists as close as possible to all the AMVs in existence.

You are correct, first and foremost, it should list all AMVs in existence. However WIPs (Works In Progress) are not in existence, aren't even created and who knows how long until they will be created. Therefore they shouldn't be listed.

How does WIP pages effect bandwidth? Well it actually depends on the Host AMV is using to host the services, what programs or methods they are utilizing to measure bandwidth. Any unwanted information, pages that create long searches waste bandwidth and cost money.

There is a reason why most message boards limit signature and avatar file sizes as well as properly trim their message board. For example lets say I do a search on this message board for lip syn and pull up 458 matches, 16 pages of information. We'll say it uses 10 MB of bandwidth (its not exact its just for an example). If old topics were removed (ie:WIP) and it was reduced in half, making the bandwidth usage 5 MB. Ok so it saved 5 MB, big deal right. The board has 198,882 users and lets say in a week only 5000 of those users do the a search, well thats 25,000 MB of saved bandwidth.

If you ever had to pay for bandwidth on an EZ Board before you would understand why most people ask that settings change topics per page to viewed to 50. It saves a lot of bandwidth. Now granted that was an example and a search on the message board probably uses less bandwidth than that, just depends on how they have their board setup.

Now lets look at a AMV Video Search. If I do a search for Linkin Park for example that pulls up 2346 results, which is about 117 pages. Most likely half of those are WIPs and the pages can be reduced to quite a bit, it helps reduce bandwidth usage. When you have a lot of users doing a lot of searches it adds up quickly. Trust me, I know since I pay for 2 websites and admin another 3 making sure they don't get abused and bandwidth doesn't get wasted.

Now I'm not an expert and there are other methods to reduce bandwidth, however this is the fastest and simplest way and it does save money to trim things down. Like I said before it should show actual videos, link or unlinked but there are A LOT of WIPs which shouldn't even be there. Add in the many requests they are behind in deleting videos/topics from members, the longer they take the more money it costs and further behind they get.

[Phade]

Hey,

With as such a big project of AMV I know it costs a lot of money to maintain. One of the reasons for the 2 week waiting period was so people wouldn't sign up multiple times, use twice as much space, bandwidth and hopefully instead decide to stick with one account. I understand it’s a huge undertaking and can be overtaxing especially if you only have a few people administrating, perhaps on volunteer who don't get paid.

Actually, one person can make 100 accounts if they really want to. They can only use one accout at a time to access the site, thus only use the bandwidth of 1 person no matter how many accounts they have. Plus, having a few entries in the database doesn't really cost the site any extra since that entry is literally 1/36,000,000th of the capacity of the web/DB server. Web/DB server storage is currently only 32% and most of that is system log files (which can be zipped if need be). Site data and scripts currently takes up only about 3% of total web/DB server storage.

One thing I've noticed is a lot of wasted space. It comes in two formats. Either someone is waiting for their video to be deleted or the topic to be removed or you have someone making a post about a project they are working on. When I do a search for videos one of the most aggravating things is the waste of time and space when you see something like, "I've decided to work on Project X but at this time I'm out of clips. Keep looking here in the future." or "I have this idea but am looking for clips.". It takes up space, bandwidth and costs money and they need to be removed.

A video entry is another thing that is exceptionally small to store. Displaying the Super Search results and video information page is now also something else that is exceptionally small. This page, for example, is only 9,657 bytes when uncompressed. Site pages are gzip compressed before being sent from the server to the client, thus making the actual page sent and bandwidth used even smaller. The same goes for forum topics.

Storing additional video files is not an incremental cost since we own the server that stores them. The only additional cost there is when the video is downloaded. Videos that suck are downloaded much less often than good videos. Good videos actually "cost" the site more than bad videos, but good videos are why we are all here downloading.

However, I do agree that works in progress entries should not be in the database. It is against the rules of the site to enter incomplete videos into the catalog. This is a completness and courtesy rule, not a bandwidth-saving rule.

There are a couple ways to do this to speed things up, allow members to delete their own work. Perhaps its a limiting in the programming that’s being used but I never understood why someone couldn't just delete their own video and topic. This would reduce probably 40-50% of the PMs/Mod Requests and free up more time for them to clean up other things.

Allowing members to delete their own works will not save anything. In fact, deleting your own works goes against the purpose of this site: to catalog every AMV ever created.

The second idea would be to either have an application process or ask some people your familiar with and trust to see if they would volunteer to help out. I know there are plenty of people including myself who would be perfectly happy volunteer to go through and remove Video posts that are "Projects In Work".

There are already 3 members in addition to myself who have privleges to do exactly that.

When you’re fully caught up then things will be much easier to manage and handle. The longer things take, the more you fall behind and the more money it costs in the long run. Getting caught up on just deleting excess video requests would probably clear up at least at most a GB or more. Then remove excess "Work in Projects" clears up a few MBs of space on whatever mySQL or database system you’re using. Not to mention helps cut down on bandwidth because when people do video searches it isn't being wasted showing things that aren't even there.

There are currently 437.81 GB of free storage space. After having the service running for more than a year, we aren't even half way done filling the current storage capacity.

Search results pages do not significantly impact the site bandwidth usage. In fact, bandwidth usage for the web site (not including video hosting) is actually quite manageable. Last month was the highest amount of web site traffic yet with ~190G transferred. With our previous host, we were allotted 700G/month at a fixed cost per month. Our current host has a much greater limit than that and still at a fixed cost. The web site alone (not hosting) could easily be run at a fixed cost around $200 per month (maybe even less) for a dedicated server and some bandwidth.

The #1 cost by far incurred to the site is by people downloading videos that are hosted locally. So far this month, we have transferred about 8.62 TB of videos. That is just for the month of April and the month isn't even over yet. If the site really wanted to save money, everyone would have to download a lot less videos. If we wanted to remove the videos that were causing the most traffic, we would have to start with all the good videos. Anyone in favor of that? No, I thought not.

So to recap, catalog entries, forum posts, and search page results have extremely low cost. Having good videos downloaded over and over again is a high cost. If we wanted to cut costs, we would have to limit the amount of downloads from the storage server. The videos that cause the greatest amount of downloads and therefore the greatest amount of cost are generally the better AMVs.

I hope this helps.

Phade.

[Zarxrax]

Hey,

The second idea would be to either have an application process or ask some people your familiar with and trust to see if they would volunteer to help out. I know there are plenty of people including myself who would be perfectly happy volunteer to go through and remove Video posts that are "Projects In Work".

There are already 3 members in addition to myself who have privleges to do exactly that.

Phade.

So if we were to post a ton of bad video information pages, say, to the mod dropbox, we could expect them to be deleted? Or is this something where noone has time to handle it? The reason I ask this is because its known that the video deletion requests take a long time to get handled. Perhaps more people could be assigned the job?

[Calim]

There are currently 437.81 GB of free storage space. After having the service running for more than a year, we aren't even half way done filling the current storage capacity.

No wonder why the server costs so much.

[Shadow Wolf]

It is against the rules of the site to enter incomplete videos into the catalog. This is a completness and courtesy rule, not a bandwidth-saving rule.

Which was why someone was appointed or handled that to help stem that. I found more WIP pages this weekend then actual videos.

Allowing members to delete their own works will not save anything. In fact, deleting your own works goes against the purpose of this site: to catalog every AMV ever created.
---------------------------------
There are already 3 members in addition to myself who have privleges to do exactly that.

From what I understand there is a backlog of deletions that need to be done. For example there is a video on my wifes account that needs to be deleted but hasn't. There are members who have remastered works to upload but can't because they have one version of it already up and can't. I understand things get busy and just trying to help provide suggestions to help reduce some of the management for the admins. Maybe if people donate a certain amount or people who have a history of remastered get the ability to remove their own work so they can upload. *shrugs* There are probably other methods you can ensure deletions don't happen unless to do an update but it helps reduce work load for you.

So to recap, catalog entries, forum posts, and search page results have extremely low cost. Having good videos downloaded over and over again is a high cost.

Compared to the bandwidth downloading a video, catalog entries, forum posts and search pages do have a low cost. Having a 100 members access those daily is a low cost but when it starts to get more and more people doing those things every day, each of those amounts add up quicker than you'd think. Maybe not at this point but at some point it will and good maintenance before it becomes an issue usually prevents it from happening. *shrugs* They were just suggestions anyways but if your confidant that they aren't issues, your the one paying the bills.

[Phade]

Hey,

It is against the rules of the site to enter incomplete videos into the catalog. This is a completness and courtesy rule, not a bandwidth-saving rule.

Which was why someone was appointed or handled that to help stem that. I found more WIP pages this weekend then actual videos.

Stupid is as stupid does. There is no way to prevent people from entering bogus entries. If we happen to find them, we can remove them. There is an annual "house cleaning" fest called "The Viewer Choice Awards" that gets a good bit of the entries. A good way to weed out WIP vids is to use the "has premiered" option to ignore videos that do not have a premiere date.

Allowing members to delete their own works will not save anything. In fact, deleting your own works goes against the purpose of this site: to catalog every AMV ever created.
---------------------------------
There are already 3 members in addition to myself who have privleges to do exactly that.

From what I understand there is a backlog of deletions that need to be done. For example there is a video on my wifes account that needs to be deleted but hasn't. There are members who have remastered works to upload but can't because they have one version of it already up and can't. I understand things get busy and just trying to help provide suggestions to help reduce some of the management for the admins. Maybe if people donate a certain amount or people who have a history of remastered get the ability to remove their own work so they can upload. *shrugs* There are probably other methods you can ensure deletions don't happen unless to do an update but it helps reduce work load for you.

Having members delete their own entries can easily be abused. The majority of the deletion requests that we get are denied because they're usually, "Ummm, I've now realized that my video actually sucks, so can it be deleted?" The answer here is "no" because the point of the catalog is to list every AMV ever created, not just the good AMVs. Again, the bandwidth used by suck videos is small because they are download much more rarely than good videos.

So to recap, catalog entries, forum posts, and search page results have extremely low cost. Having good videos downloaded over and over again is a high cost.

Compared to the bandwidth downloading a video, catalog entries, forum posts and search pages do have a low cost. Having a 100 members access those daily is a low cost but when it starts to get more and more people doing those things every day, each of those amounts add up quicker than you'd think. Maybe not at this point but at some point it will and good maintenance before it becomes an issue usually prevents it from happening. *shrugs* They were just suggestions anyways but if your confidant that they aren't issues, your the one paying the bills.

The ratio of bandwidth used by the web site to the bandwidth used by the storage server last month was 0.0223. Of the total bandwidth used last month, only 2.19% was the site info itself. If we get more members, the ratio will most likely stay close to that number.

The #1 method to reduce the 2.19% of the bandwidth used would be to convert the site pages from table-based designs to XHTML/CSS-based designs. Of the four pages that I've converted so far, the average savings was around a 6-fold reduction in page size. This, more than anything else, will save web server bandwidth. Removing database entries will have little to no effect since the search results page will generally pull the same number of records for each query (assuming that your query has >20 results, which the vast majority do).

However, even if page sizes were the same, it would have taken about 3.5x jump in the number of visitors for the entire month to get close to our previous ISP's bandwidth limit of 700G/month for the web site. Our old provider has since up-ed the limit for a similar type of account to 1500G/month which would have required nearly an 8x jump in the number of visitors. Even with that many more visitors, the amount of bandwidth used by video downloads still dwarfs the bandwidth used by the site itself.

Focusing on 2% of the total bandwidth used will not yield any significant savings. Even a 20% savings there will only yield a 0.4% total cost savings, which at our current fixed rate is only theoretical $10/month saved. Reducing cost is something I'm always on the lookout for, but I think you're barking up the wrong tree.

So to recap, the bandwidth used by the web server is nowhere near capacity limits. It would take an incredibly large increase in the number of visitors to reach current web server bandwidth limits. The bandwidth used (and thus the cost) by the web server is dwarfed by the bandwidth used by the storage server. Even a large savings in web server bandwidth usage would not yield significant cost savings due to the enormous cost of the storage server bandwidth.

I hope this helps.

Phade.