Crowdsourcing an anime background remover

Topics not related to Anime Music Videos
Post Reply
User avatar
Zarxrax
Joined: Sun Apr 01, 2001 6:37 pm
Contact:
Org Profile

Crowdsourcing an anime background remover

Post by Zarxrax » Wed Feb 15, 2023 7:05 pm

With advances in AI technology, there is the possibility of creating machine learning models that can automatically remove backgrounds from images. For anime music videos, this can be a huge time saver for rotoscoping.

You can already use this technology today. Here is a guide I wrote for using an existing model: https://docs.google.com/document/d/1YOf ... lzdw5g4bt9

But what if we could make it better? I think it might be possible.

A machine learning model is only as good as the dataset that it has been trained on. The model above was trained on anime-style artwork (as opposed to real frames from anime), and it included many poorly-made masks. It's already pretty good, but I think it could be improved.

I propose to train a new model using a dataset that is highly targeted towards real anime videos. I will need help from many people in order to put together a good dataset. I implore all AMV editors to reach into their old projects and get me some good frames of your masks (I don’t need your entire clips, just some frames from them). Reach out to anyone you know who does a lot of rotoscoping or masking, and see if they are willing to help contribute!

Edit: I am no longer requesting any additional images. See updates below.
Last edited by Zarxrax on Sun Dec 31, 2023 7:51 am, edited 6 times in total.

User avatar
Replay Studios
A Studio Within A Studio
Joined: Sat Oct 04, 2008 8:10 pm
Status: Replaying myself
Org Profile

Re: Crowdsourcing an anime background remover

Post by Replay Studios » Thu Feb 16, 2023 11:27 pm

I be happy to help!

User avatar
JazzsVids
Joined: Tue Dec 22, 2009 11:53 pm
Location: USA
Contact:
Org Profile

Re: Crowdsourcing an anime background remover

Post by JazzsVids » Fri Feb 17, 2023 1:34 am

Also available to help. :)

User avatar
AngelicDaze
Joined: Thu Jun 23, 2016 8:57 pm
Org Profile

Re: Crowdsourcing an anime background remover

Post by AngelicDaze » Sat Feb 18, 2023 8:04 am

Would anime cel scans be useful for this? I don't do masking like this but I have a bunch of cels with backgrounds that I've scanned separately.

User avatar
Zarxrax
Joined: Sun Apr 01, 2001 6:37 pm
Contact:
Org Profile

Re: Crowdsourcing an anime background remover

Post by Zarxrax » Sat Feb 18, 2023 11:04 am

AngelicDaze wrote:
Sat Feb 18, 2023 8:04 am
Would anime cel scans be useful for this? I don't do masking like this but I have a bunch of cels with backgrounds that I've scanned separately.
Yeah that could potentially work.

User avatar
Zarxrax
Joined: Sun Apr 01, 2001 6:37 pm
Contact:
Org Profile

Re: Crowdsourcing an anime background remover

Post by Zarxrax » Sun Feb 26, 2023 5:00 pm

I have updated the first post to remove some of the requirements around the masks. I have been exploring different types of networks and models which work differently than the one I had originally planned, so have loosened the requirements on the dataset accordingly.

User avatar
Zarxrax
Joined: Sun Apr 01, 2001 6:37 pm
Contact:
Org Profile

Re: Crowdsourcing an anime background remover

Post by Zarxrax » Sun Dec 31, 2023 7:49 am

Well, the end of the year is here, so I thought I would provide a little update.
While I did not get much response to this request, I forged ahead with doing things a slightly different way, and as I saw new research throughout the year, I also continued to change my approach a few times. That's one of the issues in this field--if you don't move fast, something bigger and better is always right around the corner.

After months of work, I did finally complete putting together a 26,000 item dataset which I was going to start training some models with... And that's when I came across 2 brand new projects which just made my existing plans obsolete.

So right now, I am focusing on taking the work of these projects, and turning them from tech demos into actual user-friendly applications that fit nicely into a typical AMV workflow. I will hopefully have something to show soon!

User avatar
Zarxrax
Joined: Sun Apr 01, 2001 6:37 pm
Contact:
Org Profile

Re: Crowdsourcing an anime background remover

Post by Zarxrax » Mon Jan 29, 2024 6:04 pm

And here it is. https://github.com/Zarxrax/Cutie-Roto

I made a new thread for further discussion. No need to continue here, as this topic is now out of date.
viewtopic.php?f=49&t=133573

Post Reply

Return to “General Discussion”