Crowdsourcing an anime background remover

[Zarxrax]

With advances in AI technology, there is the possibility of creating machine learning models that can automatically remove backgrounds from images. For anime music videos, this can be a huge time saver for rotoscoping.

You can already use this technology today. Here is a guide I wrote for using an existing model: https://docs.google.com/document/d/1YOf ... lzdw5g4bt9

But what if we could make it better? I think it might be possible.

A machine learning model is only as good as the dataset that it has been trained on. The model above was trained on anime-style artwork (as opposed to real frames from anime), and it included many poorly-made masks. It's already pretty good, but I think it could be improved.

I propose to train a new model using a dataset that is highly targeted towards real anime videos. I will need help from many people in order to put together a good dataset. I implore all AMV editors to reach into their old projects and get me some good frames of your masks (I don’t need your entire clips, just some frames from them). Reach out to anyone you know who does a lot of rotoscoping or masking, and see if they are willing to help contribute!

Edit: I am no longer requesting any additional images. See updates below.

[Replay Studios]

I be happy to help!

[JazzsVids]

Also available to help.

[AngelicDaze]

Would anime cel scans be useful for this? I don't do masking like this but I have a bunch of cels with backgrounds that I've scanned separately.

[Zarxrax]

Would anime cel scans be useful for this? I don't do masking like this but I have a bunch of cels with backgrounds that I've scanned separately.

Yeah that could potentially work.

[Zarxrax]

I have updated the first post to remove some of the requirements around the masks. I have been exploring different types of networks and models which work differently than the one I had originally planned, so have loosened the requirements on the dataset accordingly.

[Zarxrax]

Well, the end of the year is here, so I thought I would provide a little update.
While I did not get much response to this request, I forged ahead with doing things a slightly different way, and as I saw new research throughout the year, I also continued to change my approach a few times. That's one of the issues in this field--if you don't move fast, something bigger and better is always right around the corner.

After months of work, I did finally complete putting together a 26,000 item dataset which I was going to start training some models with... And that's when I came across 2 brand new projects which just made my existing plans obsolete.

So right now, I am focusing on taking the work of these projects, and turning them from tech demos into actual user-friendly applications that fit nicely into a typical AMV workflow. I will hopefully have something to show soon!

[Zarxrax]

And here it is. https://github.com/Zarxrax/Cutie-Roto

I made a new thread for further discussion. No need to continue here, as this topic is now out of date.
viewtopic.php?f=49&t=133573