Crowdsourcing an anime background remover
- Zarxrax
- Joined: Sun Apr 01, 2001 6:37 pm
- Location: North Cackalacky
- Contact:
Crowdsourcing an anime background remover
With advances in AI technology, there is the possibility of creating machine learning models that can automatically remove backgrounds from images. For anime music videos, this can be a huge time saver for rotoscoping.
You can already use this technology today. Here is a guide I wrote for using an existing model: https://docs.google.com/document/d/1YOf ... lzdw5g4bt9
But what if we could make it better? I think it might be possible.
A machine learning model is only as good as the dataset that it has been trained on. The model above was trained on anime-style artwork (as opposed to real frames from anime), and it included many poorly-made masks. It's already pretty good, but I think it could be improved.
I propose to train a new model using a dataset that is highly targeted towards real anime videos. I will need help from many people in order to put together a good dataset. I implore all AMV editors to reach into their old projects and get me some good frames of your masks (I don’t need your entire clips, just some frames from them). Reach out to anyone you know who does a lot of rotoscoping or masking, and see if they are willing to help contribute!
Edit: I am no longer requesting any additional images. See updates below.
You can already use this technology today. Here is a guide I wrote for using an existing model: https://docs.google.com/document/d/1YOf ... lzdw5g4bt9
But what if we could make it better? I think it might be possible.
A machine learning model is only as good as the dataset that it has been trained on. The model above was trained on anime-style artwork (as opposed to real frames from anime), and it included many poorly-made masks. It's already pretty good, but I think it could be improved.
I propose to train a new model using a dataset that is highly targeted towards real anime videos. I will need help from many people in order to put together a good dataset. I implore all AMV editors to reach into their old projects and get me some good frames of your masks (I don’t need your entire clips, just some frames from them). Reach out to anyone you know who does a lot of rotoscoping or masking, and see if they are willing to help contribute!
Edit: I am no longer requesting any additional images. See updates below.
Last edited by Zarxrax on Sun Dec 31, 2023 7:51 am, edited 6 times in total.
- Replay Studios
- A Studio Within A Studio
- Joined: Sat Oct 04, 2008 8:10 pm
- Status: Replaying myself
Re: Crowdsourcing an anime background remover
I be happy to help!
- JazzsVids
- Joined: Tue Dec 22, 2009 11:53 pm
- Location: USA
- Contact:
Re: Crowdsourcing an anime background remover
Also available to help. ![Smile :)](./images/smilies/icon_smile.gif)
![Smile :)](./images/smilies/icon_smile.gif)
- AngelicDaze
- Joined: Thu Jun 23, 2016 8:57 pm
Re: Crowdsourcing an anime background remover
Would anime cel scans be useful for this? I don't do masking like this but I have a bunch of cels with backgrounds that I've scanned separately.
- Zarxrax
- Joined: Sun Apr 01, 2001 6:37 pm
- Location: North Cackalacky
- Contact:
Re: Crowdsourcing an anime background remover
Yeah that could potentially work.AngelicDaze wrote: ↑Sat Feb 18, 2023 8:04 amWould anime cel scans be useful for this? I don't do masking like this but I have a bunch of cels with backgrounds that I've scanned separately.
- Zarxrax
- Joined: Sun Apr 01, 2001 6:37 pm
- Location: North Cackalacky
- Contact:
Re: Crowdsourcing an anime background remover
I have updated the first post to remove some of the requirements around the masks. I have been exploring different types of networks and models which work differently than the one I had originally planned, so have loosened the requirements on the dataset accordingly.
- Zarxrax
- Joined: Sun Apr 01, 2001 6:37 pm
- Location: North Cackalacky
- Contact:
Re: Crowdsourcing an anime background remover
Well, the end of the year is here, so I thought I would provide a little update.
While I did not get much response to this request, I forged ahead with doing things a slightly different way, and as I saw new research throughout the year, I also continued to change my approach a few times. That's one of the issues in this field--if you don't move fast, something bigger and better is always right around the corner.
After months of work, I did finally complete putting together a 26,000 item dataset which I was going to start training some models with... And that's when I came across 2 brand new projects which just made my existing plans obsolete.
So right now, I am focusing on taking the work of these projects, and turning them from tech demos into actual user-friendly applications that fit nicely into a typical AMV workflow. I will hopefully have something to show soon!
While I did not get much response to this request, I forged ahead with doing things a slightly different way, and as I saw new research throughout the year, I also continued to change my approach a few times. That's one of the issues in this field--if you don't move fast, something bigger and better is always right around the corner.
After months of work, I did finally complete putting together a 26,000 item dataset which I was going to start training some models with... And that's when I came across 2 brand new projects which just made my existing plans obsolete.
So right now, I am focusing on taking the work of these projects, and turning them from tech demos into actual user-friendly applications that fit nicely into a typical AMV workflow. I will hopefully have something to show soon!
- Zarxrax
- Joined: Sun Apr 01, 2001 6:37 pm
- Location: North Cackalacky
- Contact:
Re: Crowdsourcing an anime background remover
And here it is. https://github.com/Zarxrax/Cutie-Roto
I made a new thread for further discussion. No need to continue here, as this topic is now out of date.
viewtopic.php?f=49&t=133573
I made a new thread for further discussion. No need to continue here, as this topic is now out of date.
viewtopic.php?f=49&t=133573