The problem isn't with the AVI format/container, but the codec you are using to encode the footage with.
You could use a codec with higher compression but you'll lose some quality and possibly run into some issues while editing with those files.
Or, you could make clips of the footage you need instead of converting whole VOBs:
Method 1: Creating Clips for Source Footage. Depending on the amount of clips you create, and how efficient you are in making them (no extra footage), you may only use about 5 to 10 GBs of hard drive space for your video footage (depending on the resolution and colorspace).
For example, my RockFullyMAD video's clips totaled about 7.5GB for a 2 minute 35 second video. I admit that I'm bad at efficient clip making so cutting out the slack, I could have saved another 0.5 to 1 GB (I think I converted the entire clean end credits (~550MB) when I actually used only 5 seconds of it...

).
Another alternative is to make fake AVIs to edit with:
Method 2: Editing DVD footage directly.