Hmm - try getting a nice fleece. They are warmer than wool and less scratchy *and* you can get ones that look pretty smart for semi-dressy stuff. They also wash really well (though I've never tumble-dried one) and come out from spin-drying almost dry and ready to wear again.
Alternately *don't* put jumpers in your wash basket in the first place - have somewhere else to put them and do a separate wash for all the stuff you aren't going to tumble dry. It's a good habit to get into. /me has shrunk too much nice stuff from forgetting to separate it.

Fleeces tumble dry fine. They dry quicker than everything else, but they don't seem to suffer from being left in. If they did, I wouldn't own any :)