I'd have a look at the rear axle, sounds like you could have some movement there. At a guess I would say your rear bushes on the control arms are gone. I need to change mine as I have a similar problem but it is more noticeable when in reverse.
Chock the front wheels and raise a rear wheel and look for any sideways movement, then do the same for the other side. It might only be one thats gone. I found mine when I was changing wheels over and on tightening up the wheel nuts I noticed some movement, got underneath and found my rear lower bushing had a bit of play.