Do you have a Mark one cherokee or a Facelift version?
I dont know if its common or if I am even in the right area, but just a stab in the dark, on the facelift models the air direction controls work off the vacuum system so I would suspect a leak somewhere, I would have thought one of our resident experts would have been along already to advise from their deep experience. Does cruise control work if you have it?