Saito decided not to fight Kenshin because he wasn't the same man that he met during the Bakumatsu. He changed a lot mentally, plus HMR ruined his body and he lost his skills.

www.miburo.com

Check that site out for Saito info. Perhaps a historical event happened there that would explain his transfer there.