If he were leaving out of cowardice, then yes. But every time he's left a fight, it's because the fight wasn't his goal.
When fighting Morel, his only objective was to get to the King. He didn't care if he defeated anyone or not, all he cared about was getting free of the smoke, finding out where the King went, and going after him.
When fighting Knuckle and Killua, he similarly had other objectives, first killing the girl, then destroying the gungi board. Knuckle didn't have the girl, so he left, and Killua didn't have the board, so he left.
So far, I haven't seen him leave a fight because he was afraid of losing.
I think that's a stretch. I think he retreats because his true goals are often time sensitive. He could take the time to fight Morel, but in the time it would take to do it, something could happen to the King(and he ended up taking too long as it was). He could take the time to fight Knuckle or Killua, but in that time, the King could end up discovering the things he's trying to hide from him.