luffy has let others hold his hat. He's let ussof hold it.

I don't think its about who Luffy trusts more, or how he trusts them all the same, its simply that he trusts them.
with this behind us, we can play who is the closest to him all we want, but he can count on each to surpass themselves and to eventually rise to meet the situation.