It was revealed that Subaru thinks that is happening. Either he's wrong or the author messed up royally (imho). They lost soldiers, but it was nothing like when he traveled with the merchant.
They actually remember that they had 20 men, the Merchant (Odo was his name?) didn't even remember that he traveled in a group, or did he? I can't remember - and if he did, isn't it more likely that he remembers the merchants that survived, since the whale only followed Subaru. And it would be weird that the whale has to eliminate them in a specific way for them not to be erased.
At least he didn't remember that Subura had a companion.
That beast existed for 400 years. How do Survivors know that the whale killed someone? Why does the Old Man even remember his wife? etc. etc.
And why did the whale "let Subaru go" and "disappear", it clearly tried to kill him? Something is not adding up imho, but I could be wrong.