The fort didn't have soldiers or archers to attack from behind the wall. They basically had the 3 elite fighters and the general (and a bunch of fodder who let the MC go alone for some reason). If they let the enemy come close to the wall, the enemy can just burn it down.