when coming back to the village alone without him, the two of them would not have had an escort of 4 heavily armed cavalry soldiers guarding 2 VIPs to Kazura, their benefactor, with a (mostly empty) wagon... A wagon which the bandits mistook for food-aid from the city, which was their primary objective during the current ongoing famine. A wagon requiring heavily armed guards/extra security....not too many reasons for soldiers or officials to visit a remote village during a famine when taking that into account, other than to either provide food or collect taxes. So Kazura leaving, was the start of the butterfly effect, and the reason for his brooding while sitting on a box, before boarding the carriage. Valetta at the end was about to say that even she couldn't have predicted an attack, so Kazura shouldn't blame himself for the same.