US troops killed dozens of insurgents near the Syrian border with Iraq after an ambush that left five Marines dead.