Nigerian troops shelled an Islamic sect’s compound and broke into their mosque, touching off a raging gunbattle that left scores dead.