An editor of a small Malaysian newspaper has resigned for reprinting Danish newspaper caricatures of Prophet Muhammad, which have unleashed a storm of protests across the Islamic world.