Mr Rumsfeld expressed particular concern about any clear-out of Iraq's defence and interior ministries, which are at the heart of efforts to put Iraq's security forces in charge of battling the country's Sunni Muslim-led insurgency.
Some fear the Shi'ite and Kurdish parties that came to power in the January 30 election could dismiss Sunni officials after suffering decades of oppression under Saddam Hussein's regime.
"It's important that the new government be attentive to the competence of the people in the ministries and that they avoid unnecessary turbulence," Mr Rumsfeld told reporters on his plane before arriving on his ninth visit to Iraq since the invasion.
"We have an opportunity to continue to make progress politically, economically... anything that would delay that or disrupt that as a result of turbulence, or lack of confidence or corruption in government, would be unfortunate," he said.
He also warned against delays in the political process, particularly the drafting of a new constitution that is due for completion by mid-August, ahead of elections in December.
Former interim prime minister Iyad Allawi, a secular Shi'ite who led the country from June last year until last week, kept many former members of Saddam's Ba'ath Party in government.
But the new Shi'ite leadership wants to see a thorough clearing out of the old authorities, a process known as de-Ba'athification, which could fuel the two-year insurgency.
After meeting Mr Rumsfeld, new Prime Minister Ibrahim Jaafari said he hoped for a stable political transition and an inclusive government.
"I don't deny that there are challenges," said Mr Jaafari, a moderate Islamist who heads Dawa, a Shi'ite political party that opposed Saddam's regime for decades in exile.
"I am sure we are going to form very good ministries. All of them (workers) they are good technocrats. They are very effective from different backgrounds. So I think we can co-operate, all of us, and face these challenges successfully."
Despite progress with the naming of Mr Jaafari and President Jalal Talabani, a Kurd, to lead the transitional government, political factions are still jockeying for power. But Mr Talabani said he hoped a government would be finalised soon.
"We need time to form the cabinet, which we hope will be finished before the end of this week," Mr Talabani, a former guerrilla leader, said after meeting Mr Rumsfeld.
Noting a legal provision allowing a delay of six months in writing the constitution before the next round of elections, Mr Rumsfeld said the US would oppose such a move.
"If someone wants to hang around waiting for perfection in this business, you are unlikely to find it," he said.
Ahead of his arrival, US embassy officials confirmed an American contractor was kidnapped by militants while working at a project site near Baghdad. More than 150 foreigners have been abducted in a year-long wave of kidnappings in Iraq.
An embassy spokesman said no one had claimed responsibility.
Violence persisted elsewhere in the country. In Qaim, near the border with Syria, witnesses said US warplanes bombed suspected insurgent positions a day after two suicide car bombers attacked a US military base.
Local hospital officials said 11 people were killed and 17 wounded in clashes between insurgents and US and Iraqi forces, but the figures could not be separately verified.
And in the northern city of Mosul, a car bomb targeting a US military convoy killed five Iraqi bystanders and wounded three, according to police and hospital officials.