Wall Street closed out a dismal third quarter with a mixed performance Thursday after Merck & Co. said it would be forced to remove its best-selling Vioxx arthritis drug from the market.