A US state rejected a proposed settlement by an investment firm to repay nearly $6m (€4.2m) to state investors who lost money in Bernard Madoff’s fraudulent investment scheme.