Roger Federer and Andy Murray made it safely through to the quarter-finals but Nadal was on the wrong end of an incredible 6-3 6-4 3-6 4-6 15-13 scoreline.
An epic final set took two-and-a-quarter hours — more than half an hour longer than it took Federer to dispatch Grigor Dimitrov.
The Spaniard, winner here in 2008 and 2010, served to stay in the match an incredible nine times, but the 10th proved too much even for him.
At four hours and 48 minutes, the match lasted exactly the same time as his epic final against Federer nine years ago.
“It was not my best match against a very uncomfortable opponent,” said Nadal.
“Well done to him. He played great, especially in the fifth set, but I fought to the last ball.”
Muller, who has won more matches on grass than any other player so far this year, troubled Nadal with his thunderous serve and penetrating volleys throughout.
The French Open champion had not dropped any of his last 28 sets in grand slams, but suddenly Muller took the first two off him.
Nadal drew on all his experience to dig deep to draw level, only to find himself facing two match points at 4-5 in the decider against an equally resolute opponent.
Two big first serves, and two aces, averted that particular crisis but Muller was still serving the better of the two.
That was until the 18th game of a marathon final set when, having got to deuce, Nadal chiselled out four break points.
A Muller double fault on the fourth, correctly overruled by umpire Ali Nili, served to ramp up the tension yet further as the underdog held.
Nadal saved two match points in the next, and when he came out to serve for the 22nd game he complained to Nili about the reflection of part of the Court One stand — prompting the unusual sight of the umpire asking spectators to stand in front of it.
Still the players battled on until the 28th game of a gruelling but absorbing contest; when Muller eked out a fifth match point, Nadal put a forehand long.
Muller, who was the first player to beat a raw Nadal at Wimbledon back in 2005, had repeated the trick 12 years later.
“That was tough,” said Muller. “In the last two match points I just said give it 100%.”
As well as completing a personal double, Muller added his name to those of Lukas Rosol, Steve Darcis, Nick Kyrgios, and Dustin Brown in sending Nadal home early in recent years.
The match went on so long that Novak Djokovic’s clash with Adrian Mannarino of France had to be postponed until today.
Defending champion Murray came through an awkward assignment against Frenchman Benoit Paire in straight sets, 7-6 (7/1) 6-4 6-4.
He will face America’s Sam Querrey, who won the battle of the big servers 5-7 7-6 (7/5) 6-3 6-7 (11/13) 6-3 against South Africa’s Kevin Anderson.
Federer had no problems in seeing off Bulgarian 13th seed Dimitrov 6-4 6-2 6-4 and will meet Milos Raonic, who beat him in last year’s semi-final.
Raonic, the sixth seed from Canada, also went the distance in getting past Alexander Zverev of Germany 4-6 7-5 4-6 7-5 6-1.
Muller will face Croatia’s Marin Cilic, a 6-2 6-2 6-2 winner against Spaniard Roberto Bautista Agut, while Czech Tomas Berdych awaits Djokovic or Mannarino after beating Dominic Thiem of Austria.