By Martyn Herman

LONDON (Reuters) - Rafael Nadal's hopes of a third Wimbledon crown sank with the setting sun as Luxembourg's Gilles Muller returned to haunt the Spaniard with a nerve-jangling 6-3 6-4 3-6 4-6 15-13 victory in a fourth-round classic on Monday.

Twelve years after first getting the better of Nadal on the All England Club lawns -- since when his career has trundled along in unspectacular fashion -- 16th seed Muller stood firm in 135-minute fifth set of interminable tension.

Nine times in that heart-pounding decider, 31-year-old French Open champion Nadal served to stay alive.

Men of lesser resolve than the 34-year-old Muller would have cracked -- but the left-handed serve-and-volleyer was relentless in pursuit of victory and it was 15-times grand slam champion Nadal who succumbed when asked to walk the tightrope for the 10th time at 13-14.

A miss-hit forehand gave Muller two more match points and this time there was no escape for Nadal as another error off his trusty weapon ballooned over the baseline.

Muller stood motionless for a few seconds, taking in the enormity of his victory as the Court One crowd which has spent the past two hours on the edge of their seats, rose as one.

There was little doubt he deserved the ovation.

He outplayed Nadal in the opening two sets and then, after weathering a ferocious fightback, made the running in the fifth when his baseline craft matched Nadal's.

He struck 30 aces and 95 winners as he moved on to a quarter-final with Marin Cilic -- his second in majors after reaching the last eight at the 2008 U.S. Open.

"I'm just glad it's over," Muller, who was cheered on from the stands by Prince Felix of Luxembourg, said.

BIG BATTLE

"I did really well in the first two sets, then Rafa stepped it up. It was a big battle. When I had the last two match points, I thought I just had to give it a shot.

"Somehow in the end I made it."

Nadal has now failed to get past the fourth round since losing the 2011 final to Novak Djokovic, whose match against Adrian Mannarino was postponed to Tuesday because of the late-running drama taking place on Court One.

Since then Nadal has lost to the likes of Lukas Rosol, Steve Darcis and Nick Kyrgios but this year, having claimed a 10th French Open, the Spaniard arrived at Wimbledon in peak form and fitness and looked a good bet for the title.

Had he pulled this match out of the fire who knows where it would have taken Nadal. The crowd, despite loving an under-dog at Wimbledon, were firmly in his corner but even with their vocal backing he gave himself too much to do.

"I think I didn't play my best the first two sets, then I was always playing against the scoreboard. And that's so difficult against a player like him," Nadal told reporters.

"So well done to him. He played well. Especially in the fifth, he played a great game.

"It was a great atmosphere. I put everything on the court. I played with all my passion."

Muller, one of the few players on the tour who is happy to come to the net, harassed Nadal from the start and took the first set with a single break of serve in the sixth game.

Nadal made a mess of an attempted dropshot to give Muller a break at 4-4 in the second set and the man from Luxembourg moved two sets ahead when Nadal netted a backhand.

When Nadal broke early in the third though it signaled a shift in momentum that took the match to a deciding set in which logic suggested the Spaniard would dominate.

Muller had other ideas.

Nadal was forced to save two match points, one with an ace, at 4-5, then two more again at 9-10, the first with a steadfast volley after moving Muller around the court.

Muller came under fire at 6-6, saving a break point, and fought off four break points in an absorbing 18th game.

After 32 holds of serve something had to give and surprisingly it was Nadal.

(Reporting by Martyn Herman, editing by Pritha Sarkar)