DURHAM, N.C. — Herb Neubauer isn't throwing in the towel, he's just setting it aside.

"I've learned long ago all good things come to an end, just like Coach K's retirement," Neubauer said.

The 80-year-old is a big Duke fan. He's only missed one Duke basketball home game since 1984. And he's never missed a Final Four game until now.

"I'm one of only I think three people left alive that can say that they've been to them all," he said.

But he says he won't be at the NCAA Men's Basketball Tournament semifinal game in New Orleans on Saturday when Duke faces Carolina.

The Duke graduate is more than a mega fan — he's an icon. He's called "crazy towel guy."

Neubauer never imagined a way to wipe sweat off his forehead would lead to priceless memories.

"People with their sons and daughters that told me they still had a picture taken with me from late '80s to early '90s," Neubauer said.

His memorabilia is confined to his office per his wife's request.

She's going to be out of town and is the reason he won't be at the showdown on Saturday.

"If I went to the game, one of two things would happen: I'd either have my fourth heart attack or, No. 2, I'd kill someone," he said.

Of course, Neubauer is only joking. That's not to say emotions aren't running high for a rivalry game like this.

This time, Neubauer won't be in the stands. But you can bet your bottom dollar, he'll have his towel in hand.

"Little did I know that I would live this long to see this many [games]. So, thank you God, I'm happy," Neubauer said.