Grandy was born in Sioux City, Iowa, the youngest of three sons of William Grandy, who worked in his father's insurance business, and his wife Bonnie. When Grandy was eleven, his father died of a heart attack. His mother died of an aneurysm a year later.

The young Grandy was raised by Margaret Avery, his mother's best friend (a widow who later married his father's doctor). He went to public schools until ninth grade, after which he attended high school at Phillips Exeter Academy in Exeter, New Hampshire, as his father and brothers had done.