Pete Duel was born in Rochester, New York, the eldest of three children born to Dr. Ellsworth and Lillian Deuel (née Ellstrom). He had a younger brother, Geoffrey, who also became an actor, and a sister, Pamela.

He attended Penfield High School, where he worked on the yearbook staff, campaigned for student government, and was a member of the National Thespians Society.