Fisher started competing in road and track races at age 12. He was suspended in 1968 because race organizers cited a rule that his hair was too long. By 1972 this rule had been repealed and Fisher's career continued. He won the TransAlp race in Europe and a Masters XC national title.

Fisher went to work in 1975 on his 1930s Schwinn Excelsior X bicycle. His innovations to the model included drum brakes, motorcycle brake levers and cables, and triple chainrings, all taken from "junkers" Fisher found at bike shops. The next year, Fisher participated in the Repack downhill race, promoted by his roommate Charlie Kelly. This used a tortuous downhill route on Pine Mountain near Fairfax, California, just north of San Francisco, in which riders used their coaster brakes so much that they had to repack the smoking hubs with grease after every run. Fisher holds the record time on the Repack course at 4:22.