Colin Stewart IV was born and spent his childhood in Hanover, New Hampshire. He was the son of renowned Mayo Clinic doctor, Colin Stewart, III, who later relocated to Dartmouth Medical School. Colin Stewart IV attended Dartmouth, graduating in 1948. Stewart also served in World War II, stationed in Garmisch-Partenkirchen, Germany after the war. In 1949 Stewart won the Eastern Amateur slalom title and the Kandahar downhill in Mont Tremblant, Québec. He later received a graduate degree in architecture from Harvard in 1955, and after marrying Ann Edwards in 1957, the family moved to Colorado in 1959 where he designed private homes and public buildings throughout the state.