James Taylor studied at Balmain School and Sydney University, becoming a chartered accountant and later serving as the director of several companies. He was President of the Australian Swimming Union from 1909, President of the New South Wales Amateur Swimming Association from 1908, and was the founding Chairman of the Australian Olympic Federation (AOF) in 1915. Taylor was made an IOC Member in November 1924 and served until his death in 1944. In July 1928 Taylor attended a meeting in London along with representatives from Britain, Canada, South Africa and New Zealand. The meeting served to form the British Empire Sports Federation, which led to the first British Empire Games in Hamilton, Canada in 1930.