Together with his team-mate Alfred Sageder, Josef Kloimstein was one of the few rowers who participated in two different boat classes at the same Olympics. The pair won the bronze medal in the coxless pairs and reached the semi-finals in the coxed pairs at the 1956 Melbourne Olympics. At Rome in 1960 the duo won the silver medal in the coxless pairs and at Tokyo 1964 they came eighth in the coxed pairs.
Following an accident at work, Kloimstein lost part of a foot and then decided to start rowing. He started his career at the age of 23 and was also active in many other sports, including cycling and skiing. In addition to his two Olympic medals, Klomstein won 24 Austrian rowing championships in different boat classes, and with Sageder won four medals at the European Championships; three in the coxless pairs (silver in 1956 and 1957, bronze in 1959), and a coxed pairs bronze in 1956 with Franz König as their cox.