The 2004 handball matches were held at the Kleisto Gipedo, or the Olympic Indoor Arena, at the Hellinikon Olympic Complex; and at the Kleisto Gymnastirio Falirou, or the Faliro Sports Pavilion Arena, at the Faliro Coastal Zone Olympic Complex, with the final rounds held at the Olympic Indoor Arena. The Faliro Sports Pavilion Arena was finished in December 2003, and had seating for 5,800 spectators during the Athina Olympics. It was built in Palaio Faliro, a seaside town just south of Athina. It now serves as a site for entertainment and music concerts. The Olympic Indoor Arena was part of the Hellinikon Olympic Complex which was built on the former site of the Athina International Airport. Since the 2004 Olympics it has served as the home arena for several teams in the Greek Basketball League.
The format for men was two groups of six teams each that played a round-robin tournament to advance four teams from each group to the single-elimination brackets, beginning with the quarter-finals. The women had a similar format, but with only 10 teams competing, the round-robin groups had only five teams each.