The Olympics don’t need a purpose-built venue to feel special - the music, the decorations, the atmosphere, the ceremony - they transform the event. The Olympics at Santa Anita in 1984 were not at all diminished by the existence of the Santa Anita Horse Show the year before, I promise. If anything, having a well-worn venue made it a much better experience than something temporary and ephemeral.
The amazing buildout at the Kentucky Horse Park for the 2010 World Equestrian Games is still paying dividends.