The soccer venue was also the spot where Mandela made his last public appearance at the closing ceremony of the World Cup.
After the memorial, his body will lie in state for three days at the Union Buildings in Pretoria, once the seat of white power, before burial Sunday in his rural childhood village of Qunu in Eastern Cape Province.
Police promised tight security, locking down roads kilometers (miles) around the stadium.
However, the first crowds entered the stadium without being searched.
John Allen, a 48-year-old pastor from the U.S. state of Arkansas, said he once met Mandela at a shopping center in South Africa with his sons.
“He joked with my youngest and asked if he had voted for Bill Clinton,” Allen said.
“He just zeroed in on my 8-year-old for the three to five minutes we talked.”