The Association chief executive Perry Crosswhite said abandoning the Games had not been discussed by his nation.
"No, not at all. I think you have to be very careful to make that kind of decision because what you are essentially doing is making decision on behalf of 400 athletes," he said.
"The reality is, they are the ones the Games are for and in some respects they are the ones that have to make the decision about whether they want to come here or not. You certainly are in a situation where health and safety and all those issues need to be met," Crosswhite was quoted as saying by Australian Associated Press.
"But if they are - and they are - of an acceptable standard, then the Games are for the athletes and they are the ones that should come along and make those decisions. They can have their views and all the rest of it, but I don't think we certainly have the right to make their decision unless it's based on something that we are trying to protect them from a security or health issue," he added.
Crosswhite said he had spoken spoke to Australian officials from 17 sports at the Games and no other athletes had indicated any plans to withdraw.
"None of them (Australian officials) said to me that they had any issues in respect of athletes advising them that they had similar concerns to Dani Samuels," Crosswhite said.
"All athletes, particularly with what they see in the media in Australia, they have every right to be concerned because they don't see any of the better side of it which they will certainly see when they get here - but they're not here yet," he said.
The ACGA boss said "Australia's accommodation at Athletes Village was acceptable" though he acknowledged displeasure of other nations with their facilities in separate blocks.
"Some people have let themselves down, it might have been the developers of the village. There are things that need to be corrected. I'm comfortable as long as the hard work that has started now is maintained and probably increases.
"There are a number of defects that need to be fixed, but it's not like buildings are going to fall down or anything. We are in three tower blocks and in the apartments, a few of those had things that needed to be fixed and they are being fixed now," he said.
Crosswhite said heavy security around the village was an issue -- not for any lack of safety, but for expediency.
"The security, which is really tight, needs to be bedded down to some extent. It's friendly, but it's just massive and it needs to be more, I guess the word would be, more efficient letting people through, in an out.
"Because when you get large numbers here, it is going to be a hassle if it doesn't get fixed." PTI
