The legend of the temples goes that Lord Krishna had sent Pandavas to atone for their sins of Mahabharata war with a black flag and a cow with assurance, that when these two would turn white, their sins would be removed.
The Pandavas arrived at this place, where the flag and the cow became white in color.