Obeidullah Ansari, a young weaver from Pande Haveli area, said: "We have understood the game and won't fall into the trap this time."
The Aam Aadmi Party, whose leader Arvind Kejriwal has decided to contest against Modi from Varanasi, does not want to take on Modi on the communal issue.
"We don't believe in vote-bank politics and consider everyone as aam aadmi (common man)," AAP leader Rajiv Mishra told IANS.
"Our campaign is focusing on the pathetic living conditions of the people and grossly poor civic amenities - roads, public health, sanitation, education - in the constituency," he added.
The Samajwadi Party has decided not to field its candidate from Varanasi because, as its supremo Mulayam Singh Yadav had said recently, the party does not want to "divide secular votes".