He, however, defended his action saying that he has the right to wear the symbol as a candidate.
Rai also claimed that the case against Narendra Modi cannot be compared with his action as the BJP Prime Ministerial candidate had shown his lotus symbol after casting vote in Gandhi Nagar.
On April 30, BJP's Prime Ministerial candidate had courted trouble after he flashed his party's symbol lotus while taking a "selfie", and made a speech after casting his vote in Gandhinagar in Gujarat.
Modi is also fighting Lok Sabha elections from Varanasi.
Another candidate Arvind Kejriwal of AAP also raised objections about Rai wearing 'hand' badge on the polling booth and said this was not fair.
BJP leaders also raised objections and sought action against Rai by the Election Commission.
BJP said Rai's act violated the Model Code of Conduct.
"In case of Narendra Modi, he was speaking to the media with a lotus symbol beyond a prohibited zone of 300-feet which is permissible. In case of Congress candidate in Kashi, he is inside the booth with a Congress symbol.
"Would the Election Commission take cognisance of it and take appropriate action or as in the case of Rahul Gandhi, who violated the model code of conduct by fiddling with a voting machine in Amethi, be let off on a fictitious basis? These are important issues," BJP spokesperson Ravishankar Prasad said in Delhi.
He said the EC should ensure that its approach does not give any indication of discrimination.
Another BJP spokesperson Prakash Javadekar called Rai's act as a "petty little issue".
"I don't want to give reaction on petty little issues. Let's see what the Election Commission does on the issue? Will Election Commission deal equally with all candidates or it goes soft on some people," Javadekar said.
Rai, who is at the centre of the allegations, said he was wearing the Congress party symbol in a "routine" manner.
"I am a Congress candidate. That (the symbol) is on my I-card too. Definitely I was not informed. I use it usually during campaigning. This is normal and routine. It is placed on my heart," Rai said.
