Union Urban Development Minister M. Venkaiah Naidu Friday said RLD leader Ajit Singh's "threats" on being asked to vacate his government bungalow here "won't work".