It was widely believed that negotiators from the Taliban side - Samiul Haq, a radical cleric known as the ‘father of the Taliban', Jamaat-e-Islami leader Ibrahim Khan and Abdul Aziz had agreed to this condition.
There is intense scepticism about the peace initiative achieving a lasting solution to the insurgency led by the Tehrik-e-Taliban Pakistan that has claimed thousands of lives.
Aziz said he would remain part of the Taliban-nominated three-member team but would not come to the negotiating table.
Observers said his move could undermine the talks as the government committee has voiced doubts about the authority of the Taliban's representatives.
The state negotiators skipped an initial meeting on Tuesday, citing confusion about the Taliban's team after two members pulled out.
