Pakistan was all out in the last over before lunch on day three but Sri Lanka didn't enforce the follow-on.
Pakistan resumed overnight on 48-5 at the Galle International Stadium but managed to avoid its lowest ever total against Sri Lanka of 90.
Sri Lanka's 472 was propelled by an unbeaten 199 by left-hander Kumar Sangakkara.
Pakistan's middle order batsman Mohammad Ayub, making his debut at age 32, scored 25 before being ruled leg before wicket to Herath, who finished on 3-30.
Herath also dismissed the other overnight batsman Younis Khan for 29, again with an lbw decision despite TV replays appearing to show the batsman had got a big inside edge.
Wicketkeeper Adnan Akmal was run out for nine after a mix up with Ayub when both batsmen were left stranded at the same end.
Randiv, who had picked up two wickets on day two, added two more to finish with 4-13.