In addition to calling the vote itself illegal, the Obama administration said there were "massive anomalies" in balloting that returned a 97 percent "yes" vote for joining Russia.
Obama warned that Russia could face more financial punishment.
"If Russia continues to interfere in Ukraine, we stand ready to impose further sanctions," Obama said.
One of the top Russian officials hit by sanctions mocked Obama.
"Comrade Obama, what should those who have neither accounts nor property abroad do? Have you not thought about it?" Deputy Prime Minister Dmitry Rogozin tweeted. "I think the decree of the President of the United States was written by some joker."
Moscow considers the vote legitimate and Russian President Vladimir Putin was to address both houses of parliament Tuesday on the Crimean situation.
In Kiev, acting President Oleksandr Turchynov vowed that Ukraine will not give up Crimea.
"We are ready for negotiations, but we will never resign ourselves to the annexation of our land," a somber-faced Turchynov said in a televised address to the nation.
"We will do everything in order to avoid war and the loss of human lives. We will be doing everything to solve the conflict through diplomatic means. But the military threat to our state is real."
The Crimean referendum could also encourage rising pro-Russian sentiment in Ukraine's east and lead to further divisions in this nation of 46 million.
A delegation of Crimean lawmakers was set to travel to Moscow on Monday for negotiations on how to proceed. Russian lawmakers have suggested that formally annexing Crimea is almost certain - with one saying it could happen within days.
"We came back home to Mother Russia. We came back home, Russia is our home," said Nikolay Drozdenko, a resident in Sevastopol, the key Crimean port where Russia leases a naval base from Ukraine.
