The biggest problem, though, is losing your phone. Some services will let you provide a backup number, including a friend's cellphone or a landline phone. With Google, the code can be sent as a voice message instead of a text. Others offer a complex recovery code, which you'll have to jot down and keep in a safe place.
I know two-layer security is inconvenient. The first password is difficult enough to deal with. But think of the inconvenience involved should someone break into your account and shut you out. Consider the use of verification texts to be insurance.