The effectiveness of U.S. sanctions is being called into question as Iran and its oil buyers test whether penalties are worth the risk amid diplomatic negotiations.