When inaugurations became common place in the new capital city of Washington on the steps of the capitol, vice president-elects took their oath inside followed by a speech. The most well remembered vice presidential inaugural address was by Andrew Johnson on March 4, 1865. It is more remembered for the belief that Johnson 17 was drunk. He had been suffering a cold and took some whiskey to remedy himself, which led to his speech being slurred. Once outside for the presidential oath, president-elects would first give their address and then take the oath of office. With the passage of the 20th Amendment, things began to change.
Inauguration Day 1969, the outgoing and incoming administrations were comprised of former and present vice presidents. FROM L to R (the brief Johnson/Agnew administration): President Lyndon Johnson (36th POTUS and 37th VPOTUS), President-elect Richard Nixon (37th POTUS and 36th VPOTUS), Vice President Spiro Agnew (39th VPOTUS), and former Vice President Hubert Humphrey (38th VPOTUS)
Once January 20th became Inauguration Day in 1937, vice president-elects took their oath of office outside a few minutes prior to the presidential oath...but minus the address. Since then vice presidents take their oath and enjoy the day. The tradition of not having a set official to swear them in remained. Sometimes the Chief Justice swore in both men (still only men as of 2013), or the vice president chose an associate justice of their liking, or the out going vice president, or the president pro tempore or even the speaker of the House of Representatives administered the oath. The vice president even gets his own song after being sworn in, "Hail Columbia".
On January 21, 2013, the Chief Justice of the United States will once again swear-in the POTUS. Vice President Joe Biden will be sworn in by Associate Justice Sonia Sotomayor. Ruth Bader Ginsburg was the last female to swear-in a vice president, and that was Al Gore for a second term in 1997. Prior to that, Sandra Day O'Connor was the first woman to administer the Constitutional oath to a vice president, James Danforth Quayle in 1989. Sarah T. Hughes, a federal district judge in Texas, became the first to give the presidential oath in 1963 to Lyndon Johnson in Dallas.