That only applies for Christians. Other religions have different swearing-ins, for atheists it's simply "do you affirm to tell the truth under penalty of perjury" (or thereabouts).
The US was never founded in Christianity, most of that came about during the cold war when religion was seen as a good opponent to atheist Stalin and leaders figured it would fuel more hate toward our long-time-scapegoat-for-all-the-world's-evils, communism. That was even when "under God" was added to the pledge of allegiance. Most of the founding fathers weren't very religious, few of them regularly attended a church, and those that were religious made a conscious attempt not to mix their personal beliefs with the policies of the nation. The sporadic appearance of references to a god in early documents never explicitly say they're talking about a Christian god, they may merely say something generic like "the creator", and even those are few and far between.
The US not being a Christian country confuses most people because they're remembering the first settlers, who were largely religious zealots that setup theocracies in the New World, and mis-connecting them with the founding of the union, which took place a great deal later on in history. By the time of the revolution, those that wanted a church-state were very much in the minority, most everyone did not want religion to enter at all into politics, nor vise versa. Christianity wasn't a given, either. When forming the union it was made very clear America was not to be a Christian country, nor a country found on any other religion. Politics were to remain purely secular. Written in the treaty with Tripoli in 1797, the Muslim populous of that country were assured, quite plainly, "the Government of the United States is not, in any sense, founded on the Christian religion".
