Born in Antrim, Northern Ireland on Christmas Day of 1749, James Dougherty belonged to the part of the O'Dougherty Clan that had converted to Presbyterianism in the 17th century. His mother was Scottish, descended from the Dissenter or Presbyterian Scots that were settled in Ulster after the decimation and expulsion of Catholic Irish. The history of the Presbyterians in Ireland was one of persecution by the English, and known as the Scotch-Irish, they emigrated to American in waves in the eighteenth century to with an abiding hatred of the British. It is not too much to say that if the English had not treated the Scotch-Irish so badly, the revolution of 1775 would have failed. The Scotch-Irish generally made up from forty to sixty percent of the Continental Army, furnished many of its most prominent officers, and were the backbone of the patriot movement.

James and his young wife immigrated to Pennsylvania in 1774 where he answered the call for troops following Lexington and Concord. He enlisted in the 1st Pennsylvania Rifle Regiment, Captain Matthew Smith's Company, in Lancaster on June 25, 1775. His company marched to join Washington's army besieging Boston, and after minor activity was selected to be part of Benedict Arnold's expedition to Quebec. James survived the horrible march through Northern Maine to Quebec, but was captured during the assault on December 31. He remained in irons for over four months, and was finally paroled with the rest of the prisoners in September, 1776 in New Jersey. James walked home penniless and in rags, but burning with desire to continue the fight.

Within two weeks James broke his parole and re-enlisted in the 12th Pennsylvania Regiment which joined Mifflin's Brigade in Philadelphia. The 12th was sent across the Delaware following Trenton and met the British along Assunpink Creek in the 2nd battle of Trenton. When the army moved around Cornwallis to Princeton, James was in the charge that scattered Mawhood's British regiment under the direct command of Washington. Following the winter encampment at Morristown, James was involved in two battles in Jew Jersey before transferring to the artillery. As a member of Captain Clarks's independent artillery company, he fought at Brandywine and Germantown.

While at Valley Forge, James was tapped for Washington's Lifeguard, more properly termed The Commander-in-Chief's Guard, and was one of the hundred men in the model company trained by von Steuben. He fought at Monmouth in the detachment of the Guard sent to Daniel Morgan, and later at Connecticut Farms and Kingsbridge where the Guard took heavy casualties defending Washington's person. In 1779 James went on detached service in the Susquehanna River Valley to fight Indians under Captains Brady and Boone, almost losing his life at the battle of Warrior Run.

Back with the Guard, James was at Yorktown and continued to serve until the peace in 1783. He spent more than eight years in the Continental Army and put his marriage and family on hold the entire time to win freedom for his country. For all that he eventually received a small grant of land and pension of nine dollars per month.