Her family connections among distinguished Virginians had shaped her life. She was born at Culpeper Court House, only child of Elizabeth Hansbrough and William Lewis Herndon, U.S.N. They moved to Washington, D.C., when he was assigned to help his brother-in-law Lt. Matthew Fontaine Maury establish the Naval Observatory. While Ellen was still just a girl her beautiful contralto voice attracted attention; she joined the choir at St. John's Episcopal Church on Lafayette Square.

Then her father assumed command of a mail steamer operating from New York; and in 1856 a cousin introduced her to "Chet" Arthur, who was establishing a law practice in the city. By 1857 they were engaged. In a birthday letter that year he reminded her of "the soft, moonlight nights of June, a year ago...happy, happy days at Saratoga--the golden, fleeting hours at Lake George." He wished he could hear her singing.

That same year her father died a hero's death at sea, going down with his ship in a gale off Cape Hatteras. The marriage did not take place until October 1859; and a son named for Commander Herndon died when only two. But another boy was born in 1864 and a girl, named for her mother, in 1871. Arthur's career brought the family an increasing prosperity; they decorated their home in the latest fashion and entertained prominent friends with elegance. At Christmas there were jewels from Tiffany for Nell, the finest toys for the children.

At the White House, Arthur would not give anyone the place that would have been his wife's. He asked his sister Mary (Mrs. John E. McElroy) to assume certain social duties and help care for his daughter. He presented a stained-glass window to St. John's Church in his wife's memory; it depicted angels of the Resurrection, and at his special request it was placed in the south transept so that he could see it at night from the White House with the lights of the church shining through.