When looking at the architecture of Philipse Manor Hall, it is important to consider the house structure itself. The house was built in three stages by different generations of the Philipse family. Frederick Philipse I came to New Netherland as a carpenter/builder for the Dutch West India Company where he oversaw the building of churches and fortifications. Although the exact construction date of the oldest part of the house is unclear, it is likely work was begun sometime in the mid-1680s, possibly 1686, which would coincide with the arrival of the first enslaved people the Philipses imported from Africa on board their ship the Charles in 1685.
The original house faced south, overlooking the Nepperhan River and the mill complex. The original structure was almost certainly two stories tall, the large, symmetrical windows and hipped roof reflecting the Palladian style that was influencing Dutch (and English and French) architecture in the 1630s and '40s. Despite this, the original stone portion of the house was neither classically Palladian nor Georgian, despite its symmetry. The coursed rubble stone masonry walls, gambrel roofs, Dutch doors, and large windows are all indicative of Dutch vernacular architecture. This combination of styles places it firmly in the Anglo-Dutch period.
The basement was built first, with evidence of green glazed tile typical of Dutch storage areas. The basement also did not have a fireplace for heat or cooking, reinforcing the idea that it was likely used for storage rather than living space. Analysis of mortar in the the first and second floor walls suggested they were built at the same time, if not the same building season.
When Frederick I died in 1701, the Manor Hall passed to his grandson, Frederick Philipse II, who was just a child. Educated in England, Frederick II returned to New York in the 1710s. He came of age in 1716 and with his new wife Johanna they revamped the interior of the original Manor Hall. About that same time Frederick II began the construction of new addition to the house - essentially a reflection of the original two story stone structure built by his grandfather. Although nearly imperceptible to the naked eye, the addition "completed" the symmetrical look of the house, with the grand Dutch door now in the center, flanked by two banks of windows. Perhaps the most interesting thing about this addition is how little it reflected the Baroque styles of the time, and how closely it matched the original.
Frederick II was not done, however. In the 1740s, he began construction on yet another addition, this time in red brick, and changing the orientation of the house. The addition stretched north in an L shape and faced inland, toward the gardens and the Albany Post Road. Exterior porticos with Baroque-style classical columns were added. The fanlight windows over the doors were a popular feature in the 1730s. Dramatic interior changes also occurred, although the original window seats and Dutch doors were continued throughout the house, perhaps a nod to the family's long tenure on the site. Ornate classical carvings were added to fireplace mantels and woodwork.
Frederick Philipse II died in 1751, and his son Frederick Philipse III became lord of the manor, inheriting the entire Philipse Manor as well as the manor hall. Unlike his predecessors, Frederick III chose to live the life of an English country gentleman, and made the manor hall his full-time residence from the start.
Although it is unclear when the construction on the second addition was completed, it was likely nearly complete by the time Frederick III took ownership of the house. His influence can be seen in the French Rococo elements on the fireplace and the papier mâché ceiling of the southwest room.
Following the Philipse family's departure for England at the end of the American Revolution (they were staunch Loyalists), the manor hall changed private hands several times, including residence in the 1840s by Thomas Cornell, who drew maps of the house and grounds and recorded other information. In 1863 it became Yonkers Village Hall and later Yonkers City Hall. During its tenure as City Hall, the second floor of the second addition was extensively remodeled - removing the attics and rooms to become the Gothic Chamber. When that happened, the first floor fireplace ceased to function.
When the manor hall was rescued from demolition for a new city hall and became a historic site in 1911. At that time, the American Scenic and Historic Preservation Society undertook a restoration of the house, which returned many of its architectural elements to a Dutch style (accurate or not) and removed partitions installed during the City Hall era.