Illustration from Benson Lossing's The Hudson
In the fields below Sugar Loaf Mountain in Garrison stood a great house owned by Beverly Robinson. The mansion is described in The History of St. Philip's Church by E. Clowes Chorley as "a wooden house lined with brick". Robinson was one of the most prominent citizens in the Highlands, his land extended to 60,000 acres covering parts of what is now Garrison, Philipstown, Putnam Valley, Kent and Patterson. He was the first Church Warden and principal benefactor of St. Philip's Chapel. He had married Susannah Philipse, daughter of Frederick Philipse who was a nephew and heir to Adolph Philipse, the holder of the original land grant from William III.
Early in the Revolutionary War, Robinson declared himself a loyalist by leading the attack upon Fort Clinton after British Colonel Mungo Campbell was killed. Robinson had the advantage of knowing the countryside around the Highlands forts because he had hunted there as a young man. Robinson's house and land was confiscated by the American forces.
Later in 1780 when Benedict Arnold was given command of West Point by General Washington, he lived in the Robinson house which was known as "Beverly". Arnold had distinguished himself at several battles early in the war and despite being criticized for spending too much time with loyalists while he was stationed in Philadelphia, he was still admired by Washington. He had by that time fallen in love and married Peggy Shippen who was herself a loyalist. Either because of his wife's sentiments or because of his wounded pride in having his conduct criticized or because the British had offered him money which he needed, he decided to sell out his country. While in command of West Point, he sent troops off in search of wood and supplies to weaken the forces. He planned to have the great chain taken down for repairs. He met with John Andre, a British spy, and gave him the plans to the forts at West Point. Andre set off for New York where he was intercepted. The soldiers who met him found the plans to West Point in his boots and letters incriminating Arnold.
Back at the Robinson House, the morning after Andre's capture, Arnold was due to have breakfast with General Washington and Lafayette. Washington was delayed and by mid-morning a letter arrived warning Arnold that his treason had been discovered. He ran out of the house down the path to where a boat was waiting to ferry him across to West Point. He ordered his men to take him instead to a British ship. And so Benedict Arnold escaped and went to live in England.
The Robinson house was lived in by other families in succeeding years but it burned in the1890's.