In c. 1200, the manor of Lytham, the 'Lidun' of the Domesday Book, passed to the see of Durham, which housed a Prior, a few monks and their servants in their small priory. At the dissolution of Durham in 1540, the house and lands were taken by the crown and let to a sitting tenant, a Thomas Dannet. By 1597, the Manor was owned by Sir Richard Molyneux, who in 1606 sold it for £4,300 to Cuthbert Clifton of Westby, a relative by marriage. He pulled most of the medieval buildings and constructed a substantial new house in the Jacobean style. Not without difficulty and penalty, the Catholic Cliftons survived the Civil War and the various rebellions and by the middle of the 18th Century, their Fylde estates were large and they were people of consequence in the county. It was a time of improvement and rebuilding: in 1752, Thomas Clifton commissioned John Carr of York to plan and build a new house.
Set in 78 acres (320,000 m) of parkland, it is the only Grade I building in Fylde Borough and the only Country House on the Fylde Coast. The house is symmetrical in design, both inside and out. The principal rooms are on the ground floor, an early example of the break from Palladian traditions when they had been on the first floor. The main rooms are decorated chiefly in the 'robust Rococo' of the York school, with joinery probably by Fishers of York and plaster work by Giuseppe Cortese.
The house was acquired by Guardian Royal Exchange, in 1963 and was subsequently purchased from them in 1997 by Lytham Town Trust. Lytham Town Trust leased the Hall, under a partnership agreement, to the Heritage Trust for the North West in 1998, for a period of 99 years.
It is a Grade I listed building, but is on the Heritage at Risk Register because its condition is considered to be only "fair".