In 1354, Amédée 6th infeoffed the Allymes fiefdom to one of his treasurers, Nicod Francois. Nicod Francois was one of his many creditors, but still had to pay a sum of 2.000 gold florins to obtain the right of being the Lord of the domain. The count kept for himself the right to take the fortress back if needed, which effectively happened between 1362 and 1364. But in reality, the former fortress became a rather peaceful place, seat of a small fiefdom. Over time, its architecture seemed to present more and more residential characteristics. We do not have any archives on the buildings added at that time, and hardly anything remains of the main lodging house probably built then.
In 1477, Claudine François wedded Humbert de Lucinge, of a more important lineage, and brought the fiefdom to his family. King of France François 1st annexed the French-speaking Savoy territories in 1536. Charles de Lucinge, Lord of the Allymes, remained faithfull to the Duke of Savoy, and devised several plots to try and take back the land. This forced him to flee from his lands in 1557. The Senate of Savoy, under the orders of the King of France, decided to punish him by “demolishing” his castle. The Duke of Savoy got his lands back as soon as 1559. The castle wasn’t completly destroyed at that time. However, letters from René de Lucinge (Charles’ son) confirm that some destructions took place, and restoration works were done in the late 1580’s. The wooden spiral staircase of the lodging house is still a testimony of those works: a dendrochronology revealed that its central axis, made of an oak bole, was cut during the spring of 1565. The recent archeological studies led during the restorations confirm this chronology for the restoration of the top of the square tower. Its wooden framework dates back, for the most part, to the late 16th century.
René de Lucinge was the most famous lord of the Allymes, since it was he who signed the Lyon treaty (January 17th 1601) for the Duke of Savoy. The document was negociated and signed by representatives of the King of France Henry 4th and of the Duke of Savoy Charles-Emmanuel 1st. It established the sovereignty of the King of France over the lands of Bresse, Bugey, Valromey and Pays de Gex.
René de Lucinge was famous in his time for his writings and historical reflexions, as well as for his diplomatic carrier. He fell into disgrace after the signature of the Lyon treaty, and his family lost the castle a few decades after his death.
In between late 17th century and middle 19th century, the castle had three different owners. These owners (the Suduyraud, Estienne and Dujast families) did not live in the castle but employed farmers who did. The castle had become a simple farm building. In 1793, during the French Revolution and at the peak of the destructions of churches and towers throughout the country, the castle was considered a symbole of feudal obscurantism and many people wanted it put down. Fortunately, that never happened.