This octagonal-based tower, formerly the keep, is the only vestige that reminds us of the existence of a castle-fortress now long gone, whose origin dates back to the 11th century, although this tower dates back to the 15th century. Its silhouette dominates Vilalba, capital of the region known as Terra Chá. It is known that the ancient castle, which has been successively remodelled since its origin, already had three towers and a polygonal wall in the 13th century. It belonged to the Lords of Lemos, but later King Pedro I donated it to Fernán Pérez de Andrade o Boo, thus becoming part of the patrimony of this noble family, who left their heraldic symbols on the building, such as the stone representation of the wild boar. This first Andrade bequeathed the castle to his brother, who passed it on to his son Nuno Freyre de Andrade; it was with him that the fortress suffered the first revolt of the Irmandiños, but the building withstood the attack. Once again, late in the second half of the 15th century and with Fernán Pérez de Andrade o Mozo as owner, the second revolt of the Irmandiños resulted in the destruction of the castle. After their defeat, the son of o Mozo, Diego de Andrade, ordered the prisoners to rebuild the building. It was at this time that the octagonal shape of the tower was created with a defensive and strategic objective: to make it more resistant to the attacks of a revolutionary and successful instrument of war: gunpowder.
At the end of the 15th century, the Catholic Monarchs granted Diego de Andrade the title of Count of Vilalba. This title would then pass on to his son, Fernando de Andrade. Unfortunately, the Count's actions aroused the suspicion of Ferdinand the Catholic concerning his loyalty, which led the King to confiscate the fortress at the beginning of the 16th century. In the years that followed, the building was not put to any use and was eventually abandoned. Meanwhile, with the passing of time and many a marriage alliance, the county of Vilalba was incorporated as another title into the House of Lemos and, from there, into the all-powerful House of Alba.
When the First Carlist War broke out, the building regained its function and was used as a defensive bastion; to this end, a very important reconstruction and reinforcement work was carried out. Vilalba suffered the first Carlist assault in 1835 followed by two more. The then Duke of Alba claimed payment for the damages from the Town Hall, a claim that was denied and finally withdrawn. Towards the end of the 19th century and with the end of the Carlist conflicts, it was once again abandoned. It was even used as a public quarry. This caused innumerable damages: the lesser towers, the entrance gate and part of the walls of the old castle, which were still standing at that time, were lost. In the end, only the keep, which can still be seen today, was able to be maintained and, fortunately, was spared from being sold as a quarry by its then-owner, who was no longer part of the House of Alba. The defence campaign carried out by a group of Galician intellectuals saved it from disappearing but could not prevent the gradual deterioration caused by neglect.
In the 1960s, the tower was restored and converted into a Parador Nacional de Turismo (A state-owned hotel), for which other facilities were built on adjoining land. This intervention made possible both the recovery of this heritage element and its conservation.
In 1994, it was included in the Catalogue of Monuments of Spain's Historical Heritage, as an Asset of Cultural Interest (BIC).