In the years of the Livonian War (1558–1583) the castle was in the possession of the Russians who built a new line of fortifications around the castle. In the winter of 1574, the Swedes attempted to conquer the castle without any success. On 4 March 1581, the Swedish troops, led by Pontus de la Gardie, captured the castle from the Russians.
During the Polish-Swedish War of 1600–1629, the castle was partly blown up by the Polish in 1605, and later by the Swedes. The castle was left in ruins.
Today the castle belongs to the Museums of Virumaa, serving as a 16th century theme park.
In the beginning of 13th century there was wooden stronghold of Estonians called Tarvapää, in the place of the latter Rakvere Castle (Wesenberg). During the Crusades to Estonia, in the first half of 13th century Tarvanpää fell into Danish rule. In the middle of 13th century they started to build a stone castle.
The castle was under the Danish rule until 1346, when Estonian Duchy (Northern Estonia) was sold to the German Order. The next year it was sold to the Livonian Order and the bailiff of Rakvere started to reconstruct the castle to fit his needs as a convent house with large front yard.