The main form of defense for a castle is its own attack power.
It can cause a lot of destruction and has a large reach.
If you send rams to try and take it down, they won’t be successful, however, other units, such as petards, can be extremely effective in taking it down.
To try and overcome the castle, you could drop a tower or two, station crossbows within them, and send a ram in from a different direction.
This might be enough to defeat the enemy’s castle with the help of longswords.
The only thing that could stop this would be the use of mangonels or stored petards.
Additionally, the Korean Castle Tactics and the Bohemian Unique Unit provide an ideal platform to fight under castle fire.