Normally the best thing to do is drop a castle on their walls and shoot it down (can add a few vills to speed it up).
That not only takes the wall down, it also prevents repairs, it pushes your opponent into the corner, takes part of their base, denies at least one of their stone/gold mines and gives you an outpost to attack from.
Alternatively, some builds let you build your castle at home and make a few power units, for example with organ guns.
In that case you can add a siege workshop and one ram to take the walls.
You need those power units to protect the ram and take down repairs though.
Later you can make mangonels from the workshop to do more damage.
If you want to take down a castle, you should go up to imp and make trebs, instead of wasting resources and getting stuck in castle age 99% of the time.