I used to have this problem since I almost always play flank.
The first I did was learn a castle drop Build Order (I think it’s 26+2) and then depending on the civ match up I can do a castle drop myself or have a defensive one.
I like using Britons because the longbowman is an excellent unit to pick off vils as they come forward or if you do a castle drop yourself you can annoy their vils from a long distance.
At my Elo (13xx) the more common strat now is towers in feudal so I just wall up within my walls to negate the tower rush and then use my stone for towers of my own or a defensive Castle.
Another important action is to have your scout patrol the area where the Castle drop would happen or have a watch tower.
TLDR: Collect stone, Scout the area, Be ready.