Infantry is by far the weakest unit.
When the militia-line is a gold unit but feels like a waste of gold its a problem.
But a big problem with it is that if you go infantry you screw yourself on food early (most important resource until mid- late castle) and your units are extremely underpowered in castle.
A unit that slows your economy a bit while simultaneously has power drops throughout the game isnt worth it.
So lets make it better.
Tl;dr Balance
Well it is a classic unit shared by pretty much all the civilizations, so it’s normal to put effort to make it viable
Because 90% of games are cav + arbs
Better inf gives more variety