Your shoes have been making them the weakest.
Traditional court shoes do the work FOR your feet — thick foam, rigid arch support, cramped toe boxes. Over time, your foot muscles weaken because they never have to work. That's why you feel less stable every year, not more.