Spiritual energy is a lot like physical energy, you generate it constantly depending on how powerful your body and soul is. I would not worry about someone maybe taking some energy in the past.
I would recommend not giving them your time and attention because when you do you send them energy.
I know that on the witchcraft page of JoS there is a spell to sever Karmic bonds. You may want to do that if you’re worried, if you can’t find it I’ll link it when I’m at my computer and have more time.