I had the same concern, as it was my oldest card as well. Closing it did not affect me at all so I can say it’s not a big deal, specially if you have other cards. I had another secured card as I mentioned, and opened a new credit card account with a higher credit line.
Closed accounts in good standing will stay on your report for 10 years, so it’s all good!