That's because it can also pay out in case you are disabled, so there is additional risk factors in there. Social security also needs to provide meaningful assistance at the time that you start taking from it, and needs to account for the inflation and changes in population demographics.
We pay more into it than the amount we get out as well, because the net benefit goes beyond the money you "get back" when you retire. I guess if you really don't want to pay for social security, you can take the loudest and most annoying of republican advice and "if you don't love it, leave it". Good luck, you're gonna need it. Otherwise, act like a functional adult and quit complaining about one of the few things we spend money on that actually helps people.