Widow's Benefit Question

I applied for SS benefits at 62. That same year, my husband who also had started receiving his benefits at 62, passed away at 68.. Then my benefit amount was recalculated and included part of mine and part of his.
I continued to work part time for several more years, but instead of increasing my benefit amount, Social Security decreased the amount I was getting from his record and increased the amount from my record, so that my income always remained the same whether I contributed more into the system or not.
This is not fair and I wonder if there is any way it can be changed? I finally gave up working part time as it wasn't helping me financially at all. In fact, it just cost me more in taxes.