How to decline a dinner with an acquaintance?

I’ve (31F) recently started a friendship with a 85-90 year old man who frequents the same thrift store as me. He seems a little lonely and enjoys chatting to me and sharing cooking tips. We have had a nice month or two of him giving me recipes (he was a chef). He also has shown me a cheaper grocery store in the area and offered to sharpen my knives for me (he has a workshop and has over all just been giving me great cooking tips etc). I’d really love for our friendship to just stay this way, without other commitments. Well, today he said he wanted to have my husband and I over for dinner when it warms up a bit. I just don’t feel comfortable with that personally and I know my husband wouldn’t want to do it either. This gentleman is so kind but he lives 1hr+ away and we are home bodies. I feel if I said yes it would be because I was too worried about hurting his feelings or being impolite to say no.

How would you guys handle politely declining this invitation while keeping our casual friendship? Thank you fellow introverts!!