You could try "I'd like to visit your city. If I do, would you like to meet up for dinner?" Unless the city is Basildon or Slough or somewhere similar, when this lacks plausibility :-)

Do you know what relative importance they attach to hassle-of-hosting/time/money/travelling? If you must use an explicit algorithm in the decision-making process you'll probably find you each weight the elements differently.

It's possible that the reason I don't feel comfortable with issuing the invitation either way round is precisely because I don't know what relative importance they attach to hosting and travelling. Hence offering them the choice.

I'm not entirely sure, though.