Understood, that used to bother me too. After a while people realized I was firm and laid off. Other than a few occasional passive-aggressive comments from my mom about how she doesn’t have grandchildren, nobody really says anything anymore.

Edit: whoops, that posted 3 times!