rubs against my moral grain or my feeling of being a good person who tries hard for people I love

Being a good person can include other things, such as anticipating and avoiding boundaries in yourself and other people, limiting time and space in a way that's safer, even if it's less exciting. You can't say yes to what's working for you right now without saying no to the rest. That's ok.