I am confused with my relationship with my bf sometimes?

We have known each other for over a year. I have an almost 4 old (by now) son and he is still a single guy. We went thro the honeymoon stage then he slows down, asking to let him get used to having a child in the picture. I would love to see a man coming into my life, treating my son like his own. But I should and have to give him time to adjust. At the same time, the relationship seems moving forward steadily, like after the honeymoon stage, he has spent some weekends with all us together for a few months already and recently he is inviting my son into his family functions as well (I went a few times the past year on my own). The confusing part is that, sometimes I want to spend more time with my son, when I feel I haven't spent enough time or when he is not feeling well, then my bf would give me the "space" to stay with my son. I actually would love to see him come to us instead of giving me the "space". His "space" is a "distance" to me. I feels real bad when my son is sick and he is giving me the "space". He is so not involved. Is that what love supposed to be? I am confused. Even though he puts "distance" between us, but if I said I don't feel loved, he would be upset, saying I should never doubt the love between us. Someone please enlighten me. Thanks. btw, I won't waste my time for a casual relationship and hope a LTR/stable life for my son and myself, and he knows it and hope for it from the beginning.