Should I just give up?

So i am a shy person. There is this guy I have liked for e very long time and just as of late have gotten the courage to talk to im a few times...3 or 4 times briefly. I hear that he is shy so i expected him to be a little timid around me at first. I feel like its not reciprocal, i mean, the initiation of the conversation. During conversation he is somewhat quiet but still seemingly interested in the conversation and enjoying it to a degree based on his body language. I am getting tired of being the only one putting forth effort, should i just ask him out and see what he says? Shulld i wait longer and keep it up? Or should i just give it up? Keep in mind i really lke this guy.