Should He Take My 8wk old baby?!!?

Okay so I have an 8wk old daughter and I am not with the father bc he bailed a wk after we found out i was pregnant.he chose not to be a part of the pregnancy until the very end at which point he chose to cause alot of drama and emotional distress. I did allow him to sign the birth certificate and told him that he is welcome to come see her any time he wants but he has seen her maybe a total of 10 times in the 50 some odd days shes been alive by his choice. since the time i had her he has bought 1pack of diapers and some baby binadryl. thats all. I do not get child support or ask anything of him. because i feel that if he really wanted to be a father he would be and that causes problems between me and my family because they dont agree. I have never tried to keep her from him in any way I have even visited his house to let her spend time with his family... but he gets mad at me because i will not allow him to take her anywhere. Am I wrong??