But it would be selfish not to let her go, James said in a statement posted Wednesday on People magazine's website.
James was responding to an interview with Bullock on People's website in which she disclosed that she and her now-estranged husband adopted a newborn child several months ago.
The Oscar-winning actress has since split from James after revelations of his marital infidelity.
In his statement, James speaks of an unspecified addiction that he says has taken from him the child and his wife. He calls them "two of the things I love the most in life."