This bothered me, too! I feel GA DID do everything he could to protect her these three years, the whole family sure didn't cooperate with LE! Also, he said over and over he loves his daughter. He was certainly a doting grandfather, as well.
I don't think he really caught on until 12weeks...