Sam went to live with his brother Emmett at the Luther household