Sources once opened up about how Ben Affleck responded to John Miller being so close with Jennifer Garner and their kids.