The critics need to do better than that, or they need to just give up and go away. I don't know what else to say about it. It seems the reality is that he loves his wife and values his marriage. Is that wrong, just because so many in our culture have ignored and/or rejected such values? His standard in this matter is wise and Biblical. Criticism of such a standard reveals the heart of the critic.
"...do not let your good be spoken of as evil..." (Romans 14:16, NKJV). "...having a good conscience, that when they defame you as evildoers, those who revile your good conduct in Christ may be ashamed" (I Peter 3:16, NKJV). "...in all things showing yourself to be a pattern of good works; in doctrine showing integrity, reverence, incorruptibility, sound speech that cannot be condemned, that one who is an opponent may be ashamed, having nothing evil to say of you" (Titus 2:7-8, NKJV).