Whether it is hubby or anyone in family the suspician can ruin relationship. So while he has to improve but so also you have to build your image before him. One has to build confidence of spouse to remain united. The personal callers at odd hours , during week days are all nuisance for any one unless they are very close to whole family. So it is oness duty to introduce callers, they must meet each other. They must develop liking for each other and so the suspician will go. But if there is a person specially of opposite s*x who keeps relation only with individual and not with family is creating problem for that spouse and spouse will be doubted.
To my mind there is no personal thing in family. A reasonable space is given but inspite of that one must be like open book and must instill a confidence in family by their doings and specially husband and wife must be one and set an example for their children of their good behaviour and character.
Anything less then that can create havoc in family life leading to .....and finally divorce.