In some ways, it's quite convenient for your wife's boyfriend to be living so close. It means that she can drop by his place easily whenever they're in the mood for sex -- or he can drop by your house, whichever is easiest at the moment. And you and he have become pretty good friends, now that you see quite a lot of each other. But it also means that everyone in the neighborhood knows about their relationship -- and know that when there are neighborhood events of any kind, she is very likely to attend as part of a "couple" with him instead of with you.