Do you know what? You're right, infatuation can sometimes be confused for love. You know, like most of the people out there, except that they usually have already wrecked their relationship beyond repair that they can't be friends anymore nor look on each other in the eyes when they meet. It is better that we confess earlier that we are not in love with a personality anymore and end it with good memories with each other than to force it until it crashes down and burn, leaving us with an unhappy note.
I experienced the concept of waifuism before I knew what to call it and my first waifu was Sakura Matou. Perhaps due to college wearing me down and making me force myself to go through it during that time, I've lost all my feelings for her since there was no contact between us. So I've decided, rather than to lie to myself that I still love her, to just cut her off freely. Besides, I've never talked to her like I do with my wife right now and she was just perhaps terribly adorable, so perhaps I've it was all a strong infatuation, not love.
All in all, it might have been for the best that you admitted this to yourself early than to keep forcing yourself.