Learn to discern between imagination and reality with the gut feeling. It takes practice. Personally, I believe this type of intuition is a gift from God for women.
I went with a friend to his house for lunch one day. We were both about 20 years old. His father showed up for lunch as well and we met for the first time. I greeted him with a handshake and smile, and then I got the creepiest feeling, like he was seeing me naked. Maybe he looked at me a tenth of a second too long. Or he didn't release the handshake soon enough. Anyway, I avoided being alone with him. Thirty-five years later I heard gross stories about him making passes at his wife's married friends. And another story that he molested his toddler grandchild while they were in the pool. I believe he died in prison, transferred to another state, after another molestation felony.
Busfield never made an impression on me, except to say I never sought him out in other shows and movies. I simply felt nothing for him. Zero attraction, which for that time in my life, was unusual.