I don't know what you're imagining, but it's wrong.

All that happened, was that I was invited to tea; and I naturally turned it down!

Besides, that was the first time I've ever spoken to him; I'm not acquainted with him!

...You got it? Well then don't get any weird ideas.

It's really not like that!