"If I wanted people to stay out, you'd know about it."

He cuts straight to that point, ignoring the preliminaries. This is fine. He's fine. It's all fine here. The Chief may not totally get it, but he's definitely picking up on Wash's discomfort. Wash usually talks more than this. Or at least, he usually talks more than this before he gets awkward. It's a pattern.