Exhaust Pipe and Turbo

I have a IH 175b and hoping someone can solve a debate we are having. The exhaust pipe is about an inch larger than the pipe coming out of the manifold, loose and flops around, something the previous owner did with what was handy to keep exhaust out of his face. My question is, would this affect the back pressure the turbo needs to spool up at all? Would it impact power?

Thanks for any insights.