I am not suitable to give recording advices since i dont do it myself but I didnt understand, when you were monitoring [from your interface to your sound system] you liked the sound but you dont like what was recorded finally ? This doesnt make sense. Dont know about interfaces, i just have a Scarlett 2i4 that i only use it for recording if i want to rip something, and i never monitor. Monitoring is done analog>digital>analog or just the analog input is 'splited' ? I bet the way it's done is the 1st mentioned, otherwise interfaces wouldnt mention latency. So if you didnt like monitor and recorded sound, you dont like your interface analog-digital convertion. But again your connections are weird. Why you use headphone output / mic input ? Your interface has RCA outputs and your system RCA inputs, right ? Try this way.
If anything else fails, try pedal in sound system, sound system's output to interface input.