Can I use XLR to connect pre-amp to amp and RCA to connect pre-amp to sub?

Yes. No problem. As long as both the xlr and rca outputs can be driven at the same time from your pre.

What preamp?
Where you connect the external high-pass filter to? Is your preamp has a tape loop?

Does the preamp have sub out? Use whatever is supposed to connected there.

To clarify a bit more the Pre-amp I'm looking to buy is a Rotel RSP 1582. It has a full set of XLR outs (2 fronts, rears, center and two subs), as well as a full set of RCA outs. What I want to do is run all the non-sub speakers to two separate amps. One a McCormack DNA 225 Platinum for my fronts which are Vandersteen 5's (The reason it has to be XLR is because my MP5 high pass filters are XLR), my surrounds also XLR pre to a different amp (another Rotel). This leaves me with my Sub a Vandersteen V2W (I only use it for HT, when I run 2 Channel just the V5's from the pre to the McCormack), which needs and RCA as that is the input on the back of the Vandy V2W. My question being can I run all the speakers from pre to their respective amps using XLR and then use RCA output from the pre to the Vandersteen V2W (only connection being RCA) or will something weird happen? 

Alternatively should I use the XLR out for the sub and use a cable to convert XLR to RCA so I can connect to the V2W. 

The amp has both XLR and RCA outputs for everything and ,y understanding is it's bad to use both XLR and RCA outputs for the same exact out (or in for that matter). Example being running the fronts from both XLR and RCA. 


Other questions:
What's a tape loop?
Vandersteen V2W don't need or use highpass filters. I think you are thinking of the 2Wq, which I don't have. My V5's I do use the MP5 highpass. Preamp has outputs sub outputs for both XLR and RCA - so both

Looking at the manual it looks like you could do what you want run balanced XLR to all the amps but the sub amp and run unbalanced RCA to the sub just don't run both XLR and RCA to the same amp. 

@djones51 thank you. I think you're right on my first pass of manual I read it as a no, but looks like it's a no only if RCA and XLR are connected to same amp.

One other question I had is would it matter if the surrounds are connected via RCA to a seperate 5 channel amp and the fronts are connected via XLR to my two channel McCormack?

I guess I may have to play with the gains to make the volumes the same? Any other reason not to do this. For music I go two channel and only use Multi-channel for HT.