Which way do I lean...

There is no rule against it. Try it with tweets in or out to determine which sounds better.

Some Ohm speakers used to come with rotating labels in the grill corner indicating set things up your way as needed and that’s what people did.

not so with horns, at least the Heresy...

At this time I'm not trying to buy more speakers - although we will see how that works out for me.  I have Watkins Generation 4 on each side of my desk.  They sound best maybe 8-20 feet away.    I was not thinking of re-purposing them, although they are great about anywhere, I could use a smaller speaker on the desk like ProAC Tablettes (or look at ones you and others suggest), but will stick with the Watkins for now, most likely where they are.

It was really the Heresy I have some bookshelf space in another room for if they are sideways.  In that same space I could use the much smaller ML 35XT, or the 35XT sideways ear height on another shelf.  That may be my final config.  

Simply because the Heresy has a stand incorporated on bottom does not mean it can’t be turned to its side. C’mon, people, where’s your creativity?! ;)

I find it so refreshing, so interesting to turn speakers on their sides to create a Landscape orientation versus the typical Portrait orientation, that I have even done so with 4’ tall speakers such as the Daedalus Ulysses, Magneapn .7 and PureAudioProject Trio15 Horn1 (my favorite in Landscape orientation)!

It costs nothing, harms nothing, and is informative and fun! Go for it! While at it, try upside down too! Some speakers sound better inverted, even though the designer made them conversely. :)

Oh, yes, and play with the baffle slope, the angle of the front, to suit your taste. 

Is it "OK" to put speakers on their side?

Yes of course you can do it, but think of this the prototype speaker was voiced with hours/days of xover/stuffing even driver and maybe box size changes by the manufacturer to sound that way standing up, and then put into production.
Your going to be throwing away some of his hours spent voicing it, because it will sound different to what he wanted you to hear.

Now that I've made you feel real lousy for even thinking it, don't worry about him just do it.🤔

Cheers George

from Klipsch

"First introduced in 1957, the Heresy, a three-way design, started out as a compact center channel speaker to accompany the Klipschorn® in three-speaker stereo arrays. Because of its relatively "compact" size, this model offers the greatest degree of placement flexibility in the Heritage Series line."  

..........................

Now sold in Stereo Pairs, but, it's horns are still turned sideways to keep the cabinet shorter.

"The Heresy III offers the greatest degree of placement flexibility of all of the Heritage models due to its relatively compact size. It delivers the same basic frequency range as the La Scala but at more modest output levels. Corner, floor placement will deliver the highest degree of low bass reinforcement and the slant riser base helps to compensate for the Heresy’s short stature. This riser may be removed for shelf or cabinet placement. Self-stick rubber pads are supplied for attachment to the bottom of the  cabinet to prevent scratching of shelves and hard surfaces. "
They are not discussing laying it on it's side; just about removing the slant base and building it in, woofer at bottom.
.......................................

Tweeters at ear height, or, as the (any) slanted base achieves, aim the tweeter's center up toward seated ear height.

Tweeters, even horns, have narrower dispersion, so, if laid horizontally, tweeters typically go closer to the center (IF woofers not too close to corners/side walls).

Spherical cones dispersion patterns do not change (if no diffraction lens).

Horns are not symmetrical, they are most often designed for wide horizontal coverage, and limited vertical coverage to avoid floor and ceiling reflections.  The horns in the Heresy, originally intended as centers between a separate stereo pair, are mounted 'sideways' (keeping the cabinet compact) and therefore produce 'tall' vertical dispersion, narrower horizontal dispersion. The angled base minimizes the floor reflections, and sound gets to you before ceiling reflections get problematic. Projecting up some, eventual reflections off the back wall are angular in direction.
..............................................

so, Heresy laid sideways, the horns will give wider horizontal dispersion, narrower vertical: which is typical/preferred for stereo pairs, avoiding ceiling and floor reflections.
...................

Others have said, try it, cannot hurt, rig up something temporary before you spend money and cut wood!

My Magnepan MGCI is on it's side under the TV, about 20 inches off the floor.  It is only an inch or two wider than the TV.  It used for the center channel and sits about 6 inches from the wall.  Purpose, dialogue, works really well.  As far as sound anomalies, it is used in 5 or 7 channel mode.  Forwards, backwards, sideways and down, it cannot hurt to give it a try.  My side and back surround speakers are in the ceiling, this works well in a smaller space.

The Heresy stands are removable. Try them with the tweeters on the inside, and on the outside, and determine which is better to YOU. Use some rubber bumper pads between the Heresy and the shelf ( available on Amazon or the Bay ).