Compromising in a Relationship

A framily member and I were discussing happy marriages and compromising.  Curious as to what some of you think of a particular scenario.

Being happily married means that both people feel like they always consider the other during times when compromise is necessary but...what if there is no clear compromise for one person?  What if the only thing that will make one person happy completely goes against what the other person wants?  How does that resolve itself in your opinion?  I mean sure...you'd want your spouse to be happy but never at the cost of you being unhappy right?

For example...

Someone wanted to build a house the size of a castle.

Their spouse did not.

They built the house the size of a castle.

Three years later, the castle was up for sale because they were getting a divorce.

They were getting a divorce because the spouse hated the house and that what they wanted was never taken into consideration while building the house.  And the cost of the house was so far outside of what they knew they could afford but the one spouse felt they could never say anything or the other spouse would be unhappy.

So yeah...that never really worked itself out.

Is it honestly selfish to really just want what you want because if you compromise you'll never truly get what you want?