EfficientPC sell system units (i.e. without monitor, keyboard or mouse) which have been designed and tested to work with (and can come preinstalled with) Ubuntu Linux! They have a configuration builder (like the one you see on Dell’s website) that allows you to change the spec of the machine that you’re interested in, and the price updates on the fly! Prices are fairly decent too... a reasonably spec’d desktop PC for only £209.53 inc VAT.
They also sell a small quantity of components that has been tested with Linux, as well as other Linux machines like the Asus Eee PC.