If you would like to specify a single color for all layers in the Reference Display Set, click the Select All button at the bottom left corner, then click the Color bar to open the Select Color dialog and specify the desired color.
*The button will show "No Change" for the color when all layers are
selected as some layers have a different color. You will still be able
to click the Color field and specify a new color for all layers.