How to delete a color from the Apple Color Picker custom swacth?

The custom color swatch drawer is a great way to store custom colors that I repeatedly use in projects.

But how to delete colors from it? Click and delete does not work, click and drag it away doesn't work either... 

Only way I found is to click and drag the pot of color to the Trashcan! Simple enough... When you know it.