Centuries ago, in England, lots of cheeses had a natural yellowish-orange pigment. The cheese came from the milk of certain breeds of cows, such as Jersey and Guernsey. The cream in their milk tends to be richer in color from beta-carotene in the grass and flowers they eat during certain times of the year. When the orange pigment transferred to the cheese, it was considered a mark of quality.
Back in the 17th century, many English cheesemakers realized that they could make more money if they skimmed off the cream — to sell it separately or make butter from it. But in doing so, most of the color was lost.
To pass off the low-quality white cheese that was left over as a high-quality product, the cheesemakers added annatto, a yellow dye made from the seeds of a South American plant, to fake it.