Is that correct?
No; after the Fatmids.
Wikipedia to the rescue
Later on, Al-Nasir the Abbasid draped the Kaaba with green, both Al Nasir and Al-Ma'mun disagreed on the frequent color changes and switched to black, and black it remains to this day. Kiswah
Al Nasir ruled from 1180; the Fatmids appear to be 909 to 1171.