In a Facebook post, Khumalo called for black South Africans to do to white people what "Hitler did to the Jews" and cleanse the country of white people.
The Sports Promotion Officer was served with an official suspension letter on Friday morning, said department spokesperson Nomazwe Ntlokwane.
Disciplinary action was expected to follow.
"Mr Khumalo has been suspended as from today and due process will be taken. We regard the allegations against him as a serious matter," Ntlokwane said.
"His pay has not been suspended," Ntlokwane said.
In the Facebook post, Khumalo said: "We must act as Hitler did to the Jews. I don’t believe any more that the is a large number of not so racist whit people . I’m starting to be sceptical even of those within our Movement the ANC. I will from today unfriend all white people I have as friends from today u must be put under the same blanket as any other racist white because secretly u all are a bunch of racist fuck heads. as we have already seen [all sic]."
At first, Khumalo was unapologetic about his post, but on Thursday, in an e-mail to News24, he begged for forgiveness, adding that he hoped the people of South Africa would find it in their hearts to forgive him.
"I would like to apologiese [sic] to the Gauteng government for my emotional comments that I made on a public platform. I further want to apologise to the ANC for the comments I made that do not reflect the ideologies of a democratic society that are our ideals," he wrote.
He admitted that his comments were offensive and "uncomfortable".
"I have seen firsthand what racism and oppression of any race can do to a country and would not want my son to grow up in such a society," Khumalo said.
News24 earlier contacted Khumalo for comment on his suspension.
After questioning how they had tracked him down, he declined to comment.
"I will not comment on an internal issue. Contact the department spokesperson if you want a comment," he said.
Meanwhile, Ntlokwane could not immediately say how long it would take to wrap up the matter against Khumalo