Conservative Party chairman Brandon Lewis ordered him to apologise for the remarks and the demand was backed by the Prime Minister.
'I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes'.
He has been criticised by Labour MPs, some Tories and Muslim groups, who said the party was not doing enough to tackle prejudice.
Culture Secretary Jeremy Wright joined the chorus of criticism and former Cabinet minister Lord Pickles said that, while it would be a "very big leap" to suggest Mr Johnson could be thrown out of the party, "you never know how these things develop".
Lord Pickles called for Mr Johnson to apologise, but dismissed suggestions that is comments were comparable to Enoch Powell's Rivers of Blood speech.
However, an ally of Mr Johnson insisted he would not be apologising, warning: "If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists".
She said: "I've written to @PennyMordaunt and @BrandonLewis to ask what action will be taken against Boris Johnson for his ugly and naked Islamophobia".
He said: "With the scale of Tommy Robinson demonstrations, the storming of Bookmarks bookshop, and now Boris Johnson's Islamophobic comments, we can no longer ignore the rise of far-right politics in our society".
He revealed on Tuesday he had asked Mr Johnson to apologise.
He also said it made its wearers look like letter boxes or bank robbers.
Mobile back to price competition in 'unlimited'
The savings get a little better when you add two lines of Essentials , which costs $90 ($106 with the nationwide average tax). Now customers need to add taxes and fees to an advertised price we thought T-Mobile was past this .
The controversy sprung from a piece Mr Johnson wrote for the Daily Telegraph on Monday, in which he argued against following Denmark's lead in banning the burka.
However, Johnson said he did not support a blanket ban on the face veil in the UK.
'If you say that it is weird and bullying to expect women to cover their faces, then I totally agree - and I would add that I can find no scriptural authority for the practice in the Koran.
The resignation solidified Mr Johnson's position as a leader of the pro-Brexit wing of the Conservative Party, which is deeply divided over its attitude to the EU.
Fiyaz Mughal, the founder of Tell Mama, which campaigns against anti-Muslim violence, said Johnson's comments "clearly" amounted to Islamophobia.
Denmark last week followed France, Germany, Austria and Belgium in banning face-covering garments such as the burka and niqab in public places.
"It's up to a women how they should dress".
"A single case of abuse is one too many, and since becoming chairman I have taken a zero-tolerance approach", he said.
"Muslim women should not be a useful political battleground for Old Etonians", she told Channel 4 News, referring to the private school popular with Britain's political elite.