With the political debate about Islam heating up ahead of Britain's general election next month, the impact is particularly keenly felt in Birmingham -- a city where a quarter of the population is Muslim.
"The politicians are using Muslims as a punch bag," said Abdul Rashid, head of the city's main mosque.
"The politicians look for cheap and easy ways to popularity... And at the moment the cheap and easy way of gaining popularity is immigration and Islamophobia," the 73-year-old told AFP.
Related: British Muslims raise concern over new anti-terror law
Birmingham is Britain's second biggest city with 1.2 million residents -- almost half of them from ethnic minorities -- and is often singled out when fears are expressed about a growing Islamist threat.
Following the militant attacks against the satirical weekly Charlie Hebdo in Paris in January, an analyst on the US network Fox News even spoke of Birmingham as a "no-go zone" for non-Muslims.
He since apologised for the comments, which hurt many inhabitants in the former industrial hub.
The first Muslim immigrants -- Yemenis -- arrived in Birmingham in the early 20th century to work in the metal industry. There have since been waves of arrivals, many of them from the Indian subcontinent but also from Afghanistan, Bosnia and Somalia.
The city is known now for a mostly peaceful coexistence between communities, although it was hit by a nationwide spate of riots in 2011 following the police killing of a mixed-race man in London.
Related: UK Muslims to get first national radio station
Birmingham is "a city where people feel that they can live together and tolerate each other," said Mashuq Ally, deputy head of a special local government department in charge of community cohesion.
Ally is in constant contact with the city's dozens of ethnic communities to avoid any disputes or violence.
"Although we are said to be the second most vulnerable city to terrorism after London, we are fairly safe because the community is part of the solution," he said, hailing cooperation between the city council, Muslim communities and the police.
Philippe Trzebiatowski, a young Frenchman working on a Birmingham travel guide for the company Expedia, said that "people mix quite naturally here".
Not everything is rosy however in Brummie-land -- the diminutive nickname used for city residents.
A scandal erupted last year when it emerged that radical Muslims were trying to take over the boards of several city schools, and many communities choose to live separate lives in a type of self-segregation.
According to the 2011 census, three areas have a Muslim population of more than 70 percent -- Washwood Heath, Bordesley Green and Sparkbrook.
Professor Carl Chinn, who heads up a community history project at the University of Birmingham, said the tendency to stick together does not apply just to Muslims in Birmingham but also the white working-class and happens in cities across Europe.
"That is an issue we need to address without hyperbole," he said in an interview with The Guardian.
Around 4,000 faithful come out for Friday prayers at Birmingham's main mosque.
As children from a local school arrived for a guided tour, Rashid said that in recent years all faiths in the city have made more of an effort to build ties.
Determined to fight against prejudice, he is encouraging Muslims to go and vote on May 7 to show they are active members of British society.
"There was a time when it was thought that all migrants will vote for Labour and I think that is gone," he said.
"As the community is maturing, they are considering other political parties, other opinions," he said.
Professor Matthew Francis, an expert in ethnic minority voting at the University of Birmingham, said that between 55 and 60 percent of voters of Asian origin in Britain usually vote for Labour.
For Conservatives, support among Muslims is only about 10 percent, while among Hindus and Sikhs it is slightly higher at 15 to 20 percent, he said.
"Part of the reason for the general hostility to the Conservatives is the longer legacy that they have of being far more vocally opposed to immigration and far more sceptical about multiculturalism," he said.