BALTIMORE — The portrait used to hang in the hallway, welcoming children and parents to the Archbishop Borders School in Baltimore: a smiling Dr. Ben Carson in surgical scrubs, rubbing together the careful, steady hands that helped him become the nation’s most famous black doctor.

“The person who has the most to do with your success is you,” it reads.

That was before Carson’s presidential bid, before he withdrew from the race and endorsed Donald Trump, and before he was tapped to run the Department of Housing and Urban Development. It was before the president failed to condemn white supremacists who marched in Charlottesville, Virginia. And before Carson pushed policies critics say walk back civil rights protections for those living in subsidized housing.

“I took it down,” said Principal Alicia Freeman of the portrait she’s since moved from the school’s second floor hallway to a less visible spot inside a reading room bearing Carson’s name. The doctor’s inspirational message now feels hostile, she said

“He was starting to become offensive.”

Carson’s story of climbing out of poverty to become a world-renowned surgeon was once ubiquitous in Baltimore, where Carson made his name. In some schools his memoir was required reading, an illustration of the power of perseverance. For a working-class, majority African-American city wracked by racial division and neglect — where a baby born in a wealthy white neighborhood is expected to live two decades longer than one in a poor black area — Ben Carson was hope.

But his role in the Trump administration has added a complicated epilogue, leaving many who admired him feeling betrayed, unable to separate him from the politics of a president widely rejected by African-Americans here. In the last presidential election, nearly 85 percent of city voters cast ballots for Hillary Clinton.

“The Trump virus is weakening Ben Carson’s image,” said Bishop Frank Reid, a former pastor at Baltimore’s Bethel AME Church who met Carson at Yale, where both received their bachelor’s degrees. Carson is still respected, Reid said. “But he is no longer the hero he once was.”

Carson declined to be interviewed for this story. Instead, he sent a written statement.

“I understand what it means to be poor because I grew up poor,” the statement said. “I was fortunate to have my mother who was my compass – always steering me on course, helping me to see beyond our circumstances. That’s what I hope to do for the millions of low-income families HUD serves.”

Carson was born in Detroit, but Baltimore is the city that claims him. He rose to fame for his groundbreaking surgeries at Johns Hopkins Hospital, and launched his scholarship program here. Carson would sometimes arrange for high school students to visit the National Great Blacks in Wax Museum on the city’s east side, where a figure in his likeness stands glossy and smiling in a white lab coat and stethoscope, in the middle of the Famous Marylanders display.

“This young African-American who grew up in poverty and could have been put in jail or suspended from school made something great of himself. It was the American Dream,” Reid said. “We believed he could walk on water.”

Emmanuel Williams, 33, grew up in Northwest Baltimore. He learned about Carson in elementary school, a few years after Carson grabbed headlines for successfully separating conjoined twins attached at the skull.

“He was taught during Black History Month,” Williams said. “And everyone was so proud because it was happening here in Baltimore. It was mythic.”

“Sometimes I think the country looks down on us,” he said. “So to have such a brilliant person who’s making history and making these great medical advancements in Baltimore? He was our crown jewel, and he was here.”

Since taking the reins at HUD, Carson has proposed sweeping rent increases for the poorest subsidized housing tenants, and has begun dismantling key Obama-era regulations designed to address racial segregation. Carson has considered stripping anti-discrimination language from the department’s mission statement, and voiced strong support for implementing work requirements for housing aid recipients. In a radio interview shortly after being confirmed, Carson said poverty “is a state of mind.”

Now, Williams said, “people feel betrayed.”

“He can’t come back from this,” he said.

The seeds of Carson’s approach to policy are scattered throughout his memoir. He has long promoted self-sufficiency and enthusiastically embraced the bootstrap ideology popular with conservatives.

“Success is determined not by whether or not you face obstacles, but by your reaction to them,” Carson wrote.

But those messages, now coming from a politician, are being received differently.

“There’s a certain consistency to his message, it’s just the language is now different,” said Kurt Schmoke, Baltimore’s first elected African-American mayor. Carson and Schmoke, now the president of the University of Baltimore, have been friends for decades.

“It’s more political, more partisan, and in my view, it’s harsher,” he said.

Schmoke said Carson’s achievements and philanthropic work haven’t been entirely eclipsed by his association with the Trump administration. But Carson’s political turn has changed the way many people see him.

“You can’t take away the fact that he’s done outstanding things for people throughout his life, that can’t be erased,” Schmoke said. “But I do think there’s clearly more people who view him through a political lens and that affects how he’s viewed in this community.”

Shaun Verma, a Ben Carson Scholarship recipient from Georgia, says Carson’s use of his story of hard work and determination to justify scaling back the safety net for the same communities that raised and revered him “is really disappointing.”

“He gave funding to inner city schools with big African-American populations, and opened reading rooms with the message that through education we can fight poverty and discrimination, and he was looked up to because he escaped his circumstances,” said Verma, who recently graduated from Johns Hopkins University and now lives in New York.

At 15, Verma founded MDJunior, a nonprofit that aims to improve health care accessibility to underserved communities. As a Carson scholar he attended board meetings and banquets, and got to know Carson personally. Carson’s policies, Verma said, have “tainted his long career and commendable service. It’s hard to associate all this with a person I looked up to for years.”

Some Maryland conservatives embrace Carson’s transition to politician. Antonio Campbell, a professor of political science at Towson University and state chairman for Carson’s 2016 presidential campaign, said he “remains impressed.” Those disappointed with his performance as HUD secretary likely feel that way because of fundamentally divergent values, he said.

“The question is, what is the role of government?” said Campbell, a Republican who is running for U.S. Senate against Sen. Ben Cardin, D-Md. When assistance is scaled back, those accustomed to the status quo are bound to be disappointed, he said, adding that Carson “is learning” how to sell his policies to skeptics.

As a child, growing up in a low-income, predominantly black neighborhood in a Maryland suburb, Boateng Kubi saw Carson as the embodiment of possibility.

“He’s one of the first people who truly indicated that black boys, black children, have the right to scientific curiosity,” said Kubi, a rising second-year student at Johns Hopkins School of Medicine and president of the school’s chapter of the largest association for minority medical students.

Kubi calls Carson’s policy proposals “shocking.”

“It feels like he’s neglecting the communities he came from, the people who grew up admiring him, who might not have all the money in the world,” he said. “I no longer speak of wanting to be the next Ben Carson.”

Carson has come back to Baltimore in an official capacity only three times since becoming HUD secretary. Last month the department gave $144 million in revitalization grants to five cities, including Baltimore. But for the announcement Carson sent a representative to the Perkins Homes housing complex 40 miles from HUD headquarters, opting instead to go to Flint, Michigan.

Carson doesn’t send groups to the wax museum anymore, said co-founder Joanne Martin. But his figure still gets plenty of attention.

Martin has heard her share of complaints about the figures in her museum: Martin Luther King Jr. is too short. Frederick Douglass is too light-skinned. But no figure has been nearly as controversial as Carson’s.

“There have been objections to him being in the museum,” she said. “People have posted on our website demanding that we remove him.”

Martin refuses. He’s earned his place in history as a doctor, Martin said, and she’s committed to honoring his contributions to medicine.

But she says she feels uneasy about Carson’s next chapter, and made a deliberate decision not to update his placard to include any information about his political life.

“We include figures of people who fought for civil rights and appreciated that struggle,” she said. “That’s not the person he is today.”