“The Covid-19 pandemic financially exacerbated America’s affordable housing crisis,” Federal Housing Finance Agency Acting Director

Sandra L. Thompson

said. “Eliminating the [fee] will help families take advantage of the low-rate environment to save more money.”

U.S. home prices over the past year have risen at the fastest pace in decades, as a shortage of homes on the market collides with soaring demand from prospective buyers. The situation has exacerbated concerns about housing affordability and ratcheted up pressure on the Biden administration to respond.

With interest rates low by historical standards, millions of Americans have also refinanced existing mortgages to reduce their monthly payments. But the fee charged by Fannie and Freddie eliminated a chunk of those savings for affected borrowers.

Greg McBride,

chief financial analyst at personal-finance firm Bankrate.com, said the fee added about one-eighth of a percentage point to the average mortgage rate, or about $20 a month for a $300,000 loan.

“Lenders may use this as an opportunity to pad their own margins that have been squeezed by low rates and heated competition, so it is important for homeowners to seek out the lenders offering the best terms,” Mr. McBride said.

Average refinancing rates for a 30-year fixed-rate mortgage stood at 3.1% on Thursday, Mr. McBride said, citing Bankrate figures. That compares to an all-time low of 2.88% in January.

Fannie and Freddie encourage low borrowing costs by purchasing many mortgages from lenders and subsequently guaranteeing them. The fee was intended to cover potential losses related to the pandemic and government measures to soften its economic impact, including a foreclosure moratorium and another program that allowed some borrowers to suspend monthly payments.

Such programs didn’t see as much use as expected, as job losses were less severe than initially feared among more-affluent workers, who are more likely to own their homes. Rising home prices further diminished the potential for loan losses, and Fannie and Freddie both posted robust profits last year, even before the fee was in effect.

The fee prompted criticism from mortgage lenders and a bipartisan group of lawmakers.

The announcement of its elimination comes less than a month after President

Biden

ousted former FHFA Director

Mark Calabria,

a Trump administration appointee. Mr. Calabria had prioritized returning Fannie and Freddie to private hands after 12 years of government control—known as conservatorship—following the 2008 financial crisis.

The fee generated $5 million in additional revenue for every $1 billion of eligible mortgages acquired by Fannie and Freddie, bolstering their finances and enhancing their potential attractiveness to investors.

“There’s no continuing rationale for charging the fee, so they’re getting rid of it,” said

Bob Broeksmit,

president of the Mortgage Bankers Association. “It’s a clear manifestation of Acting Director Thompson’s pivot toward mission and consumers, as opposed to previous director Calabria’s single-minded focus on capital-building and release from conservatorship.”

Write to Paul Kiernan at paul.kiernan@wsj.com