The World Trade Center was not always so beloved. A generation ago, when the Rockefeller brothers—Governor Nelson and banker David—first devised the idea of two monumental towers, one powerful group stood implacably opposed: the city’s real-estate developers. Harry Helmsley and Lawrence Wein, the partners who owned the Empire State Building, formed a group called the Committee for a Reasonable World Trade Center. They took out ads in The New York Times arguing that 10 million square feet of government-subsidized office space would depress the value of their real estate. They sued to stop the project. They lost.

Now the Twin Towers are gone, and a debate is afoot about what should replace them. And history has begun to repeat itself.

Ever since the attacks, one developer, Larry Silverstein, has been tirelessly pushing his plans to redevelop the site. “To do anything less,” Mr. Silverstein vowed in an emotional speech to business leaders six days after the attacks, “would be to simply give an incredible victory to those who sought to destroy our way of life.” At the time, no one dared dispute him.

That’s changing. A powerful opposition has begun to coalesce in recent weeks and is questioning whether commercial redevelopment of the site is really the best way to show up the terrorists. It includes advocates for the families of those who died on Sept. 11, who have pronounced Mr. Silverstein’s plans “sacrilegious,” and former Mayor Rudolph Giuliani, who, in his farewell address to the city, appealed for the site to be preserved from economic development.

Most surprising—at least superficially—and perhaps most dangerous to Mr. Silverstein’s plans, however, is the steadily growing consensus forming against him within the city’s real-estate community. Quietly, privately, many of Mr. Silverstein’s peers have begun to rethink their initial support for rebuilding. Now as in the 1960’s, they’re making an argument based on economics—and no small measure of self-interest. At a time when the recession has cut deeply into the occupancy rate for office space, they say Mr. Silverstein’s buildings could further depress real estate and actually hinder lower Manhattan’s recovery.

Their fears haven’t been calmed by Mr. Silverstein’s first moves. He has pledged to begin rebuilding 7 World Trade Center, the 2-million-square-foot office building that collapsed after the two larger towers, by this summer. With a big insurance settlement in hand, Mr. Silverstein hopes to lure tenants with deeply discounted rents.

“It’s not good for downtown,” said one lower Manhattan real-estate executive.

“It would just be a repeat of what happened the first time, when Rockefeller built it,” said M. Myers Mermel, a real-estate investor and the chief executive of Tenantwise.com. “[The World Trade Center] was a drag on downtown the first 15 years.”

“You’ll have the same thing [happening],” warned another real-estate executive.

Few developers or real-estate brokers want to say such things publicly; no one wants to be cut out of a piece of the action if Mr. Silverstein succeeds. “I think that, sure,” one industry bigwig said recently. “Can I say that aloud? I think not yet.”

But the anti-redevelopment message has already reached some of the people who count.

“Everyone in the real-estate business has an interest in what happens here, ”JohnC. Whitehead, the new chairman of the Lower Manhattan Redevelopment Corporation, said in a recent interview. “They fear flooding the market with too much space, which will damage the ability of older buildings to be leased up.”

Mr. Silverstein’s partisans attribute such sentiments to the usual infighting among business competitors, and point out that numerous powerful constituencies still favor redevelopment. Mr. Whitehead, who is Governor George Pataki’s appointee, and Mayor Michael Bloomberg have both said that portions of the site should be put to commercial use. The Port Authority of New York and New Jersey, which owns the land, could lose billions in revenue if nothing is rebuilt.

Not rebuilding would also mean that the city sees none of the billions in insurance money that Mr. Silverstein stands to collect reinvested in downtown. (Just how much he’s owed is still in dispute.)

“We firmly believe that the best use of that space is an appropriate combination of commercial development and a significant memorial,” said Steve Solomon, Mr. Silverstein’s spokesman. “And we believe the future of lower Manhattan, as well as all of New York, will be best served by redeveloping that site in that particular manner.”

Signs of Progress

Mr. Silverstein can point to some signs of progress. His lawyers recently argued his case on the insurance policy in federal court in Manhattan, and he hopes for a decision on the crucial issue of whether the two planes constituted one coordinated attack or two separate attacks by this summer. If they were two attacks, as Mr. Silverstein argues, he is entitled to twice the insurance money in his policy, or about $7.2 billion. In the meantime, he has been lobbying Port Authority officials and Mr. Whitehead, whose support he needs to go forward. His architects have begun to produce drawings of new skyscrapers. He envisions somewhere between four and six buildings, in addition to a memorial, a museum and cultural attractions.

He says work on 7 World Trade Center, the smallest of the three collapsed buildings, will begin soon. Usually developers must wait until they have tenants signed up before embarking on such a large-scale development. But because Mr. Silverstein’s $860 million insurance policy on that building is not in dispute, he may just be able to go it alone with little risk to himself. (The Blackstone Group holds the building’s $383 million mortgage, which in turn is backed by bonds. It’s not clear whether Mr. Silverstein will have to pay off the mortgage before rebuilding.)

“He expects to be able to deliver an extraordinarily designed building at very affordable rental [rates],” Mr. Solomon said.

That terrifies other Manhattan landlords, who are already looking at office-vacancy rates of around 14 percent, according to Mr. Mermel. And it could just be a harbinger of things to come if Mr. Silverstein gets his insurance money for the rest of the site.

Mr. Silverstein’s legal rights to rebuild the rest of the site remain a complicated issue. But through a sophisticated public-relations campaign that began almost immediately after the attacks, he has firmly established himself in the public mind as redevelopment’s indispensable man.

Many in the real-estate community warn against underestimating Mr. Silverstein. “Larry did not fall off the turnip truck,” said Michael Cohen, chief executive of the real-estate brokerage GVA Williams. “He’s a dyed-in-the-wool New York real-estate developer, and he understands the market as well as anybody—and better than most.”

But clearly, opponents of development are gaining momentum. While a month ago, some sort of commercial development of the 16-acre site seemed a fait accompli, now some prominent players involved with the plans to revitalize lower Manhattan think it’s becoming less and less likely that the Trade Center will be rebuilt. Mr. Giuliani—indisputably the most popular public figure in the city—will be a formidable opponent, even out of office. Some business leaders privately worry that the politics of grief—and the former Mayor’s vocal backing—have so empowered the bereaved families that they cannot be refused on their belief that the site should be reserved for a memorial only.

Proponents of rebuilding argue that, with time, emotions will calm and the economic rationale for a new office complex will become clear. And that’s why opposition from his own industry is so dangerous to Mr. Silverstein. If other developers publicly turn against him, they can turn his justification for a new World Trade Center—that it will revive the city’s battered economy—on its head.

“It’s going to be very hard for anybody to build on that site any kind of commercial venture any time in the near future,” said one prominent developer.

Some, like Mr. Cohen, doubt that Mr. Silverstein really intends to build speculatively. His talk about 7 World Trade Center, they say, is a ploy to demonstrate his seriousness about rebuilding in the only venue that matters to him right now: the federal courtroom where his insurance dispute is being judged.

“I think Larry is just taking a negotiating position,” the developer said. “I don’t think he has any intention of building, ever.”

“Who wants to be sitting in their office on top of a graveyard?” said another real-estate executive. “It’s hard to believe something is going to be rebuilt that’s commercially feasible on the site. None of us in the industry believe it.”

However, “don’t expect him to do anything stupid,” Mr. Cohen countered. “There’s not a lot of upside in betting against Larry Silverstein.”