It has been a rough week for proponents of New York City's pied-à-terre tax. On Monday, former Commerce Secretary Wilbur Ross filed a lawsuit against the state, arguing the levy is unconstitutional. On Tuesday, a judge ruled that the city must roll back the notices it had sent to property owners and effectively start over.

The developments leave the fight unfolding on two tracks: the city's handling of the notices and the broader constitutional challenge to the surcharge itself.

The setbacks follow a chaotic rollout for the tax, along with threats and warnings that a video in which the city's young mayor announced the measure outside billionaire Ken Griffin's home would set off an exodus of wealth from New York. That exodus has yet to materialize. The lawsuits, however, have.

Ross on Monday joined a growing list of plaintiffs suing either the city or the state over the tax. In an interview with Fortune, he said the new tax on luxury second homes singles out the one group of owners who cannot vote against it.

"They don't want voter retribution for taxes at the ballot box, so they impose these taxes on people who have no way to defend themselves," Ross said. "That's what this is all about."

Ross, his wife, Hilary Geary Ross, and casino developer Steve Wynn sued the State of New York on Monday in Suffolk County Supreme Court, arguing that the pied-à-terre surcharge violates both the state and U.S. constitutions. All three are Florida residents who own Manhattan apartments. According to the complaint, the city's Department of Finance billed the Rosses $83,531.52 on their co-op and Wynn $183,094.69.

The Case Against the Surcharge

Ross's suit, led by Pillsbury Winthrop Shaw Pittman partner James Catterson, advances three main arguments. The first is that the "surcharge" is in substance a property tax: it is based on property value, billed through the city's property tax system, and becomes a lien on the home if left unpaid. The complaint contends this violates the state constitution, which caps how much the city can raise through real estate taxes. Under the new law, surcharge revenue does not count toward that cap.

"The state, by fiat, cannot change the constitutional reality of what it is," Ross said. "Surcharge on what?"

The complaint further argues that taxing owners according to where they live discriminates against out-of-state residents. The suit says that violates the U.S. Constitution's Privileges and Immunities and Commerce clauses, as well as equal protection guarantees in the federal and state constitutions.

"By their theory, there's no limit to what they could do to nonresidents," Ross said. "'We'll put 100% tax every year on the property. 200%.'"

Ross rejected officials' claims that part-time owners do not pay their share. Owners subject to the tax, he said, use none of the city's spending on education or health and welfare, and draw on less policing, fire protection, and trash pickup, because they spend less than half the year in the city.

"How can you possibly consume more in less than six months than other people do in a whole year?" he said.

He added that nonresidents already pay more, both because their homes are worth more and because nonresident co-op and condo owners do not receive the tax abatement granted to primary residents.

"If my next-door neighbor in the building is a resident and he has the same size apartment I have, I pay more than he does," he said.

When Fortune noted that the tax, according to New York Gov. Kathy Hochul, was meant to close the gap on New York property owners who "do not live in the city or pay city income tax," Ross was unmoved.

"There is no gap. This is an imaginary gap," he said.

He added that he does not object to higher taxes on the wealthy as long as the rate applies to everyone in a bracket.

"I don't think it's a good idea, but it's legal," he said.

The Counterattack

Forbes put Ross's net worth at around $600 million in 2019, following controversy over allegations that he had inflated his fortune to between $2.7 billion and $3.7 billion. Even so, in the view of Hochul's office, Ross and Wynn—whose net worth is $4.3 billion, per Forbes—are proving why the pied-à-terre tax exists in the first place.

"When Steve Wynn and Wilbur Ross try to cast themselves as sympathetic figures in a fight over paying their fair share on multimillion-dollar second homes, they're making the case for the pied-à-terre tax as well as anyone could," Jen Goodman, Hochul's director of rapid response, told Fortune in a statement.

"Governor Hochul believes some of the wealthiest people in the world, and the powerful interest groups fighting on their behalf, can afford to help pay for the police officers, trash pickup, and snow removal that keep New York City running," she continued. "The Governor was proud to sign this legislation, and the state will defend it in court."

Ross called the statement "silly" and said it did not address the legal question.

"It either is constitutional or it isn't," he said. "Unconstitutional is unconstitutional."

Matt Rauschenbach, a spokesperson for New York City Mayor Zohran Mamdani, told Fortune in a statement: "The pied-à-terre surcharge demands that the wealthiest people who own second homes in NYC but don't live in them pay their fair share towards funding safer streets, cleaner parks, and better schools."

"The city is moving to intervene in these suits and will stand with our partners in Albany to defend the surcharge," he continued. "And while the legal process moves forward, we will continue administering the surcharge fairly, efficiently, and in full compliance with the law, as we have done from day one."

Ross argued that such statements do little to address the tax's constitutionality—and that he and his wife do, in fact, spend money in the city despite rarely being there.

"We aren't here that much, so we have to ration what we do," he said. "We eat out all the time. We shop here. We use Ubers and cars and things like that. So we do spend money here, and we help some of the cultural institutions."

That spending, he said, supports jobs whose workers pay city income tax. Owners subject to the tax "hire more maids, use more drivers, use more Ubers, use more taxis, buy more things in the stores, support the charities more," he said. "You can't just look at it the other way. You've got to take everything into account."

This story was originally featured on Fortune.com.