A Texas-based white nationalist didn’t wait for the dust to settle in Charlottesville, Virginia, where neo-Nazis and white nationalists spurred violence that left dozens of counter-protesters injured and a 32-year-old woman dead over the weekend, before announcing his plan to hold a “White Lives Matter” rally on Texas A&M’s campus on September 11. “TODAY CHARLOTTESVILLE TOMORROW TEXAS A&M,” Preston Wiginton, who organized a talk with white nationalist Richard Spencer on the campus last December, wrote in a press release announcing the event this weekend. The backlash was immediate, as Aggie alums, the entire House chamber of the Texas Legislature, and many others called on the university to stop the rally from happening.

On Monday afternoon, the university announced that the event would be canceled, citing “concerns about the safety of its students, faculty, staff, and the public.” A&M’s statement said that Wiginton “linking the tragedy of Charlottesville with the Texas A&M event creates a major security risk on our campus,” and that “the risks of threat to life and safety compel us to cancel the event.”

For many, the university’s decision to cancel the rally seemed like a move to protect its students. But to others it looked like a free speech violation.

“If all it took to shut down a speaker was to threaten a riot, we would all find ourselves with very little free speech at all,” said Ari Cohn, who works with the Foundation for Individual Rights in Education. “Allowing the possibility of a bottle-throwing mob to justify curtailing speech only incentivizes violence. That is an untenable result, from a legal and practical standpoint.”

Cohn said that the Wiginton’s link between Charlottesville and the planned A&M rally—a main reason the university gave for canceling the event—is protected under free speech. “Under applicable constitutional precedent, it constitutes neither incitement to imminent lawless action nor a true threat, and without more, that notice does not justify canceling the event,” Cohn said. “Moreover, violence committed in one place does not justify curtailing freedom of expression in other places. We do not discard the First Amendment’s protections because someone, somewhere reacted inappropriately to speech—and to be clear, violence is never an acceptable response to speech. But to sacrifice our First Amendment rights on the altar of preventing violence is incompatible with our understanding of free speech.

And now, A&M could have potentially opened itself up to a lawsuit. After learning that his rally was canceled, Wiginton told the Texas Tribune that he was preparing to sue the university. “I guess my lawyers will now be suing the state of Texas,” Wiginton said. “We have two lawyers in Texas who are alt-right leaning who will get in on the action too. We will probably get the ACLU to file suit as well.”

The ACLU hasn’t responded to requests for comment on whether it plans to be involved in any potential court proceedings, and, as of Tuesday afternoon, A&M spokesperson Amy Smith said the university hadn’t received any notice of a lawsuit. But there’s certainly precedent for legal action. In April, student groups at the University of California at Berkeley filed a lawsuit against the school for canceling a talk by prominent conservative media figure Anne Coulter amid safety concerns. The lawsuit, which had the backing of a prominent Republican lawyer, alleged that the university acted to silence the speech of conservative students, who are in the minority at Berkeley.

Similar legal action could be taken in College Station. “[Texas A&M], in denying White Lives Matters [and Richard] Spencer’s access to public areas to engage in speech—notwithstanding how deplorable and lacking in value that speech may be—more likely than not is violating the First Amendment by engaging in viewpoint-based discrimination,” JT Morris, an Austin-based attorney specializing in First Amendment law, said in an email. Morris said Wiginton could file a lawsuit immediately seeking an emergency temporary restraining order or preliminary injunction, asking a judge to force the university to grant Wiginton’s group access to campus for the rally. “This is a really tough situation to balance First Amendment rights and public safety,” Morris said. “We’re looking at protecting speech of minimal value—to put it gently—that is espoused by a very small group of people, at the risk of serious harm to public safety.” As Morris notes, the First Amendment protects what most of us consider to be hate speech, unless that speech crosses the line and incites violence. “I’m not sure the White Lives Matter [and] Richard Spencer stuff is quite there—yet,” Morris said.

Texas A&M, meanwhile, doesn’t seem particularly worried about fighting this out in court. “Our statement on the compelling reasons to block this event from happening stands on its own,” Smith said in an email, adding that the potential for legal ramifications did not weigh heavily in the university’s decision to cancel the rally. “The safety of our students, faculty and staff was—as it should have been—the critical factor in this decision,” Smith said. It remains to be seen, however, if that rationale could hold up in court.