Kennedy’s departure from the Supreme Court has, understandably, prompted widespread concern about Roe v. Wade and abortion rights. Less salient but equally at risk is the court’s progress on LGBTQ rights.

While he was on the court, Kennedy was the decisive vote for the court’s 5-4 pro-gay rights rulings. He wrote each of the court’s landmark gay rights decisions, outlawing bans on same-sex sexual conduct, LGBTQ political advocacy and same-sex marriage at the state and federal level. Kennedy is the voice of the court’s gay rights doctrine.

Having tracked judicial progress on gay rights over the last decade, I’dargue that Kennedy’s retirement puts this doctrine, and the movement, in jeopardy. Although his rulings on same-sex relationships have ushered in new freedoms for lesbian and gay couples, they rest on fragile constitutional arguments. With Kennedy’s retirement, there is greater opportunity for anti-gay activists to dismantle the court’s tenuous legal framework supporting gay rights.

An unlikely advocate

Prior to his ascent to the court, few could have predicted that Justice Kennedy would spearhead a judicial revolution for LGBTQ rights. Kennedy joined the court in 1988 as a Reagan appointee whose only decision on gay rights, written during his tenure on the 9th Circuit Court of Appeals, narrowly upheld military regulations prohibiting “homosexual conduct.”

In 1996, less than a decade after his appointment to the court, Kennedy would author his first of multiple Supreme Court decisions protecting the rights of lesbians and gay men.

The 1996 case, Romer v. Evans, involved a constitutional amendment enacted by Colorado voters in 1992. The amendment barred any state or local public official from including sexual orientation in local nondiscrimination laws. If upheld, the amendment would have rendered any discrimination against gays or lesbians in housing, employment and public accommodations both legal and untouchable. The state argued that they were protecting the “liberties of landlords or employers who have personal or religious objections to homosexuality” and were doing “no more than deny[ing] homosexuals special rights.”

Kennedy authored the 6-3 decision to overturn the amendment, arguing that the state’s reasoning failed to meet even basic legal requirements. The policy, Kennedy stated, was “born of animosity toward the class of persons affected” and rendered members of the gay community “unequal to everyone else.”

For the gay community, Romer marked what New York Times reporter Linda Greenhouse described as a “historic shift in the Court’s response to anti-gay discrimination,” stopping other states from following suit. It also provided “a strong statement” against anti-gay discrimination, because it was delivered by “a conservative member of a basically conservative Court.” Yet, in overturning the Colorado amendment, Kennedy rested his decision on the easiest of judicial standards – arguing that the state failed to provide any rational reason for the ban.

The court could have argued instead that members of the LGBTQ community have historically been excluded from a wide range of state protections – and have far more frequently been the targets of hostile policy actions. The court treats policies that target historically excluded communities as “suspect” and makes it more difficult for policies to prevail.

Policy actors who target historically excluded groups need to come up with a significant reason for creating the policy. They must also prove that there is no other way to achieve their policy goal.

More importantly, once a group is treated as historically excluded in court precedent any policy that targets the group will be evaluated with this more challenging standard of judicial scrutiny.

While Kennedy’s rulings have struck down important limitations on lesbian and gay rights, he has resisted treating the LGBTQ community as historically excluded – leaving them with a far less certain set of legal protections.

Relationships and marriage

In 2003, Kennedy led the court’s majority in Lawrence v. Texas, overturning a Texas statute that criminalized same-sex sexual intimacy. In doing so, the court also overturned it’s prior 1986 decision in Bowers v. Hardwick, which upheld similar legislation in Georgia. Kennedy admonished the Bowers court for its “failure to appreciate the extent of the liberty at stake” and described bans on same-sex sexual conduct as “an attempt to control a personal relationship that, whether or not entitled to formal recognition in the law, is within the liberty of persons to choose without being punished as criminals.”

Bans on same-sex sexual intimacy licensed private citizens to treat lesbians and gay men as criminal. Free from the presumption of criminality, lesbians and gay men now had legal leverage to live openly and to fight for full equality.

Although the decision provided a legal mechanism for decriminalizing same-sex couples, Kennedy again resisted adopting the “historically excluded” framework. Instead, Kennedy argued that same-sex couples were being deprived, in this one instance, of important constitutional liberties.

Kennedy’s decisions on marriage equality provided similarly important but limited victories.

On June 26, 2013, Kennedy dismantled one of two barriers to marriage equality – a federal ban on same-sex marriage that prevented legally married same-sex couples from receiving the same federal benefits and protections offered to heterosexual married couples. In U.S. v. Windsor, Kennedy struck down the federal ban arguing that its only purpose was to “harm a politically unpopular group.”

State and federal judges used Windsor to overturn similar bans in more than 30 states.

Two years later, in Obergefell v. Hodges, Kennedy led the court in overturning the remaining state-level bans. Kennedy called specific attention to the ways in which bans on same sex marriage “harm and humiliate the children of same-sex couples” and impose on them the “significant material costs of being raised by unmarried parents.”

Windsor and Obergefell changed the lives of same-sex couples and their children. Children now have the security of being legally tied to both of their parents. Spouses can access critical health care or insurance benefits. And, in many communities, same-sex headed households have become more common and less vilified.

Yet, again, Kennedy sidestepped the question of historical exclusion, potentially limiting the reach of the decisions and weakening their capacity to withstand coming legal challenges.

Legal challenges remain

Kennedy’s gay rights doctrine is relatively new and still highly contested.

For instance, Texas’s ban on same-sex sexual conduct is still on the books, despite Lawrence. And, the Texas Supreme Court is still questioning whether Obergefell requires the state to provide benefits to married same-sex couples.

Wedding service providers are refusing to serve same-sex couples and are making headway in court. Kennedy recently upheld the rights of a Colorado baker to refuse to bake a wedding cake for a same-sex couple. He argued that compelling the baker “to exercise his artistic talents to express a message” that violated his religious beliefs could present a First Amendment problem. In the hands of a more conservative future court, that precedent could provide ammunition for the increasing number of states that are passing legislation allowing small businesses to refuse to work with same-sex couples on religious grounds.

Perhaps more importantly, there are many challenges to equality untouched by Kennedy’s rulings. This reality is a byproduct of his resistance to treating the LGBTQ community as historically excluded for the purposes of court doctrine. Discrimination against LGBTQ individuals in education, employment, housing and public accommodations is still legal in more than 20 states. And recent legislative attacks targeting transgender individuals suggest that anti-LGBTQ activity is still prevalent in many communities.

There is no doubt that Justice Kennedy’s legacy has brought about an era of unprecedented – even unimagined – rights for gays and lesbians. The question is will his legacy persist and grow without him on the bench?