It takes a city nearly a decade to plan for the Olympics, from the invitation to declare interest by the International Olympic Committee to the selection to the opening ceremony seven years later. Previous hosts used the Games to bulk up infrastructure (stadiums, housing, transportation) in a bid to attract global investment. Pledges were made to repurpose existing structures or tear down and reuse what would be built, as Rio plans to do with some venues. That didn’t always happen.

An ongoing photography project by Jon Pack and Gary Hustwit explores the legacy of the Olympics on former host cities after the throngs of athletic teams and spectators gives way to the morning after: empty stadia, leftover signage, heaps of trash and the prevailing wonder of “was this worth it?” This project thoughtfully provokes viewers to answer that question on their own, showing a balance of ruin porn versus reuse.

Pack says the idea first emerged in 2008 during the Summer Olympics in Beijing. He and his wife were watching an event when she wondered aloud what would happen to it all after the last medal was awarded and everyone went home. Pack was curious, too. So that autumn he rented a car and drove to the nearest ex-host to Brooklyn, where he lives: Lake Placid in upstate New York, a two-time host in 1932 and 1980. He continued to travel to former host cities and, the next year, started to collaborate with Hustwit, who had been working on a documentary about the design of cities.

The pair typically visits each city solo. In the past, they would visit some solely to document them for their book, The Olympic City, which was published in 2013. Now, usually one of them finds their way to a new ex-host when they’re in the area for another purpose, like a work trip. They email photos back and forth, riffing about how it may or may not fit into what they’ve seen elsewhere. All told, they have visited 16 former hosts: from Munich (1972) to Moscow (1980), Helsinki (1952) to Seoul (1998).

In Athens (2004), Pack found the volleyball stadium in disuse. In Berlin (1936), Hustwit went inside the Olympic Village room of track star Jesse Owens, who won four gold medals at the Games in Nazi Germany. It had been turned into a museum in his honor. In Montreal (1976), when Pack visited in 2012, the 1.36-mile-long Olympic Basin was being used by a group of rowers. Major aquatic competitions are still held at the site.

In Sarajevo (1984), which became a battleground during the Balkans war in the 1990s, Pack says former event sites were still in use but not for their intended purpose, like a youth camp held near the ski jumps. Other photographers over the years have seen bikers riding down the bobsled track. Everybody had a story about the Olympics, he recalls: “It seemed like it was such a big point of pride that the city had done it even though they had gone through such tragedy there.”

In Barcelona (1992), the hilltop water complex with an incredible view of the city below is still used for competitions and partially open to the public. “It’s not all ruin porn,” Hustwit says, “but it really just depends on the city’s situation.”

In the Italian city of Turin (2006), where the pair traveled with a fixer in July, the Olympic Village became a hotbed for squatters and later for some 1,000 migrants and refugees. “It’s one of the most fascinating afterlives of the Olympics structures,” Hustwit says, weighing it against the utopian idea of the Games. “In one way, it makes perfect sense—maybe this is how Olympic Villages should be used, to house people in need.”

Rio is the latest city to tackle that issue. The city, which spent some $25 billion on the Games, reused several venues made for previous competitions. Whether the city makes good on a pledge to tear down and reuse stadia, and perhaps turn the athletes accommodations into luxury real estate, won’t be known for a while.

Pack and Hustwit have no immediate plans for a next destination but will continue to photograph as the opportunities arise. They plan to put out a second book and also to exhibit the photos again this year. “That’s what I like the most about the project,” Hustwit said. “It is about time and the long view.”

Hustwit suggested, admittedly as an oversimplification, that cities outside the states tend to build up more and show off what they can do as the global spotlight shines on them. “They want to be taken seriously on the world stage and they’re going to show it with all this stuff that’s being built,” he said. Yes, it’s about athletics, but also about national pride and turning a profit. “It’s kind of amazing to see what cities build and do for a relatively short event and how all that development meshes into the city in the decades afterward—or if it doesn’t.”

Jon Pack is a Brooklyn-based photographer. Follow him on Instagram @jnpck.

Gary Hustwit is an independent filmmaker and photographer based in New York. Follow him on Instagram @gary_hustwit.

Andrew Katz, TIME’s International Multimedia Editor, edited this photo essay. Follow him on Twitter @katz.