The video for Andy Shauf’s song “Clove Cigarette” has received a fair amount of attention for its unusual visuals and complex workflow.
In the wake of receiving a Vimeo Staff Pick last month, No Film School asked me if I would be willing to break down the process of just how the video came to be. I sat down with my co-director Jared Raab and technical director Luca Tarantini to break down the production process and discuss how the pandemic forced us to think outside the box.
But first, check out the music video for "Clove Cigarette" below.
The Idea
Colin: The idea for “Clove Cigarette” came out of COVID-19 restrictions, really. I had made Andy’s last video for the “Neon Skyline” on Super-8, but I didn’t want to do anything like that again. It had been years since Jared and I had worked on anything together, and knowing that he’s a big fan of Andy’s I thought he might want to collaborate on something. I don’t think either of us had shot much of anything since the pandemic started, and I was definitely a bit apprehensive. But I remember our first meeting, we were sitting in your backyard and I think the first idea came from just being there and thinking, “Okay, what can we do safely?”
Jared: We said, “How do we make a video that is just this—me, and you, and this backyard? And maybe a camera, and maybe Andy, and keep it that simple and still have it be an interesting video?”
Colin: Yes, because the song takes place in what could be a backyard, you know, with plastic lawn chairs, and it was still summertime so it just felt appropriate with the lyrics of the song. And then I think you presented the idea of shooting with a LiDAR camera.
Jared:Probably because our friend Luca had been showing me some really interesting point cloud tests he’d been doing, I thought if we could do scans, maybe that was a way to make something very simple but very beautiful. We could keep the number of locations as minimal as the number of locations in the song, which were essentially a bar and a moment between two people on a summer evening.
Colin: We didn’t really want to make it into a straightforward narrative, but we did want to kind of tell the story of the song. Conveniently enough, the bar where the album takes place, the Skyline, is a five-minute walk from your backyard. Andy lives around there, too, so it was nice to just kind of situate everything in Parkdale (Toronto) and kind of bring the album to life in the actual locations. So yeah, once we had that idea it was just a matter of figuring out the technology behind it and bringing Tristan and Luca onboard.
Jared:Tristan Zerafa is a good friend of mine who works as a visual effects supervisor on much higher budget operations. I asked him about the LiDAR scanning part of it and whether or not he would be interested in being involved, and he said, “For sure.” And like I said, you, Luca, were the original reason why I even thought we had a chance at pulling this off because of the amount of Kinect scans that you were doing.
Luca:I’d been experimenting with Kinect scans and animation for a couple of years at that point, so I guess that’s what made you think of point clouds.
Xbox Kinect Scans by Luca Tarantini from a previous music video collaboration with Raab.Credit: Raab, Tarantini
Jared:Yeah, but you brought up something much more interesting, which was the ability to use photogrammetry, which is the process of taking overlapping photographs of something and turning them into a 3D model. The moment you started showing me some of the photogrammetry that you were doing, it looked in some ways more interesting than the Kinect scans because it was higher resolution but it was also a little bit more painterly. So with you and Tristan on board, we felt like we had enough to bring to Andy, which is when Colin went to him to say, “This is what we want to do.”
The Process
Luca:In the beginning, we really had no idea how we were going to do this. The path ahead was dark and dense, with a lot of methods to test and a lot of dead ends to discover, but Jared and I get really excited when we encounter technical barriers because it always means there's some unclaimed treasure on the other side. We were looking at what had been done so far with music videos and point clouds, and there was almost nothing except for this one guy doing really cool stuff, RubenFro. He was using Unity, which handled point clouds really nicely and allowed him to use deformers and do cool dissolves on the point clouds. But we wanted to try Unreal Engine because it had recently gotten point cloud support, and we just loved how it looked, like, we loved the lighting in Unreal. And Unreal was something that we’d been looking for an excuse to learn because it’s so versatile as a filmmaking tool, and learning to wield it, even in a basic way, showed so much promise to be hugely rewarding.
Raw point cloud scan data for 'Clove Cigarette.'Credit: Raab, Medley, Tarantini
Jared:Yeah. Unreal Engine is what they’re using to create the digital backdrops for shows like The Mandalorian, and soon countless other movies and TV shows. I think the real advantage is that it can render things in real-time that would take days for 3D graphics render farms normally.
Luca:Yeah, we were trying early on, another well-known and well-used MoGraph program called Cinema 4D, but the tests I was doing were going to take forever to render. And also I was trying to do animation with X-Particles which is this plugin for it, and that was going to be, like, a thousand-dollar license, and we said, “Okay, we’ve got to find another way to do this.”
Jared:Yeah, and then we found out that Unreal Engine is free to download, and you can start playing around with it right away.
Point clouds for 'Clove Cigarette' being worked on in Unreal Engine.Credit: Raab, Medley, Tarantini
Luca:Early on, we were talking about doing animated point clouds, because we were thinking about using the Kinect to maybe record Andy playing live or something like that. But before we ever got that working, we did some tests of just a still, painterly scene, without any animation, with beautiful lighting and the camera just floating through it, and we figured out that that was, like, enough. That was all we needed. So we didn’t pursue the animation. We had stumbled into this creative limitation that we leaned into instead of fighting. Just tableaux and lighting and graceful cinematography.
Colin: Yeah, totally. Once we decided to do tableaux, I think that gave us more direction as far as what we wanted these scenes to be. Just having tableaux kind of pushed it further into the dreamlike story we wanted to tell and not be stuck in a conventional music video type of mode.
Jared:And I think the thing that kept coming up is to keep it as simple as possible.
Luca:Super simple. Like, if you were to storyboard it, the whole video would be five panels. (Laughter)
The Technology
The Skyline Restaurant in an early point cloud test for the video.Credit: Raab, Medley, Tarantini
Luca:We made the point clouds, the initial raw material, using a Leica LiDAR device, and an iPhone that shot 4K video. That data was turned into photogrammetry afterward using a program called Metashape. We then imported those point clouds into Unreal Engine and assembled them into one continuous world that the camera would move through in one continuous camera move. And then lighting was added, atmosphere was added, and just, tweaked and tweaked and tweaked. So yeah, that’s all one take in Unreal Engine, and in the end, there are no hidden cuts in the video, it’s, like, one continuous world.
Jared:Which is an interesting point to be made, that your process was to take the point cloud scans that we did, and you assembled the world as a single continuous space.
Luca:Yeah, it was like in a video game level, when you zoom out and you see the whole map; like a Counter Strike level, basically.
'Clove Cigarette'Credit: Raab, Medley, Tarantini
Luca:Unreal can only handle a certain amount of points before it starts crashing. So, where we landed was about 20 million, so at any point on screen you’re seeing 20 million points. And almost every frame has 20 million points in it. And then I think overall the total world has about half a billion points with the combined point clouds together. The combination of LiDAR and photogrammetry. With that kind of resolution, we could have rendered it in 8K and beyond, but we settled for 4K in the final output.
Colin: Yeah, I just wanted to also add that it was amazing to be able to add so much as we went along. There was all the stuff we captured on the day of the shoot, but then, you know, even up until the last few days before we delivered we were like, you know, we had a cut we were happy with and I was like, “Oh, there should really be a bartender there,” and then, like, the next day Jared, I think Jared you just scanned yourself? And Luca was able to throw that in.
Jared:Cosette, who is my wonderful, brilliant, beautiful partner, who also appears in the video as the babe with golden hair, she did the scan of me in the kitchen on my iPhone just by walking around me, and I pretended to be a bartender so that we could add a bartender in at the Skyline so it didn’t feel so empty.
Colin: And, also the taxi, because I think, I don’t even know if we were fully considering when we were scanning the exterior of the Skyline, but when it was actually in the video, it looks like the album cover, which is a painting. So I think, yeah, Luca, you added—
Luca:Yeah, that look came later in Unreal, but we weren’t thinking about that on the day.
Colin: No, you added all these elements to bring it in even closer to the album cover, and the album cover has a yellow cab.
'Clove Cigarette'Credit: Raab, Medley, Tarantini
Jared:In order to get the element for the yellow cab, I had to find a YouTube video of a guy on the streets of New York getting a phone recording of a cab, just walking around it, being like, “Oh, cool, look at this!” and he’s just walking around the outside of the cab. And we managed to take that video piece and I turned it into photogrammetry to make a 3D element.
Luca:If we were shooting this project on film, or even on video, we would have never been able to add that taxi cab, or the plane in the sky, or all the other little things. But by digitizing our whole world, and even our actors, the details stay malleable for so long. It was amazing to be able to try dozens of camera moves and lighting setups on these “real-world” locations to arrive at a final product that we all loved.
The Challenges
Jared:Because all there is to the video are tableaux and camera motion, that camera motion had to be perfect. And Luca, you had a first pass of the video almost right away, you assembled the worlds, you put them together, you had a basic lighting approach, and that took you a week. And then the rest of it was just trying to tweak those camera movements, making lens choices and that final sequence, where things are appearing and disappearing, was a little bit finicky.
Luca:Yeah, that final sequence really took a lot of love to get right. But the first early pass of the camera movement had all the pacing for the song and it only took a day and a half, maybe, to do the camera moves. But then tweaking it and trying different options, adding little things in there, and playing with the depth of field and the flow of the camera and all that stuff—that took, like, another month. Once our pointillistic world was assembled, all the hard work was finding the vibe of the video through the camera work.
Still from 'Clove Cigarette.'Credit: Raab, Medley, Tarantini
Jared:Right, it was a matter of moving the camera through. You could animate lighting on and off, but nothing in the world moved outside of the camera and the lighting changes. And so, it was just a matter of getting that pacing right so that you were hitting the right places at the right time.
Luca:And, like, me animating the video alone wouldn’t have been anywhere close to where it landed. It was collaborating with you, Jared, and you, Colin, that’s how it got to be as smooth and simple and beautiful as it is. Like, just all of our tastes combining—it was a good collaboration.
Colin: For sure. It was amazing to have the freedom to refine the camera movements or try different transitions between the scenes.
Luca:Another challenge we had to overcome was the inherent digital coldness that you have with any 3D animation, and especially point clouds, which, initially, just look like floating pixels in black space. That didn’t suit Andy’s warm, hazy summer vibes. So I’m thinking, thinking, and one day the solution just hit me in the face and it was so obvious. Soft, warm, point-based imagery was perfected over a hundred years ago by pointillist painters like Pissarro and Seurat. They had developed this beautiful grammar of color and tone for a style of painting that in essence was identical to our modern point clouds. This oil-and-canvas art form was directly interfacing with our cutting edge 3D medium. We really took the “every frame a painting” thing seriously. As a visual artist, it’s important to collapse the past into the present and be aware of everything and not discriminate about where you take inspiration from. People have been solving creative problems for millennia, and all the answers are out there.
'Peasants' houses, Eragny,' Camille Pissarro (1887)
The Future
Luca:I have a question for Colin! Colin, as a more or less non-technical filmmaker, at least not in the digital technologies way—
Colin: No, that’s very accurate—
Luca: —how is it working on a project like this?
Colin: It was so fun. Like, I’ve made a lot of music videos on my own and I always think about what I’m capable of doing with my skills, so it was just so fun to work with you guys and make something completely different than I would have ever done on my own. So yeah, it was great. I want to make more stuff like that.
Luca:Do you have any advice for a non-technical filmmaker who is going to be doing a digital project like this?
Colin: Just find good collaborators and trust them completely. (Laughter)
Jared:That’s good advice. Well, I trust you guys completely and I think the video could not possibly have turned out half as good—that’s even giving it too much credit. What I’m trying to say is I couldn’t have made this video without you guys.
Luca:Yeah. Same, dude. I think this is, like, definitely one of my favorite projects I’ve ever done. It was cool doing it with you guys.
Jared:It genuinely, for the first time, made me excited and not terrified by the future of filmmaking becoming more virtual.
