md"# Aligning video recordings with Julia"

md"Due to the restrictions imposed by COVID-19, dance teachers around the world are taking their classes online. As a dance student, it would be helpful to watch yourself against a recording of your teacher.

​

Something like this:"

html"<iframe width=\"680\" height=\"383\" src=\"https://www.youtube-nocookie.com/embed/I8kuhv0FcTE\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"

md"It is very unlikely the song in both videos starts at exactly the same. Manually syncing them is not easy.

​

On the other hand, computers can compare the sound waves of each video file and quickly determine how long to wait before playing one video or the other so they are both aligned."

md"## Sound waves"

md"[Sound waves](https://pudding.cool/2018/02/waveforms/) are represented in digital audio by very rapidly sampling the distortions of the medium (e.g. air) through a microphone and storing the resulting data in a vector."

md"Jazmine, the teacher, sent Julia a [recording of a jazz song](https://freesound.org/s/37750/) ([cc0](https://creativecommons.org/publicdomain/zero/1.0/)), sampled 16,000 times per second:"

begin

    using FileIO: load

    import LibSndFile

    using Plots

    using SampledSignals

    using FFTW

end

md"Each sample in $$x$$ corresponds to a microphone reading. The first 5 samples at $$1s$$ are:"

md"Plotting the audio wave:"

md"Julia, then danced against the same song and recorded herself near [the sea](https://freesound.org/s/9332/) ([cc-by-nc](https://creativecommons.org/licenses/by-nc/3.0/)):"

md"Let's look at Julia's audio wave:"

md"We would like to know how much lag there is between the 2 different recordings of the same song: $$x$$ and $$x2$$."

md"## Using Cross Correlation to align audio signals"

md"Visually, we could superimpose both signals and slide each other over the time axis until they match as closely as possible. 

​

For example, take Jazmine's recording ($$x$$) and place it next to Julia's ($$x2$$). They are clearly not aligned."

begin

    x_right_padding_1 = zeros(Float32, length(x2) - length(x), 1)

    x_padded_1 = cat(x, x_right_padding_1, dims=1)

    mixed_1 = cat(x2, x_padded_1, dims=2)

    plot(domain(mixed_1), mixed_1, xlabel="time (s)", ylabel="amplitude", title="At the sea + Jazz song", linealpha=0.85, legend=false, fmt = :png, dpi=300)

end

md"Then slide Julia's recoring ($$x2$$) $$0.1$$ seconds to the right, then again $$0.1$$ seconds, …

​

 $$1$$ second later:"

begin

    x_left_padding_2 = zeros(Float32, convert(Int, 1 * x.samplerate), 1)

    x_right_padding_2 = zeros(Float32, length(x2) - length(x) - length(x_left_padding_2), 1)

    x_padded_2 = cat(x_left_padding_2, x, x_right_padding_2, dims=1)

    mixed_2 = cat(x2, x_padded_2, dims=2)

    plot(domain(mixed_2), mixed_2, xlabel="time (s)", ylabel="amplitude", title="At the sea + Jazz song. Offset 1 second", linealpha=0.85, legend=false, fmt = :png, dpi=300)

end

md" $$2$$ seconds later:"

begin

    x_left_padding_3 = zeros(Float32, convert(Int, 2 * x.samplerate), 1)

    x_right_padding_3 = zeros(Float32, length(x2) - length(x) - length(x_left_padding_3), 1)

    x_padded_3 = cat(x_left_padding_3, x, x_right_padding_3, dims=1)

    mixed_3 = cat(x2, x_padded_3, dims=2)

    plot(domain(mixed_2), mixed_3, xlabel="time (s)", ylabel="amplitude", title="At the sea + Jazz song. Offset 2 seconds", linealpha=0.85, legend=false, fmt = :png, dpi=300)

end

md" $$2.5$$ seconds later both $$x$$ and $$x2$$ are perfectly aligned."

begin

    x_left_padding_4 = zeros(Float32, convert(Int, 2.5 * x.samplerate), 1)

    x_right_padding_4 = zeros(Float32, length(x2) - length(x) - length(x_left_padding_4), 1)

    x_padded_4 = cat(x_left_padding_4, x, x_right_padding_4, dims=1)

    mixed_4 = cat(x2, x_padded_4, dims=2)

    plot(domain(mixed_2), mixed_4, xlabel="time (s)", ylabel="amplitude", title="Ocean Wave + Jazz. Offset 2.5 seconds", linealpha=0.85, legend=false, fmt = :png, dpi=300)

end

md"This sliding process is essentially what a cross-correlation measures. It tells us how similar 2 signals are when displaced relative to each other."

md"Mathematically, the [cross-correlation](https://en.wikipedia.org/wiki/Cross-correlation) of two temporal series $$u$$ and $$v$$ is defined as:"

md"$$(u \star v)(t) \triangleq\ \int_{-\infty}^\infty \overline{u(\tau)} v(t + \tau) \, d\tau$$"

md"Where $$\overline{u(\tau)}$$ is the [complex conjugate](https://en.wikipedia.org/wiki/Complex_conjugate) of $$u(\tau)$$. And $$t$$ is the lag."

md"The cross-correlation ($$\star$$) of two functions is equivalent to the convolution ($$*$$) of one of the signals reversed in the time axis: 

​

​

$$(u \star v)(t) =  u(−t) ∗ g (t)$$

​

Moreover, the Fourier transform ($$\mathcal{F}$$) of a convolution [is equivalent](https://en.wikipedia.org/wiki/Convolution_theorem) to multiplying the Fourier tranforms of the functions:

​

$$\mathcal{F}\{u * v\} = \mathcal{F}\{u\} \cdot \mathcal{F}\{v\}$$

​

$$(u \star v)(t) = \mathcal{F}^{-1}\big\{\mathcal{F}\{u(-t)\}\cdot\mathcal{F}\{v(t)\}\big\}$$

​

As such we can [implement the cross-correlation](https://dsp.stackexchange.com/questions/736/how-do-i-implement-cross-correlation-to-prove-two-audio-files-are-similar) with the following steps:"

md"1. zero padding the signals and reversing one of them:"

md"2. Performing a convolution between both signals with their Fourier transforms: "

md"The point where the signals are most correlated corresponds to the peak of the convolution. This is the lag between $$x$$ and $$x2$$:"

md"The lag is exactly 2.5 seconds. We can now use this value to align the audio/video files for Jazmine and Julia."