View Issue Details

IDProjectCategoryView StatusLast Update
0006096ardourbugspublic2020-04-19 20:17
Reporterdon3 Assigned Tox42  
PrioritynormalSeveritymajorReproducibilitysometimes
Status closedResolutionfixed 
Summary0006096: Random glitches/pops/clicks in exported audio
DescriptionApologies in advance for a very long bug report!

This _may_ be a duplicate of bug 0005880, although some of the details
of the symptoms are different. It also may (or may not) be related to
forum post

    https://community.ardour.org/node/8447

and others...

Test system:
    HP Elitebook 8540w
    Intel Core i7 M 620 @ 2.67 GHz
    CPU frequency scaling disabled, FWIW
    8 GiB RAM
    Fedora 20 with some CCRMA extensions (standard kernel, which from
      what I've read should be fine for editing and exporting; I use an
      RT kernel on a different machine for recording)
    M-Audio FastTrak Pro (USB)
      JACK settings: 44100 frames/s, 256 frames/period, 3 periods/buffer
      qjackctl reports latency of 17.4 ms
    JACK version 1.9.9.5
    Ardour 3.5.403 (from web site)
    kernel version 3.15.10-201.fc20.x86_64

My procedure usually goes something like this:
  - Record 10 to 60 minutes of audio on 3 or more tracks. (I'm not sure
    the number of tracks is important. 1 may be enough -- my recordings
    are typically 3 to 8 tracks, so that's what I've tested. The length is
    probably not important either, but it should be at least a few minutes.)
  - Quit Ardour. (At this point I usually transfer the project from the
    recording computer to the lap-top.)
  - Start Ardour and open the project.
  - Make some changes... For me, this often includes
      - locking the positions of all recorded regions
      - copying them all (ctrl-left drag) to some later point on the
        time line and then on the copies,
      - changing region gains to maximize amplitudes
      - possibly cutting out some slices of time by splitting regions
        and shifting things around
      - adding a couple buses
      - adding fader automation points on tracks and buses
      - possibly adding LV2 plug-ins to one or more tracks and buses
        (eg, LinuxDSP BlackEQ, MBC2B on master bus).
    Usually nothing too exotic. :)
    (I'm not sure that any of this is relevant, but I haven't been
    able to reproduce the problem if I just create one track, import
    some audio, and export it without any of these things.)
  - Create a range (range markers) and export it as wav or FLAC.

To that we add, for testing:
  - Rename/move the exported file. (test1)
  - Make no changes and export again.
  - Rename/move the exported file. (test2)
  - Quit Ardour.
  - Restart Ardour and reload the project.
  - Export and rename again. (test3)
  - Rename/move the exported file.
  - Export and rename again. (test4)

Expected results:
All the exported files should contain identical audio, and they should
be free from pops/clicks (that aren't in the source, that is ;).

Actual results:
On the rare occasions I've checked, the files exported from the initial
editing session (eg, test1 and test2 above) are essentially identical,
within a sample quantum or two. [Why there's even that variation is
perhaps another topic...] But after quitting and reloading the project
(perhaps a few times), each exported file contains glitches in
apparently random locations. Some are audible, some are very subtle,
largely depending on the surroundings. And after that, multiple exports
from a single session (again without making any changes) also produce
differing output files.

I'll be attaching images (Audacity screenshots) showing waveforms from a
a recent example. After I had finished the first round of editing this
project, I exported it 3 times, shown in the screenshots as test1,
test3, and test4. The files are 16-bit FLAC in this case, but the
format doesn't seem to matter. I believe test1 was from the initial
editing session -- as far as I know, there are no glitches in this file.
test3 and test4 were exported after quitting Ardour and re-starting it.
Between each of the exported files are two difference files, generated
like this:

    $ sox --combine mix -v 1 test1.flac -v -1 test3.flac diff_1-3.flac
    $ sox --combine mix -v 1 test1.flac -v -1 test4.flac diff_1-4.flac

Even zoomed all the way out (first image), one can visually see what
look like glitches in the diff files. One may also notice that they do
not occur in the same times from one export to the next.

If we zoom in on a glitch (second image), we can see that that it's
really a run of differing samples, many of which are zero in either
test3 or test4 but non-zero in test1. In addition, on the leading edge,
the sample values in the glitch area decay over time until reaching
zero, whereas on the trailing edge, there's a one-sample jump from zero
back to the value in test1.

After examining several of these visually, I discovered that the width
of the difference areas was consistently equal to or a few less (it can
be hard to see exactly where the values start to diverge, even when
zooming in vertically) than the frames/period setting for JACK,
initially 256. When I changed JACK to use 512, the width of the
glitches changed correspondingly.
Additional InformationA bit of background... Since I first started experimenting with using
Ardour for work on recordings -- starting with A2, but continuing with
A3 -- I've had fairly persistent issues with pops/clicks and dropouts at
seemingly random locations in audio being played or exported. This has
been on multiple computers, running different distros and kernels, and
different audio hardware. In some cases I could correlate glitches with
xruns reported by qjackctl, but most of the time qjackctl would appear
clean (0 xruns, at least while exporting/playing). The pops/clicks
sounded like the usual sort that can be caused by dropped frames --
sometimes very obvious, other times very subtle. On the other hand,
what I'm calling "dropouts" sounded very much like the dropouts we used
to get back in the dark ages with cassette tapes -- which is to say a
momentary volume reduction, I'm guessing perhaps a hundred milliseconds
at the most in length, varying in depth, and without a (perceptible)
click. Because dropouts and clicks sounded so different, I generally
assumed their causes were probably also different -- although both
seemed random, relatively rare (seconds or minutes apart), and neither
came with a reported error by JACK (qjackctl) or Ardour. I don't
actually remember hearing a dropout during the last few months of using
A3 (ie, the latest 2 releases). I'm not saying that they don't happen,
but I've been noticing clicks much more than dropouts these days. Which
is to say that I'm still not sure whether they are related, but I found
the leading-edge decays on these glitches to be very interesting...

Although I've long experienced issues with both exporting and playing,
I've limited this bug report to exporting, mainly because it's easier to
instrument. Also, from what I understand, exporting doesn't involve
real-time interaction with the audio hardware, and it should run fine
with stock kernel -- it runs as fast as the slowest software in the
system, and shouldn't drop frames in any case. (Please correct me if
I'm wrong.)

As a work-around, I've hacked together a program on top of libsndfile
which reads and compares two audio files and writes a new file, with the
gaps in one file filled in with (hopefully valid) samples from the
other. Naturally this requires at least two exports from a project in
order to get a usable output file -- a waste of time and disc space,
especially for long recordings -- and it relies on the glitches not
being aligned in time, so it's rather specific to this application.

I've also made a couple enhancements to sndfile-cmp which have allowed
me to verify that the "repaired" output from my program, given test3 and
test4 above as inputs, effectively (within a couple sample quanta)
matches what's in test1. That's why I'm pretty sure that test1 contains
no glitches.
TagsNo tags attached.

Relationships

has duplicate 0006238 closedx42 Exported wav files contain random skips 
related to 0005880 closedSaBer Ardour 3 multitrack project export to audio file: glitches or dropouts in resulting file 

Activities

2014-12-30 03:31

 

scrshot_audacity_1.png (130,714 bytes)   
scrshot_audacity_1.png (130,714 bytes)   

2014-12-30 03:31

 

scrshot_audacity_2.png (83,032 bytes)   
scrshot_audacity_2.png (83,032 bytes)   

don3

2015-01-08 05:15

reporter   ~0016221

In case it isn't clear from the description, I haven't had much success
at isolating the "trigger" for this issue. It seems to be related
(somewhat) to "complexity" -- ie, it's generally more likely as the
number of tracks, buses, plugins, automation points - or just changes -
goes up. Unfortunately I still don't have any better info about that,
but I've done a little more experimenting...

A few days after submitting this bug report I worked on a new recording
session, using the same basic workflow described above. This time when
I finished my initial edits, I exported a range twice, and got two
completely _identical_ files -- that is, cmp(1) reports no differences.
(This is what I would have expected actually, at least with dithering
turned off, but it surprised me based on what I had seen before.) I
quit Ardour and re-started it, opened the same session, and exported
twice. Again the two files were exactly identical (to each other), as
verified by cmp. They were not identical to the two files exported in
the original editing session -- but the audio samples they contained
disagreed by no more than +/- 1 LSB value, which (for now) I consider to
be "matching". I quit and re-started Ardour a third time, and again
exported twice. The new files were again identical to each other _and_
they were also identical to the two files exported in the second run of
Ardour.

A couple days later I opened the session again to work on a different
part of the recording. I added some new markers and ranges, a new bus,
a couple more LinuxDSP plug-ins, and a number of fader automation
points. When I was finished with that round of changes, I exported the
new stuff twice and got files that did *not* match -- the differences
were similar to those reported in the original description (ie, random
glitches).

So then I went back and exported the first range (which I hadn't changed
in the second round) again, and found that it was different in many
places compared to the earlier exports by +/- 2 LSB values (larger
dithering noise? I believe it was still turned off), and more
importantly, it now also included random "glitch" differences, as
compared to the 3 earlier sets of exports.

So it seems that something I did with the later work triggered this
issue, and it now affects all exports from this project. But in other
sessions, the issue is present from the very first export, even with
less "complexity".

In addition, the most recent pair of exports exhibited a variation in
about half the difference areas that I hadn't noticed before: Instead of
decaying to zero samples, the original signal seems to be amplified.
I'll upload another screenshot, showing one of these zoomed in. In this
case I believe the second export ("exp2") is correct, as it agreed with
a later third export (not shown) in this area.

2015-01-08 05:18

 

scrnshot_audacity.3.png (87,843 bytes)   
scrnshot_audacity.3.png (87,843 bytes)   

don3

2015-01-31 06:17

reporter   ~0016318

I'm seeing the same issue with a recent nightly build (3.5-4392-gde85bfd, free version). The session I used for testing had exhibited the problem on 3.5.403 - and it has no plugins, which I thought should eliminate some variables.

I'm willing to try debugging this if I can get some tips about possible strategies or areas to look at. (I have past experience with development in C for *NIX and with embedded real-time systems, but I don't know anything of Ardour's internal architecture.)

x42

2015-03-22 16:45

administrator   ~0016465

That's very worrying. I did lots of export tests recently and was not able to reproduce. All are bit accurate (if dithering and normalization disabled) and I diff'ed them with original material (polarity invert). but I only checked wav (not flac, nor ogg). no time-offset, either.

JACK 1.9.9.5 has a few issues regarding freewheeling/export. Can you reproduce the issue with a later version of JACK, or better yet: Export using the Ardour-nightly "Dummy" backend.


What export format settings do you use?

How large is the session? (since this is a session that's always reproducible it would be great, private by email is fine too).

don3

2015-04-06 18:21

reporter   ~0016557

Last edited: 2015-04-06 23:18

x42: Sorry, I did't notice that you had responded to this; I got an e-mail notification from Mantis only yesterday evening (2015-04-05 19:25:35 -0500). To answer your questions...

For me, the appearance of the issue seems to be influenced in some way by "complexity". With very simple sessions - let's say, a couple tracks with one region each, no plugins, no automation - it's relatively unlikely. As I make changes (eg, splitting regions, shifting them around, adding buses, adding fader automation), it becomes more likely but still not guaranteed. However, many/most of the sessions I do involve a similar work-flow, and over the last 6 months or so I'd say the probability that I'll see the issue some time during work on a session is about 80-90%. Sometimes it's like my initial report, where the first exports from a session are okay (more precisely, no diffs seen in back-to-back exports) but I see the issue after quitting and restarting Ardour, but other times I see it right from the first couple export attempts in my initial editing session. Sometimes there are only a couple/few glitches in (let's say) 20 minutes of audio; other (most) times there are 10's (or more) in that time span. Sometimes the difference periods I see are the decay-to-zero type I initially reported; sometimes the amplitudes diverge (almost always the errant one is lower) but don't decay to zero; occasionally there is some other waveform distortion that I don't recognize. (Only once hace I seen the _increasing_ amplitude case I reported in the first note. Also keep in mind that I don't look at every single diff in detail -- often, there are too many, and/or I don't have time, so I just spot check them.)

So I think it's possibly that one could do many tests with simple scenarios and short time spans and not see the issue. Of course it's also possible (likely) that there's something "special" about my environment or configuration that causes it, which would explain why there have been so few reports of it.

Export settings: It varies, but typically the format is either wav or FLAC. I haven't seen any difference in behavior between the two. Here are some typical ones for me:

  - CD (Red Book): WAV, 44.1 kHz, 16-bit, shaped noise or no dithering
  - FLAC, 44.1 KHz, 16-bit, shaped noise or no dithering
  - FLAC, 44.1 KHz, 24-bit, no dithering

I almost always export a range - by right-clicking a range marker.

Since my last update I've set up a new computer with a faster CPU, more RAM, and a 9-year-old Delta 66 (PCI) sound card instead of the FastTrack Pro (USB). This one is also running Fedora with CCRMA, but it's F21, which comes with JACK 1.9.10 (so no more 1.9.9.5) and an RT-patched kernel: 3.14.29-200.rt26.1.fc21.ccrma.x86_64+rt
On this machine I see pretty much the same thing, except that decay-to-zero differences are much less common than the other types.

I also tried JACK1 on the original machine, with the same result. However I didn't report it because I built it from source and (because it was temporary) I did not follow the very clear and explicit instructions to install it on *top* of the JACK2 (package) installation - I set PATH and LD_LIBRARY_PATH on qjackctl and Ardour to point to my temporary JACK1, and from what I could see they picked up the intended binaries, but I realized the test still might not be valid.

I've tried two nightly builds during the last 3 months, with the same result, but yes I can try another, and the "Dummy" back-end - probably later this week. I'd love to test the "full" version, but so far I've only used the free ones because I can't seem to get PayPal to cooperate for a subscription. (I've donated, but I know that doesn't count.) Been meaning to try building from source, but so far haven't had time...

I came up with a way to compress the test session I mentioned in the previous note down to 230 MB, from 9 GB. Also mentioned here:
    https://community.ardour.org/node/8634#comment-39128
I might be able to reduce it even further (probably half or better) by removing regions that are not part of the test case and running clean-up (and verifying that the issue is still reproducible like that). Hoping to work on that later this week as well.

x42

2015-04-08 07:53

administrator   ~0016573

Don3, please send me a link to the session (robin@gareus.org).

x42

2015-04-13 21:13

administrator   ~0016594

I managed to reproduce it. The problem is coming from gain-fader automation (disable those and the issue disappears).

There's no fix yet, but we're a step closer to understanding the issue.

x42

2015-04-15 16:48

administrator   ~0016599

The issue can be explained with [fader] automation. A fix for this is now in Ardour 4.0-rc3-22-g96ce9c3

I have tested 3 exports since, and they're all good. Still, there may still be more to it [1].

It'd be great if you could confirm that Ardour 4.0-rc4 fixes this issue.


PS. [1] there is more to it, declick on rec-stop for example see git 0ffde76 (but that has no effect during export or playback)

don3

2015-04-15 19:46

reporter   ~0016602

Thanks x42 !! Ah, I see - locking issues - interesting...

Okay, I will try to test as soon as I can. Seems I still cannot get past the subscription issue with PayPal (see https://community.ardour.org/node/2260, p.3 2015-01-28 13:18; I've written to them, but no response yet) to download a nightly build without the fade-outs. Or do the fade-outs not affect exporting? If they do, I'll try installing the deps and see if I can build locally. It may take me a couple days to get to it, unfortunately.

x42

2015-04-15 21:51

administrator   ~0016603

should be fixed in 4.0-rc4. please test

don3

2015-04-16 19:39

reporter   ~0016610

I've exported (twice each) 4 regions from 3 different sessions which previously exhibited differences/glitches, ranging in length from about 20 minutes to over an hour. (Long enough that the probability would have been high that at least a few would show up, if they were going to show up at all.) I've seen no differences in output samples so far (ignoring differences due to dither), which is a very good sign!

In one case I exited and restarted A4, and exported a third time, and still no differences. (Previously there were (rare) occastions when exporting from one Ardour instance would produce matching files, but those would not match files produced by a later instance. Usually this only happened early on, like from the first editing session, but I thought I would try it anyway.)

These sessions have between 2 and 6 recorded tracks. I'd like to try one with 8 tracks or more, but it seems I don't have any of those loaded on this machine right now, and the computer that has them is not nearby (and the network connection to it is slow) - so that will probably have to wait until I can get to it physically, which might be a few days. But the results I have now are very promising indeed.

I'm planning to try this version for "production" work in the next week, to see how that goes.

Note: I had numerous issues with JACK2 1.9.10 (current with Fedora 21), including a UI lock-up at the end of exporting, and UI unresponsiveness and a run-away jackd process when trying to restart after that. I built JACK1 0.124.1 from source, and with that I haven't seen a single issue so far. As a (minor) bonus, qjackctl and Ardour now show 100% DSP load during export, whereas JACK2 typically showed 8-10%.

system

2020-04-19 20:17

developer   ~0023360

Issue has been closed automatically, by Trigger Close Plugin.
Feel free to re-open with additional information if you think the issue is not resolved.

Issue History

Date Modified Username Field Change
2014-12-30 03:31 don3 New Issue
2014-12-30 03:31 don3 File Added: scrshot_audacity_1.png
2014-12-30 03:31 don3 File Added: scrshot_audacity_2.png
2015-01-08 05:15 don3 Note Added: 0016221
2015-01-08 05:18 don3 File Added: scrnshot_audacity.3.png
2015-01-31 06:17 don3 Note Added: 0016318
2015-03-22 16:45 x42 Note Added: 0016465
2015-04-06 00:25 x42 Relationship added has duplicate 0006238
2015-04-06 18:21 don3 Note Added: 0016557
2015-04-06 23:18 don3 Note Edited: 0016557
2015-04-08 07:53 x42 Note Added: 0016573
2015-04-13 21:13 x42 Note Added: 0016594
2015-04-15 16:48 x42 Note Added: 0016599
2015-04-15 19:46 don3 Note Added: 0016602
2015-04-15 21:51 x42 Note Added: 0016603
2015-04-15 21:51 x42 Status new => feedback
2015-04-16 18:37 x42 Relationship added related to 0005880
2015-04-16 19:39 don3 Note Added: 0016610
2015-04-24 09:08 x42 Status feedback => resolved
2015-04-24 09:08 x42 Resolution open => fixed
2015-04-24 09:08 x42 Assigned To => x42
2020-04-19 20:17 system Note Added: 0023360
2020-04-19 20:17 system Status resolved => closed