View Issue Details

IDProjectCategoryView StatusLast Update
0009049ardourbugspublic2022-11-22 07:21
Reportergonsolo Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformUbuntuOSLinuxOS Version(any)
Product Version7.0 
Summary0009049: Crash when trying to jump and play
DescriptionA session upgraded to Ardour 7 seems to be seriously confused.
When playing and trying to jump after location marker 8 Ardour crashes.

Stack trace:

```
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
0000001 __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff343bc46 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
0000004 0x00007ffff34227fc in __GI_abort () at ./stdlib/abort.c:79
0000005 0x00007ffff58a2d27 in Temporal::TempoMap::get_grid(std::__cxx11::list<Temporal::TempoMapPoint, std::allocator<Temporal::TempoMapPoint> >&, long, long, unsigned int) const
     (this=0x55555793c700, ret=empty std::__cxx11::list, start=282401617680, end=282401617680, bar_mod=0) at ../libs/temporal/tempo.cc:1916
#6 0x00007ffff774b722 in ARDOUR::LV2Plugin::connect_and_run(ARDOUR::BufferSet&, long, long, double, ARDOUR::ChanMapping const&, ARDOUR::ChanMapping const&, unsigned int, long)
     (this=0x55556d9d8e40, bufs=..., start=48027486, end=48027486, speed=0, in_map=..., out_map=..., nframes=1024, offset=0) at ../libs/ardour/lv2_plugin.cc:2668
#7 0x00007ffff73d0e57 in ARDOUR::PluginInsert::connect_and_run(ARDOUR::BufferSet&, long, long, double, unsigned int, long, bool)
     (this=0x55556d9d3d60, bufs=..., start=48027486, end=48027486, speed=0, nframes=1024, offset=0, with_auto=true) at ../libs/ardour/plugin_insert.cc:1107
0000008 0x00007ffff73d2785 in ARDOUR::PluginInsert::run(ARDOUR::BufferSet&, long, long, double, unsigned int, bool)
    (this=0x55556d9d3d60, bufs=..., start_sample=48027486, end_sample=48027486, speed=0, nframes=1024) at ../libs/ardour/plugin_insert.cc:1329
0000009 0x00007ffff74c62a6 in ARDOUR::Route::process_output_buffers(ARDOUR::BufferSet&, long, long, unsigned int, bool, bool)
     (this=0x55556c1bd8d0, bufs=..., start_sample=48027486, end_sample=48027486, nframes=1024, gain_automation_ok=false, run_disk_reader=false) at ../libs/ardour/route.cc:543
0000010 0x00007ffff74c7441 in ARDOUR::Route::run_route(long, long, unsigned int, bool, bool)
    (this=0x55556c1bd8d0, start_sample=48026446, end_sample=48026446, nframes=1024, gain_automation_ok=false, run_disk_reader=false) at ../libs/ardour/route.cc:734
0000011 0x00007ffff74dcd49 in ARDOUR::Route::no_roll_unlocked(unsigned int, long, long, bool)
    (this=0x55556c1bd8d0, nframes=1024, start_sample=48026446, end_sample=48026446, session_state_changing=false) at ../libs/ardour/route.cc:4054
0000012 0x00007ffff731ce15 in ARDOUR::MidiTrack::no_roll_unlocked(unsigned int, long, long, bool)
     (this=0x55556c1bd8d0, nframes=1024, start_sample=48026446, end_sample=48026446, state_changing=false) at ../libs/ardour/midi_track.cc:389
0000013 0x00007ffff74dcbfa in ARDOUR::Route::no_roll(unsigned int, long, long, bool)
    (this=0x55556c1bd8d0, nframes=1024, start_sample=48026446, end_sample=48026446, session_state_changing=false) at ../libs/ardour/route.cc:4024
0000014 0x00007ffff6f3a24e in ARDOUR::Graph::process_one_route(ARDOUR::Route*) (this=0x555558223800, route=0x55556c1bd8d0) at ../libs/ardour/graph.cc:547
#15 0x00007ffff74c7262 in ARDOUR::Route::process() (this=0x55556c1bd8d0) at ../libs/ardour/route.cc:705
0000016 0x00007ffff6f45b45 in ARDOUR::GraphNode::run(ARDOUR::GraphChain const*) (this=0x55556c1bdd20, chain=0x7fffb40826d0) at ../libs/ardour/graphnode.cc:65
#17 0x00007ffff6f38853 in ARDOUR::Graph::run_one() (this=0x555558223800) at ../libs/ardour/graph.cc:346
0000018 0x00007ffff6f38c11 in ARDOUR::Graph::helper_thread() (this=0x555558223800) at ../libs/ardour/graph.cc:374
0000019 0x00007ffff6f44bb1 in boost::_mfi::mf0<void, ARDOUR::Graph>::operator()(ARDOUR::Graph*) const (this=0x7fffa1ffa958, p=0x555558223800) at /usr/include/boost/bind/mem_fn_template.hpp:49
0000020 0x00007ffff6f4413d in boost::_bi::list1<boost::_bi::value<ARDOUR::Graph*> >::operator()<boost::_mfi::mf0<void, ARDOUR::Graph>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, ARDOUR::Graph>&, boost::_bi::list0&, int) (this=0x7fffa1ffa968, f=..., a=...) at /usr/include/boost/bind/bind.hpp:259
0000021 0x00007ffff6f43090 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, ARDOUR::Graph>, boost::_bi::list1<boost::_bi::value<ARDOUR::Graph*> > >::operator()() (this=0x7fffa1ffa958)
    at /usr/include/boost/bind/bind.hpp:1294
0000022 0x00007ffff6f41601 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, ARDOUR::Graph>, boost::_bi::list1<boost::_bi::value<ARDOUR::Graph*> > >, void>::invoke(boost::detail::function::function_buffer&) (function_obj_ptr=...) at /usr/include/boost/function/function_template.hpp:158
0000023 0x0000555555cb32d3 in boost::function0<void>::operator()() const (this=0x7fffa1ffa950) at /usr/include/boost/function/function_template.hpp:763
#24 0x00007fffd7fba431 in ARDOUR::JACKAudioBackend::_start_process_thread(void*) (arg=0x555558394410) at ../libs/backends/jack/jack_audiobackend.cc:955
0000025 0x00007ffff3490402 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
0000026 0x00007ffff351f590 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
```
Steps To Reproduce1. Open the session.
2. Play.
3. Jump via mouse to location marker 8 or a bit later.
4. Crash.
Additional InformationArchive (3MB which is bigger than the allowed 2MB):
https://drive.google.com/file/d/1qI6lSKW8yoTqECEjxAuxoKdlyNQomG0k/view?usp=share_link
TagsNo tags attached.

Relationships

related to 0009067 assignedpaul Crash when playing back a tempo ramp and a TS change 

Activities

paul

2022-10-29 17:05

administrator   ~0026752

There are at least 3 other bug reports withthe same backtrace.

Thanks for the session archive, should be useful.

gonsolo

2022-10-29 17:39

reporter   ~0026753

I find https://tracker.ardour.org/view.php?id=9027 which is marked fixed with 7.0-116-g489c9ace9f which it isn't (I believe).
A search for "TempoMap::get_grid" doesn't reveal any other reports.

And sorry for being a PITA. :)

paul

2022-10-29 17:43

administrator   ~0026754

Not a pain at all. Please keep reports coming.

gonsolo

2022-10-30 10:48

reporter   ~0026756

I think the culprit is TempoMetric::bbt_at at tempo.cc:622 because it is confused of the 3/4 in the middle of a bar.
I managed to manually get it to work by moving the 3/4 to the beginning of a bar.

gonsolo start0: 47001057, end: 47001057
 start0 superclock: 276366215160
 end superclock: 276366215160
  sc: 276366215160
  dg.get_beats(): 10
  _meter->note_value(): 4
  note_value_count: 10
  ticks: 1077
  _meter->bbt(): 290|1|0
  bbt_offset: 0|10|1077
  add: 293|2|1077
  bbt: 293|2|1077
  new_start: 276441352257
  sc: 276366215160
  dg.get_beats(): 10
  _meter->note_value(): 4
  note_value_count: 10
  ticks: 1077
  _meter->bbt(): 290|1|0
  bbt_offset: 0|10|1077
  add: 293|2|1077
  sc: 276366215160
  dg.get_beats(): 10
  _meter->note_value(): 4
  note_value_count: 10
  ticks: 1077
  _meter->bbt(): 290|1|0
  bbt_offset: 0|10|1077
  add: 293|2|1077
gonsolo start0: 47001057, end: 47001057
 start0 superclock: 276366215160
 end superclock: 276366215160
  sc: 276366215160
  dg.get_beats(): 10
  _meter->note_value(): 4
  note_value_count: 10
  ticks: 1077
  _meter->bbt(): 290|1|0
  bbt_offset: 0|10|1077
  add: 293|2|1077
  bbt: 293|2|1077
  new_start: 276441352257
  sc: 276366215160
  dg.get_beats(): 10
  _meter->note_value(): 4
  note_value_count: 10
  ticks: 1077
  _meter->bbt(): 290|1|0
  bbt_offset: 0|10|1077
  add: 293|2|1077

paul

2022-10-30 13:34

administrator   ~0026757

hmm, meter (time signature) changes are supposed to be allowed on bars only. This is part of the conceptual changes that make full compatibility with 6.x hard. Importing a 6.x session that allowed those changes off-bar would definitely cause issues.

paul

2022-10-30 14:49

administrator   ~0026758

Last edited: 2022-10-30 14:51

that said, the .ardour file you attached doesn't seem to show any meter changes off-bar:

      <Meter note-value="4" divisions-per-bar="4" sclock="0" quarters="0:0" bbt="1|1|0"/>
      <Meter note-value="4" divisions-per-bar="3" sclock="112409227592" quarters="400:0" bbt="101|1|0"/>
      <Meter note-value="4" divisions-per-bar="4" sclock="132422609396" quarters="478:0" bbt="127|1|0"/>
      <Meter note-value="4" divisions-per-bar="4" sclock="237611970470" quarters="986:0" bbt="254|1|0"/>
      <Meter note-value="4" divisions-per-bar="3" sclock="274559752262" quarters="1130:0" bbt="290|1|0"/>
      <Meter note-value="4" divisions-per-bar="4" sclock="300066321800" quarters="1154:0" bbt="305|1|0"/>
      <Meter note-value="4" divisions-per-bar="4" sclock="323415267238" quarters="1266:0" bbt="333|1|0"/>

when loading here, the 3/4 meter starts at bar 101

Is that not the case for you?

paul

2022-10-30 16:23

administrator   ~0026761

OK, I understand the bug now. But it will take a day of putting up tiles in my bathroom to come up with a solution.

The problem is that when we are computing the grid near a sample position, we first determine meter and tempo in effect at that point. Then we compute the number of quarter notes since the tempo change, and then use that to compute the delta in quarter notes between the meter change and the given position.

Problem is, if the meter is before the tempo, then for the interval from the meter to the tempo, the duration of a quarter note is defined by the *previous tempo*, not the one in effect at the time we're interested in. This ends up leading to the "we've gone backwards" error.

To the tiles!

paul

2022-10-30 16:25

administrator   ~0026762

BTW, I did try just using the *later* of the tempo or meter change for the computation, which initially looked like a simple fix. But this just breaks other stuff.

gonsolo

2022-10-30 16:40

reporter   ~0026763

```
<Meter note-value="4" divisions-per-bar="3" sclock="112409227592" quarters="400:0" bbt="101|1|0"/>
```
If you're saying that the last number in "bbt="101|1|0" has to be always zero, then yes, my ardour file is sane.
But after opening the meter change is still off-bar as can be seen in the picture.

I see that you now have an understanding of the bug and I expect the fix just after the tiles. (Kidding, as I said, I manually fixed the session, but it's still nice to hear that I can help to make Ardour rock solid again.)
wrong.png (20,368 bytes)   
wrong.png (20,368 bytes)   

gonsolo

2022-10-30 17:13

reporter   ~0026764

Since it is related (or the same bug): Dragging the tempo marker at bar 300 around is also crashing.
Stack trace:

#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
0000001 __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff343bc46 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
0000004 0x00007ffff34227fc in __GI_abort () at ./stdlib/abort.c:79
0000005 0x00007ffff342271b in __assert_fail_base
    (fmt=0x7fffef6039b5 "%s%s%s:%u: %s%sZusicherung »%s« nicht erfüllt.\n%n", assertion=0x7ffff58d7f1e "qn >= _quarters", file=0x7ffff58d7d47 "../libs/temporal/tempo.cc", line=471, function=<optimized out>) at ./assert/assert.c:92
#6 0x00007ffff3433596 in __GI___assert_fail
    (assertion=0x7ffff58d7f1e "qn >= _quarters", file=0x7ffff58d7d47 "../libs/temporal/tempo.cc", line=471, function=0x7ffff58d7eb0 "Temporal::superclock_t Temporal::TempoPoint::superclock_at(const Temporal::Beats&) const") at ./assert/assert.c:101
#7 0x00007ffff58936cd in Temporal::TempoPoint::superclock_at(Temporal::Beats const&) const (this=0x5555864279f0, qn=...)
    at ../libs/temporal/tempo.cc:471
0000008 0x00007ffff58b7551 in Temporal::TempoMetric::superclock_at(Temporal::Beats const&) const (this=0x7fffffffafa0, qn=...)
    at ../libs/temporal/temporal/tempo.h:480
0000009 0x00007ffff58a3978 in Temporal::TempoMap::get_grid(std::__cxx11::list<Temporal::TempoMapPoint, std::allocator<Temporal::TempoMapPoint> >&, long, long, unsigned int) const (this=0x5555844a6230, ret=std::__cxx11::list = {...}, start=282572606127, end=286108540200, bar_mod=0)
    at ../libs/temporal/tempo.cc:2001
0000010 0x00005555560b5ea2 in Editor::compute_current_bbt_points(std::__cxx11::list<Temporal::TempoMapPoint, std::allocator<Temporal::TempoMapPoint> >&, long, long) (this=0x555558391250, grid=std::__cxx11::list = {...}, leftmost=46659834, rightmost=48657915) at ../gtk2_ardour/editor_tempodisplay.cc:413
0000011 0x0000555556083994 in Editor::metric_get_bbt(std::vector<ArdourCanvas::Ruler::Mark, std::allocator<ArdourCanvas::Ruler::Mark> >&, long, long, int) (this=0x555558391250, marks=std::vector of length 0, capacity 256, lower=46659834, upper=48657915) at ../gtk2_ardour/editor_rulers.cc:1172
0000012 0x00005555560b62af in Editor::maybe_draw_grid_lines() (this=0x555558391250) at ../gtk2_ardour/editor_tempodisplay.cc:462
0000013 0x00005555560b8ab4 in Editor::mid_tempo_change(Editor::MidTempoChanges) (this=0x555558391250, what_changed=Editor::TempoChanged)
    at ../gtk2_ardour/editor_tempodisplay.cc:801
0000014 0x0000555555fa35fb in TempoMarkerDrag::motion(_GdkEvent*, bool) (this=0x5555876af7f0, event=0x7fffffffbc30, first_move=false)
    at ../gtk2_ardour/editor_drag.cc:3320
#15 0x0000555555f90a86 in Drag::motion_handler(_GdkEvent*, bool) (this=0x5555876af7f0, event=0x7fffffffbc30, from_autoscroll=false)
    at ../gtk2_ardour/editor_drag.cc:522
0000016 0x0000555555f8f7fa in DragManager::motion_handler(_GdkEvent*, bool) (this=0x5555583b6650, e=0x7fffffffbc30, from_autoscroll=false)
    at ../gtk2_ardour/editor_drag.cc:231
#17 0x000055555600db0d in Editor::motion_handler(ArdourCanvas::Item*, _GdkEvent*, bool)
    (this=0x555558391250, event=0x7fffffffbc30, from_autoscroll=false) at ../gtk2_ardour/editor_mouse.cc:2300
0000018 0x0000555555f86b57 in Editor::typed_event(ArdourCanvas::Item*, _GdkEvent*, ItemType)
     (this=0x555558391250, item=0x5555776cbbf0, event=0x7fffffffbc30, type=TempoMarkerItem) at ../gtk2_ardour/editor_canvas_events.cc:223
0000019 0x0000555555f89793 in Editor::canvas_tempo_marker_event(_GdkEvent*, ArdourCanvas::Item*, TempoMarker*)

paul

2022-10-30 18:01

administrator   ~0026765

At present, if you see ::get_grid() in the stacktrace, then more useful output is obtainable via running ardour from the cmdline with -D grid,temporalmap

paul

2022-10-31 01:36

administrator   ~0026771

should be fixed in git master, 9c2c08973d894

was easier than tiling!

gonsolo

2022-10-31 17:47

reporter   ~0026782

Not yet:

#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
0000001 __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff343bc46 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
0000004 0x00007ffff34227fc in __GI_abort () at ./stdlib/abort.c:79
0000005 0x00007ffff342271b in __assert_fail_base
    (fmt=0x7fffef6029b5 "%s%s%s:%u: %s%sZusicherung »%s« nicht erfüllt.\n%n", assertion=0x7ffff58d7f1e "qn >= _quarters", file=0x7ffff58d7d47 "../libs/temporal/tempo.cc", line=471, function=<optimized out>) at ./assert/assert.c:92
#6 0x00007ffff3433596 in __GI___assert_fail
    (assertion=0x7ffff58d7f1e "qn >= _quarters", file=0x7ffff58d7d47 "../libs/temporal/tempo.cc", line=471, function=0x7ffff58d7eb0 "Temporal::superclock_t Temporal::TempoPoint::superclock_at(const Temporal::Beats&) const") at ./assert/assert.c:101
#7 0x00007ffff589272d in Temporal::TempoPoint::superclock_at(Temporal::Beats const&) const (this=0x555577625950, qn=...)
    at ../libs/temporal/tempo.cc:471
0000008 0x00007ffff5893de8 in Temporal::TempoMetric::superclock_at(Temporal::BBT_Time const&) const (this=0x7fffffffb090, bbt=...)
    at ../libs/temporal/tempo.cc:667
0000009 0x00007ffff58a2be3 in Temporal::TempoMap::get_grid(std::__cxx11::list<Temporal::TempoMapPoint, std::allocator<Temporal::TempoMapPoint> >&, long, long, unsigned int) const (this=0x555588116670, ret=std::__cxx11::list = {...}, start=290064432769, end=345781174612, bar_mod=1)
    at ../libs/temporal/tempo.cc:2027
0000010 0x00007ffff58a57fe in Temporal::TempoMap::count_bars(Temporal::Beats const&, Temporal::Beats const&) const
    (this=0x555588116670, start=..., end=...) at ../libs/temporal/tempo.cc:2182
0000011 0x0000555556083303 in Editor::compute_bbt_ruler_scale(long, long) (this=0x5555583ad640, lower=43142526, upper=58760601)
    at ../gtk2_ardour/editor_rulers.cc:1078
0000012 0x0000555556081b99 in Editor::update_tempo_based_rulers() (this=0x5555583ad640) at ../gtk2_ardour/editor_rulers.cc:754
0000013 0x00005555560b8aa5 in Editor::mid_tempo_change(Editor::MidTempoChanges) (this=0x5555583ad640, what_changed=Editor::TempoChanged)
    at ../gtk2_ardour/editor_tempodisplay.cc:800
0000014 0x0000555555fa351f in TempoMarkerDrag::motion(_GdkEvent*, bool) (this=0x555565786e40, event=0x7fffffffbc10, first_move=true)
    at ../gtk2_ardour/editor_drag.cc:3320
#15 0x0000555555f909aa in Drag::motion_handler(_GdkEvent*, bool) (this=0x555565786e40, event=0x7fffffffbc10, from_autoscroll=false)
    at ../gtk2_ardour/editor_drag.cc:522
0000016 0x0000555555f8f71e in DragManager::motion_handler(_GdkEvent*, bool) (this=0x5555583d1720, e=0x7fffffffbc10, from_autoscroll=false)
    at ../gtk2_ardour/editor_drag.cc:231
#17 0x000055555600dab1 in Editor::motion_handler(ArdourCanvas::Item*, _GdkEvent*, bool)
    (this=0x5555583ad640, event=0x7fffffffbc10, from_autoscroll=false) at ../gtk2_ardour/editor_mouse.cc:2300
0000018 0x0000555555f86a7b in Editor::typed_event(ArdourCanvas::Item*, _GdkEvent*, ItemType)
    (this=0x5555583ad640, item=0x5555777ac030, event=0x7fffffffbc10, type=TempoMarkerItem) at ../gtk2_ardour/editor_canvas_events.cc:223
0000019 0x0000555555f896b7 in Editor::canvas_tempo_marker_event(_GdkEvent*, ArdourCanvas::Item*, TempoMarker*)
    (this=0x5555583ad640, event=0x7fffffffbc10, item=0x5555777ac030, marker=0x5555777ab460) at ../gtk2_ardour/editor_canvas_events.cc:1095

I attached the output of "-D grid,temporalmap"

One tile down.
debug_output.gz (79,887 bytes)

paul

2022-10-31 18:09

administrator   ~0026783

Could you repeat that with Edit > Preferences > Performance > processor usage set to 1 processor ? The debug output is unreadable when multithreaded ... thanks

Alternatively, let me know the steps to make this happen (your existing instructions no longer crash for me)

gonsolo

2022-10-31 18:46

reporter   ~0026787

Output attached (with set to 1 processor).

Session: https://drive.google.com/file/d/1cUZyxJnVvgjXUO5DhSBd15JnknysQJK2/view?usp=sharing

1. Open session
2. Drag the tempo marker at bar 306 around.

gonsolo

2022-10-31 18:50

reporter   ~0026788

Debug output with setting processors to 1, restarting, and making sure that processor is set to 1.
debug_output-2.gz (2,826 bytes)

paul

2022-10-31 19:07

administrator   ~0026790

and this was generated/triggered how?

gonsolo

2022-10-31 20:47

reporter   ~0026794

Open the session and drag the tempo marker around.

gonsolo

2022-11-07 10:00

reporter   ~0026862

Also note how Ardour doesn't respect the 4/4 change at bar 298.
Also the time change is in the middle of bar 306 which should not be the case.
That is still the session above.
bars.png (28,829 bytes)   
bars.png (28,829 bytes)   

gonsolo

2022-11-07 10:01

reporter   ~0026863

https://tracker.ardour.org/view.php?id=9067 seems to be related as it also crashes at tempo.cc:471.

paul

2022-11-22 01:26

administrator   ~0026929

archive no longer readable. would like to check if f5887b978d4 fixes this issue

gonsolo

2022-11-22 07:21

reporter   ~0026931

The session in https://tracker.ardour.org/view.php?id=9049#c26787 is essentially the same (with processor set to 1 for debugging).

I opened the session and dragged said marker around for half an hour now (slightly exaggerating) and couldn't get Ardour to crash so it seems to be fixed.

Issue History

Date Modified Username Field Change
2022-10-29 16:27 gonsolo New Issue
2022-10-29 17:05 paul Note Added: 0026752
2022-10-29 17:39 gonsolo Note Added: 0026753
2022-10-29 17:43 paul Note Added: 0026754
2022-10-30 10:48 gonsolo Note Added: 0026756
2022-10-30 13:34 paul Note Added: 0026757
2022-10-30 14:49 paul Note Added: 0026758
2022-10-30 14:51 paul Note Edited: 0026758
2022-10-30 16:23 paul Note Added: 0026761
2022-10-30 16:25 paul Note Added: 0026762
2022-10-30 16:40 gonsolo Note Added: 0026763
2022-10-30 16:40 gonsolo File Added: wrong.png
2022-10-30 17:13 gonsolo Note Added: 0026764
2022-10-30 18:01 paul Note Added: 0026765
2022-10-31 01:36 paul Note Added: 0026771
2022-10-31 17:47 gonsolo Note Added: 0026782
2022-10-31 17:47 gonsolo File Added: debug_output.gz
2022-10-31 18:09 paul Note Added: 0026783
2022-10-31 18:46 gonsolo Note Added: 0026787
2022-10-31 18:50 gonsolo Note Added: 0026788
2022-10-31 18:50 gonsolo File Added: debug_output-2.gz
2022-10-31 19:07 paul Note Added: 0026790
2022-10-31 20:47 gonsolo Note Added: 0026794
2022-11-07 10:00 gonsolo Note Added: 0026862
2022-11-07 10:00 gonsolo File Added: bars.png
2022-11-07 10:01 gonsolo Note Added: 0026863
2022-11-20 06:01 x42 Relationship added related to 0009067
2022-11-22 01:26 paul Note Added: 0026929
2022-11-22 07:21 gonsolo Note Added: 0026931