Changed the title, as this bug appears more broad, and affects some or all objects with finalizers. Some more cases that can easily repro:

1000.do { StackLayout() };
1000.do { VLayout() };
1000.do { Download() };
1000.do { QPalette() }; // the leak hear appears negligible or non-existent - a clue?

There are two problems that may be at fault here:

1. Finalizers are not being called often enough, resulting in a build-up of memory of e.g. QT objects that appear small to SuperCollider, but are in fact larger.
2. Finalizers are not being called at all.

My observations:

• I see many finalizers being created in the above cases via InstallFinalizer -> NewFinalizer
• NewFinalizer is never calling Collect (inRunCollection is always false). As a quick experiment, forcing it to Collect did not immediately resolve the problem.
• NewFinalizer seems to be hitting the mNewPool.Alloc branch often in the above cases, even though I would expect free'd blocks would be available to re-use.
• The loop in ScanFinalizers is rarely hit. I can put a breakpoint in it and see no hits for many many iterations of the reproducing code.
• In all but a few situations, I NEVER see QObjects deleted via a finalizer apart from in RunAllFinalizers, which occurs when the lang is restarted.

Remaining questions:

• Is there something preventing these objects from being collected at all? (e.g. are they somehow reachable?)
• Is something cleaning up white objects or flipping the color WITHOUT calling ScanFinalizers?
• Do we need to trigger ScanFinalizers calls more often, considering the memory pressure due to external objects with finalizers might be significantly greater than we are accounting for.
• For the cases were we are seeing finalizers hit BEFORE RunAllFinalizers (e.g. lang shutdown) - how are these finalizers being hit? Why do we see some, but we can allocation e.g. 10000 and see nothing?

Is this only related to Qt? While doing some benchmarking on stuff that merely created Routines, I've noticed regular sclang crashes and slowdown occurring (before the crashes), presumably due to GC pressure.

The kind of testing I was doing. After about a dozen of those in a row sclang was usually gone. Actually with some others it crashed even faster (fewer runs needed, probably because more Routine-creating reps in the actual bench).

bench { 10000 do: {{: [x,y], x<-(:3..5), y<-(:4..6), x !=y }.all} }

Just a note: I tried about 30-40 executions of @eleses code above. No crashes or obvious memory issues.

@scztt's examples seem to leak as before.

SuperCollider 3.11.0 Qt version 5.13.2

Just wanted to float an alternative idea...

what I have been able to get working is a wrapper around std::shared_ptr<void>.

I've got a working branch but here is the gist...

Create a new type of obj (omitting all the registering code, but importantly a new class is made).

struct PyrRCResource : public PyrObjectHdr {
    std::shared_ptr<void> ptr;
};

When objects are recycled in Pyr::Allocate, check their class, if it is class_rc_resource then call the destructor.
You can't call the destructor when freeing the object because Flip does this without iterating through them.

 PyrObject* obj = (PyrObject*)gcs->mFree;
    if (!IsMarker(obj)) {
        // from free list
        gcs->mFree = obj->next;
        assert(obj->obj_sizeclass == sizeclass);

        if (obj->classptr == class_rc_resource) {
            std::cout << "freeing a rc_resouce\n";
            auto resource = reinterpret_cast<PyrRCResource*>(obj);
            std::destroy_at(resource); // calls destructor
        }

New PyrRCResources are made like this.

PyrRCResource* newPyrRCResource(class PyrGC* gc, std::shared_ptr<void> ptr) {
    constexpr auto sz = sizeof(std::shared_ptr<void>);
    PyrRCResource* obj = gc == nullptr
        ? (PyrRCResource*)PyrGC::NewPermanent(sz, 0, obj_rc_resource)
          : (PyrRCResource*) gc->New(sz, 0, obj_rc_resource, true);
    obj->classptr = class_rc_resource;
    obj->size = 0;
    new (&obj->ptr) std::shared_ptr<void>(ptr);
    return obj;
}

I've tested this with a little lifetime struct, printing when the constructor and destructor gets called, along with the total in existence.

This means the shared_ptr will be free'd at some point in the future when the free list get to them.

... if I spam them in a loop, my system will have about 250 of these alive at anyone time. No segfaults or double frees so far, but I've still got copying to sort out, but that should be easy (fingers crossed).

A few issues here, if I'm understanding your proposal correctly:

When objects are recycled in Pyr::Allocate, check their class, if it is class_rc_resource then call the destructor.

This only frees resources when a new object is allocated in the slot, if I'm reading it correctly - NOT when the original object is destroyed. So the resources are leaked until you re-use that slot in the allocation pool, which will be in the future / could easily be never. If we want to free resources, why not do this when the object is freed rather than on the next allocation? I'm also not sure what kinds of guarantees we have that obj->classptr is valid in this case, since we just pulled this memory out of a pool of freed memory.

If the goal is to free some non-sclang resources when an object is destroyed, we already have a mechanism for this - Finalizers, they just seem partly broken :)

Another consideration here:

• if a resource is ONLY used by sclang, then we already have mechanism for tracking lifetime - the garbage collector. We definitely don't want to build another mechanism on top GC, especially something like shared_ptr's which have some significant limitations (specifically, they won't handle circular references).
• If a resource needs to live longer than the associated SC object because e.g. it is needed by an external C++ system, we have two choices:

1. We can allocate using an external mechanism (e.g. new rather than sclang's allocation), and manage lifetime with a shared_ptr or similar. The sclang object carries one shared_ptr instance and frees it in a finalizer. The external system needs to be able to handle a shared_ptr resource - for most cases we care about, these are QT systems, so a QSharedPointer would be the right construct.
2. We allocate the memory from sclang's pool as a normal sclang object (whose contents are opaque), and lifetime is handled using our normal GC. Because our external system (e.g. QT) needs to potentially own an object longer than any sclang object owns it, it needs to somehow track the PyrObject - we could just stick our resource object in a hidden list, e.g. "Kernel.externallyHeldObjects" to ensure it isn't collected - and used a shared_ptr / QSharedPointer object that removes the item from Kernel.externallyHeldObjects when it's ref count == 0.

I have an intuition that option #2 is the best one, since it keeps things working within the same system (SuperColliders GC and alloc pool) as much as possible - and, I've seen this exact mechanism for tracking lifetime when interfacing with outside systems in other scripting languages, which is a clue that it may solve some non-obvious problems.

We have three or four different ways that external resourecs are tracked in sclang - there's some very worthy clean-up that could be done here that would I'm sure fix a bunch of bugs and ugly code. But I think first we need to figure out if we have finalizer problems and fix them.

@scztt thank you for this! Yes you're right I didn't really consider that the memory might never be recycle, duh!

I thought this might be cleaner because I spent quite a while trying to find the issue with finalizers and qt and couldn't. There was the double free issue when copying (the mess I suggested was supposed to address that), and another issue that I couldn't pin down.

Your second suggestion there seems really clean and simple!