QUERY · ISSUE

ffi callbacks are not collected

openby abiliojropened 2023-01-26updated 2023-01-26

bug

While working with the UNIX port, I realized that my libcurl wrapper was causing a "memory leak". Upon further investigation, I managed to reduce the code to reproduce the issue to:

import ffi
import gc

def cb():
    pass

for i in range(1000000):
    ffi.callback("v", cb, "") # this is never collected
    if (i % 1000) == 0:
        gc.collect()
        print(gc.mem_alloc())

You will see the counter increasing until it runs out of memory.

By negating this condition:
#if MICROPY_FORCE_PLAT_ALLOC_EXEC

In file ports/unix/alloc.c The problem goes away. The function ffi_closure_free has an interesting 'TODO' comment that's been sitting there for 7 years.

The comment in this conditional guarded section reads save a lot of code size. How much is a lot?

Micropython version: 1.19
Board: x64 and ARM64 (Linux)

CANDIDATE · ISSUE

modffi callbacks may get garbage collected

openby stinosopened 2021-05-17updated 2021-05-19

bugport-unix

As reported on the forum, this code on MIPS32 with setjmp for collecting registers seems to result in the bound method which gets used as callback in ffi to get garbage collected:

import ffi
import uctypes
import gc
gc.threshold(50 * 1024) # make it crash faster
try:
    libc = ffi.open("libc.so.1")
except:
    libc = ffi.open("libc.so.6")
qsort = libc.func("v", "qsort", "piiC")
class ClassCmp:
    def cmp(self, pa, pb):
        a = uctypes.bytearray_at(pa, 1)
        b = uctypes.bytearray_at(pb, 1)
        return a[0] - b[0]
c = ClassCmp()
cmp_cb = ffi.callback("i", c.cmp, "pp")
# instead, store c.cmp so it doesn't get collected: no crash
# k = c.cmp
# cmp_cb = ffi.callback("i", k, "pp")
s = b"foobar"
while True:
    qsort(s, len(s), 1, cmp_cb)
    a = b"5" * 1021 # use some memory, eventually overwriting c.cmp passed to cmp_cb

I didn't look into this in detail, but from looking at modffi.c I suspect this might be an issue in MicroPython itself, with the following reasoning: mod_ffi_callback in modffi.c takes the MicroPython function object (func_in argument in piece of code below) and then passes it into FFI without storing it anywhere else:

mp_obj_fficallback_t *o = m_new_obj_var(mp_obj_fficallback_t, ffi_type *, nparams);
...
o->clo = ffi_closure_alloc(sizeof(ffi_closure), &o->func);
...
res = ffi_prep_closure_loc(o->clo, &o->cif, call_py_func, MP_OBJ_TO_PTR(func_in), o->func);

Since I don't immediately see anything which guarantees that what FFI allocates is reachable by the GC, it's normal that callback gets GC'd because the pointer is only stored somewhere in a piece of memory allocated by FFI (hence, I assume, from the C heap). However I don't understand why I cannot reproduce this on the unix x64 port.

I suggested the following patch to fix this and this works for MIPS:

typedef struct _mp_obj_fficallback_t {
    mp_obj_base_t base;
    void *func;
    mp_obj_t py_func;  //add this one
    ffi_closure *clo;
    char rettype;
    ffi_cif cif;
    ffi_type *params[];
} mp_obj_fficallback_t;

and in mod_ffi_callback assign the object like

o->py_func = func_in;

Also: there is no matching ffi_closure_free call so it looks like the memory allocated by ffi_closure_alloc leaks?

ffi callbacks are not collected

modffi callbacks may get garbage collected

Keyboard