What happened + What you expected to happen

I started a fastapi app and an interface continuously created remote tasks.

I use memray tool get memory leak report:

After 1w+ request "/tasks":
0x1 56.3 MiB malloc 1239346 invocation at /usr/local/lib/python3.10/site-packages/ray/remote_function.py:411

After 2w+ request "/tasks":
0x27 150.3 MiB malloc 3198029 invocation at /usr/local/lib/python3.10/site-packages/ray/remote_function.py:411

Versions / Dependencies

ray version == 2.7.1, and ray cluster run in docker.

Reproduction script

Here is my fastapi.py

@ray.remote
def run_task(entrypoint, tid, metadata):
    entrypoint += f" --job='{json.dumps(metadata)}'"
    cmd = shlex.split(entrypoint)
    logging.info(f"run {cmd}")
    with Popen(args=cmd) as p:
        ...

@app.post("/tasks/")
async def fop_jobs(req: FopReq):
    a = run_fop.options(
        num_cpus=req.entrypoint_num_cpus,
        num_gpus=req.entrypoint_num_gpus,
        runtime_env=req.runtime_env
    ).remote(req.entrypoint, req.tid, req.metadata)
    return req

if __name__ == "__main__":
    uvicorn.run(app, host='0.0.0.0', port=8000)

Issue Severity

High: It blocks me from completing my task.