Skip to content

Execution backends

Backends control where and how ListMapper executes map/filter/foreach/reduce.

Default: SerialBackend (regular in-process execution).

Available backends

  • SerialBackend: sequential, pure Python
  • LocalBackend: serial, threads, processes
  • in mode="serial", LocalBackend can optionally use Cython-compiled C extensions (if built into the wheel)
  • AsyncBackend: concurrent execution with asyncio (async functions)
  • RayBackend: distributed execution with Ray (optional)
  • supports chunk_size to batch items per Ray task (recommended for benchmarks)
  • DaskBackend: distributed execution with Dask (optional)
  • supports chunk_size to batch items per Dask task (recommended for benchmarks)

Choosing a backend

SerialBackend vs LocalBackend(serial)

  • SerialBackend is always pure Python.
  • LocalBackend(mode="serial") uses the same semantics, but calls internal helpers from functional_list.accelerators.
  • If optional Cython extensions are available, those helpers run as compiled C code.
  • Otherwise, LocalBackend falls back to pure Python.

1) Per call

from functional_list import ListMapper
from functional_list.backend import LocalBackend

out = ListMapper[int](1,2,3).map(lambda x: x*x, backend=LocalBackend(mode="threads", workers=4))

2) Global default

from functional_list import ListMapper
from functional_list.backend import LocalBackend

ListMapper.set_default_backend(LocalBackend(mode="processes", workers=4))

3) Temporary (context manager)

from functional_list import ListMapper
from functional_list.backend import LocalBackend

with ListMapper.using_backend(LocalBackend(mode="threads", workers=8)):
    out = ListMapper[int](1,2,3).filter(lambda x: x>1)

Notes

  • processes: the function must be pickleable.
  • async: requires async def functions.
  • ray/dask: optional dependencies + cluster initialization.
  • ray/dask performance: for small per-item functions, use chunk_size to reduce scheduling overhead.