Performance and caching

This guide explains how delta.sharing downloads and caches shared files, and how concurrency, materializer choice, and batch size affect reads. Start with vignette("delta-sharing") if you have not yet created a client and read a table.

A read has two distinct phases: staging files and reading rows. The package writes the files selected by the sharing server directly to its disk-backed session cache before opening an Arrow stream; complete downloaded files are not held in R memory. Eager materializers can still exhaust RAM by collecting the scan output. A lazy Arrow reader avoids that collection, but it does not avoid staging or stream remote Parquet files directly from their signed URLs.

The read lifecycle

Every snapshot or change read follows the same broad path:

1. R requests the relevant file actions from the Delta Sharing server.
2. Missing Parquet and deletion-vector files are downloaded into the table’s session cache.
3. R writes a new local Delta log that points to the cached files.
4. Delta Kernel scans the local files and emits Arrow batches.
5. An eager materializer, when used, collects those batches into its result.

The stages have different reuse behavior:

Work	Happens on another materializer call?	Reused from the session cache?
Sharing query and pagination	Yes	No
Download selected files	Only when missing or invalid	Yes
Build the local Delta log	Yes	No
Delta Kernel scan	Yes	No
Collect an eager result	Yes	No

Reusing cached files avoids downloading them again, but each materializer call still performs the Sharing request, local log construction, Delta Kernel scan, and result collection.

What the cache contains

Each table has a deterministic cache directory under R’s session temporary directory. Its identity includes the profile endpoint and the table’s share, schema, and name. New handles for the same table therefore share cached files, even when they use different download concurrency:

client <- sharing_client("~/config.share")

orders <- client$table("sales.default.orders", concurrency = 4)
faster_downloads <- client$table("sales.default.orders", concurrency = 8)

identical(orders$cache_path, faster_downloads$cache_path)
#> TRUE

Snapshot and change data feed reads for the same table use this shared cache. When the server selects the same underlying file for both reads, its local copy is reused. Files that do not overlap are downloaded normally.

The cache contains local copies of selected Parquet and deletion-vector files. Files are identified using immutable IDs supplied by the sharing server, so they can be reused even after a signed download URL changes. When the server provides a file size, the package checks it before reusing the cached file.

Downloads are completed in temporary files before becoming available to the cache, so interrupted transfers are not reused. Cache directories and files are created with user-only permissions.

Cache lifetime

The cache lasts for the R session, not for the lifetime of a table handle. It lives under tempdir(), R’s per-session temporary directory. It is normally removed when the session ends, but it is not durable storage: abnormal termination may leave files behind, and system temporary-directory policies may remove them during a long-running session. See ?tempdir for how R chooses the directory.

The cache is not persistent across sessions and does not contain materialized tibbles or Arrow tables.

The read-only cache_path property is available for inspection:

orders$cache_path
fs::dir_info(orders$cache_path)

After all lazy readers and streams for the table have been closed, the directory can be removed to reclaim disk space:

fs::dir_delete(orders$cache_path)

The next read recreates the directory and downloads any required files. A long session that touches many large tables can retain substantial data, so cache_path is also useful for monitoring local disk usage.

Download concurrency

Set concurrency when creating a table handle:

orders <- client$table(
  "sales.default.orders",
  concurrency = 8
)

The default is four. concurrency controls the maximum number of missing files downloaded at once. It does not affect Sharing API pagination, Delta Kernel scanning, Arrow batch size, or reads whose files are already cached.

Higher concurrency is most useful when a query selects many files and network latency leaves the connection idle. It may make little difference when there are few files, a single file saturates the connection, or the provider throttles requests. Benchmark representative uncached reads before increasing it.

Interactive reads show progress for missing downloads. When every file has a known size, the display also shows the total bytes to download. Cache hits are not counted as downloads.

Shape the read before tuning it

Query options affect different stages:

• columns reduces local scan and materialization work, but not network transfer or cache disk usage. The sharing server selects files before Delta Kernel applies the projection, and the complete selected Parquet files are downloaded.
• limit is sent to the server as a hint and is enforced exactly by Delta Kernel. A provider may use the hint to return fewer files, but this is not guaranteed.
• predicate is a best-effort server hint. It may improve file pruning, but it is not an exact row filter; apply an exact filter in the downstream consumer.

Choose a materializer

Materializers differ in how they consume Arrow batches:

Method	Collection behavior	Typical use
to_tibble()	Eager, in R memory	Ordinary R analysis
to_data_frame()	Eager, via the tibble path	Base data-frame consumers
to_arrow()	Eager, in Arrow memory	Repeated Arrow or DuckDB scans
to_arrow_reader()	Lazy after file staging	One-pass Arrow or DuckDB scan
to_arrow_stream()	Low-level and lazy after staging	Arrow C Stream consumers

Arrow is a required dependency. Eager R materializers use Arrow’s converter and return BIGINT columns as bit64::integer64. Close an Arrow reader when it is no longer needed, and release a low-level stream if its consumer does not take ownership.

Query larger-than-memory results with DuckDB

When the analysis can be expressed in SQL, DuckDB is usually simpler than processing Arrow batches manually. DuckDB can spill larger-than-memory operations to disk, while delta.sharing keeps the selected source files in its disk-backed session cache. This requires the optional DBI, duckdb, and withr packages.

Register a lazy Arrow reader so the full table is not first collected in R:

reader <- orders$snapshot(
  columns = c("status", "amount")
)$to_arrow_reader()

con <- DBI::dbConnect(duckdb::duckdb())
duckdb::duckdb_register_arrow(con, "orders", reader)

summary <- withr::with_options(
  list(arrow.use_threads = FALSE),
  DBI::dbGetQuery(con, "
    SELECT
      status,
      count(*) AS orders,
      sum(amount) AS total_amount
    FROM orders
    GROUP BY status
    ORDER BY status
  ")
)

duckdb::duckdb_unregister_arrow(con, "orders")
DBI::dbDisconnect(con)

Let Arrow manage the registered reader’s lifetime. Calling reader$Close() directly can release its stream while the scanner is still reading ahead, including after a SQL query returns early (for example, with LIMIT).

Arrow registration does not copy the full source into DuckDB. Only the final query result returned by dbGetQuery() is collected into R memory. Use SQL to filter or aggregate the data to a manageable result; collecting a very large final result can still exhaust R’s memory.

The local Arrow option disables parallel scan execution; background read-ahead can still occur. DuckDB can execute the rest of the query in parallel.

See DuckDB’s larger-than-memory documentation and its Arrow registration reference for details and current limitations.

Arrow batch size

batch_size controls the maximum number of rows emitted in each Arrow batch. The default is 65,536 rows. It does not control the number or size of downloaded files.

Smaller batches can reduce peak memory while a batch is being processed, at the cost of more conversion and boundary overhead. Larger batches can reduce that overhead but use more memory. An eager materializer still collects every batch, so reducing batch_size does not make to_tibble(), to_data_frame(), or to_arrow() safe for a result larger than memory.

Change the default only after measuring a representative workload.

A practical tuning order

1. Use columns to reduce local scan and materialization work. Use limit and predicates as server-side pruning hints, while respecting their exact and best-effort semantics.
2. Choose the execution path for the result size: eager R materialization when it fits comfortably in memory, or DuckDB for larger-than-memory SQL analytics.
3. Reuse an existing result when the same data is needed again. Let the session cache handle overlapping reads automatically.
4. If downloading uncached files is the bottleneck, compare a small number of concurrency values on a representative read.
5. Tune batch_size last and only when measurements show batch overhead or memory pressure.

Measure end-to-end elapsed time, peak memory, and cache disk usage. Network throughput, latency, provider behavior, file count, compression, schema width, query shape, and local hardware all affect performance, so there is no universally optimal configuration.