Performance issue with BlobFuse 2.5.1 #2062
Description
Hello,
We are using Blobfuse to access Azure Blob Storage from an Azure VM (Ubuntu 22.04) with the latest patches installed.
After upgrading from Blobfuse 2.5.0 to 2.5.1, we observed a significant performance degradation over time as the cache size increases.
Rolling back to 2.5.0 immediately resolved the issue.
Observed Behavior
After some time, multiple processes like the following start appearing:
/usr/bin/du -sh /mnt/resource/blobfusetmp/...
The directory /mnt/resource/blobfusetmp is located on the Azure SSD temporary disk attached to the VM (i.e., /mnt/resource).
The execution time of the du command increases with the cache size:
~2 seconds for a 500 MB cache
~4 seconds for a 1.2 GB cache
However, the total performance degradation observed in our application is much higher than just the du execution time.
Our process reads and writes files (it converts XML files to PDFs), and the slowdown becomes very noticeable as the cache grows.
Configuration
Below is the configuration we are using:
#!/bin/sh
# Refer ./setup/baseConfig.yaml for the full set of config parameters
# Common configurations
allow-other: true
nonempty: true
logging:
type: syslog
level: log_warning
components:
- libfuse
- file_cache
- attr_cache
- azstorage
libfuse:
attribute-expiration-sec: 120
entry-expiration-sec: 120
negative-entry-expiration-sec: 240
file_cache:
path: /mnt/resource/blobfusetmp/dm-balance
timeout-sec: 120
max-size-mb: 4096
attr_cache:
timeout-sec: 7200
azstorage:
type: block
account-name: XXXXXXXXX
account-key: XXXXXXXXXX
endpoint: XXXXXXXXXX
mode: key
container: XXXXXXXX
Expected Behavior
Blobfuse performance should remain stable over time, independent of cache growth, as observed in version 2.5.0.
Actual Behavior
Performance gradually degrades as the cache grows.
Frequent /usr/bin/du calls on the cache path consume increasing time.
Overall file access latency becomes unacceptable for read/write workloads.
Environment
OS: Ubuntu 22.04 (fully patched)
Blobfuse version: 2.5.1 (regression not present in 2.5.0)
Storage type: Azure Block Blob
VM Disk: Azure temporary SSD (/mnt/resource)
AdditionalNotes
It appears that Blobfuse 2.5.1 may be performing more frequent or heavier cache size checks (e.g., via du), leading to cumulative overhead as the cache grows.
Any guidance or confirmation from the team on whether this behavior is expected, or if a fix/workaround exists, would be appreciated.