Troubleshooting

This guide covers common issues and their solutions when working with DittoFS.

Table of Contents

• Connection Issues
• Mount Issues
  • SMB mount permission denied (macOS)
• Permission Issues
• File Handle Issues
• Performance Issues
• Cross-Protocol Issues
• Logging and Debugging

Connection Issues

Cannot mount: Connection refused

Symptoms:

mount.nfs: Connection refused

Solutions:

1. Check if DittoFS is running:

   ps aux | grep dfs

2. Verify the port is correct:

   netstat -an | grep 12049
   # or
   lsof -i :12049

3. Check firewall rules:

   # Linux
   sudo iptables -L | grep 12049
   
   # macOS
   sudo pfctl -s rules | grep 12049

4. Verify configuration:

   # Check the config file
   cat ~/.config/dfs/config.yaml
   
   # Start with debug logging
   DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start

Connection timeout

Symptoms:

mount.nfs: Connection timed out

Solutions:

1. Check network connectivity:

   ping localhost
   telnet localhost 12049

2. Review timeout settings in config:

   adapters:
     nfs:
       timeouts:
         read: 5m
         write: 30s
         idle: 5m

Mount Issues

Invalid file system

Symptoms:

mount: /mnt/nfs: invalid file system.

Cause: The mount point directory does not exist.

Solution: Create the mount point before mounting:

# Linux
sudo mkdir -p /mnt/nfs
sudo mount -t nfs -o tcp,port=12049,mountport=12049 localhost:/export /mnt/nfs

# macOS (use /tmp since /mnt doesn't exist, sudo not required)
mkdir -p /tmp/nfs
mount -t nfs -o tcp,port=12049,mountport=12049 localhost:/export /tmp/nfs

Permission denied when mounting

Symptoms:

mount.nfs: access denied by server while mounting

Solutions:

1. On Linux, allow non-privileged ports:

   sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0

2. On macOS, use resvport option:

   sudo mount -t nfs -o tcp,port=12049,mountport=12049,resvport localhost:/export /mnt/test

3. Check export configuration:

   shares:
     - name: /export
       allowed_clients:
         - 192.168.1.0/24  # Make sure your IP is in this range
       denied_clients: []

4. Verify authentication settings:

   shares:
     - name: /export
       require_auth: false  # Set to false for development
       allowed_auth_methods: [anonymous, unix]

No such file or directory

Symptoms:

mount.nfs: mounting localhost:/export failed, reason given by server: No such file or directory

Solutions:

1. Verify the export path exists in configuration:

   shares:
     - name: /export  # This is the export path

2. Check share names are correct:

   # Mount using the exact share name from config
   sudo mount -t nfs -o nfsvers=3,tcp,port=12049,mountport=12049 localhost:/export /mnt/test

SMB mount permission denied (macOS)

Symptoms:

zsh: permission denied: /tmp/smb-test/file.txt

This happens after mounting an SMB share, even with 0777 permissions.

Cause: macOS has a security restriction where only the mount owner can access files, regardless of Unix permissions. This is enforced at a level below file permissions - no SMB traffic even reaches the server. Apple confirmed this is “works as intended”.

Solution - use dfsctl (handles this automatically):

The dfsctl share mount command automatically handles this by using sudo -u $SUDO_USER to mount as your user instead of root:

# This works - mount owned by your user, not root
sudo dfsctl share mount --protocol smb /export /mnt/share

Alternative - mount to user directory without sudo:

mkdir -p ~/mnt/share
dfsctl share mount --protocol smb /export ~/mnt/share

If using manual mount_smbfs:

# Mount as your user, not root
sudo -u $USER mount_smbfs //user:pass@localhost:12445/export /mnt/share

Note: This is a macOS-specific issue. On Linux, dfsctl share mount uses uid/gid options which work correctly.

See Known Limitations for details.

Permission Issues

Permission denied on file operations

Symptoms:

touch: cannot touch 'file.txt': Permission denied

Solutions:

1. Check identity mapping configuration:

   shares:
     - name: /export
       identity_mapping:
         map_all_to_anonymous: true  # Try this for development
         anonymous_uid: 65534
         anonymous_gid: 65534

2. Verify root directory permissions:

   shares:
     - name: /export
       root_attr:
         mode: 0777  # Wide open for debugging
         uid: 0
         gid: 0

3. Check your client UID/GID:

   id  # Check your UID and GID

4. Enable debug logging to see auth context:

   DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start
   # Look for lines showing UID/GID in requests

Read-only filesystem

Symptoms:

touch: cannot touch 'file.txt': Read-only file system

Solutions:

1. Check share configuration:

   shares:
     - name: /export
       read_only: false  # Must be false for writes

2. Verify mount options:

   # Make sure you're not mounting with 'ro'
   sudo mount -t nfs -o nfsvers=3,tcp,port=12049,mountport=12049,rw localhost:/export /mnt/test

File Handle Issues

Stale file handle

Symptoms:

ls: cannot access 'file.txt': Stale file handle

Causes:

• Server was restarted with in-memory metadata (handles lost)
• File was deleted while client held a handle
• Metadata backend was changed

Solutions:

1. Unmount and remount the filesystem:

   sudo umount /mnt/test
   sudo mount -t nfs -o nfsvers=3,tcp,port=12049,mountport=12049 localhost:/export /mnt/test

2. For persistent handles, use BadgerDB metadata:

   ./dfsctl store metadata add --name persistent --type badger \
     --config '{"path":"/var/lib/dfs/metadata"}'
   ./dfsctl store block add --kind local --name default --type memory
   ./dfsctl share create --name /export --metadata persistent --local default

3. Clear client NFS cache (Linux):

   # This varies by distribution
   sudo service nfs-common restart

Performance Issues

Slow read/write operations

Diagnostics:

# Run benchmarks to identify bottleneck
./scripts/benchmark.sh --profile

# Check server logs for slow operations
tail -f ~/.config/dfs/dfs.log | grep -i "slow\|timeout"

Solutions:

1. Tune buffer sizes:

   metadata:
     global:
       filesystem_capabilities:
         max_read_size: 1048576   # 1MB
         max_write_size: 1048576  # 1MB

2. Use memory stores for development:

   ./dfsctl store metadata add --name fast --type memory
   ./dfsctl store block local add --name fast --type memory

3. For S3, verify configuration:

   ./dfsctl store block remote add --name s3-store --type s3 \
     --config '{"region":"us-east-1","bucket":"my-bucket"}'

High memory usage

Diagnostics:

# Profile memory usage
go test -bench=. -memprofile=mem.prof ./test/e2e/
go tool pprof mem.prof

Solutions:

1. Check for memory leaks in logs

2. Reduce max connections:

   adapters:
     nfs:
       max_connections: 100

3. Monitor metrics:

   server:
     metrics:
       enabled: true
       port: 9090

   Then visit http://localhost:9090/metrics

Logging and Debugging

Enable Debug Logging

Via environment variable:

DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start

Via configuration:

logging:
  level: DEBUG
  format: text  # or json
  output: stdout  # or file path

Understanding Log Output

Key log patterns:

• [INFO] NFS: Accepted connection from 127.0.0.1:54321 - Client connected
• [DEBUG] NFS: LOOKUP(handle=..., name=file.txt) - Operation details
• [ERROR] NFS: Failed to read file: no such file - Error conditions
• [DEBUG] Auth: UID=1000, GID=1000, GIDs=[1000,4,20] - Authentication context

Capture Traffic

For deep debugging, capture NFS traffic:

# Linux
sudo tcpdump -i lo -w nfs.pcap port 12049

# macOS
sudo tcpdump -i lo0 -w nfs.pcap port 12049

# Analyze with Wireshark
wireshark nfs.pcap

Check Server Health

# Check if server is responding
./dfs status

# Check metrics (if enabled)
curl http://localhost:9090/metrics

# Check configuration
DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start 2>&1 | grep -i "config"

Cross-Protocol Issues

When running both NFS and SMB adapters simultaneously, the shared LockManager coordinates caching state between protocols. This section covers common cross-protocol issues and their resolution.

File Locked by Another Protocol

Symptoms:

NFS: NFS4ERR_SHARE_DENIED or NFS4ERR_LOCKED
SMB: STATUS_SHARING_VIOLATION or STATUS_LOCK_NOT_GRANTED

Cause: An SMB client holds an exclusive lease (RWH) or byte-range lock on a file that an NFS client is trying to access, or vice versa.

Diagnosis:

# Check active locks and leases via debug logging
DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start
# Look for log entries containing:
#   "cross_protocol_break" - Break initiated across protocols
#   "lease_break" - SMB lease being broken
#   "delegation_recall" - NFS delegation being recalled

Resolution:

1. Wait for the lease/delegation break to complete (the LockManager automatically initiates breaks)
2. If the break times out (35s for SMB leases, 90s for NFS delegations), the server force-revokes and the operation proceeds
3. Check if the client holding the lock is still connected

Delegation Recall Timeouts

Symptoms:

[WARN] delegation recall timeout: delegID=abc123 client=nfs-client elapsed=90s

Cause: An NFS client did not respond to a CB_RECALL (callback recall) within the configured timeout (default 90 seconds). This happens when:

• The NFS client is unresponsive or has network issues
• The NFS client’s backchannel is broken
• The CB_RECALL message was lost

Resolution:

1. After timeout, the delegation is force-revoked and the conflicting operation proceeds
2. Check NFS client connectivity and backchannel health
3. Adjust timeout if needed:

   adapters:
     smb:
       cross_protocol:
         delegation_recall_timeout: 120s  # Increase from 90s default

4. Enable debug logging to see the recall flow:

   DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start 2>&1 | grep "delegation"

Lease Break Storms

Symptoms:

[INFO] cross_protocol_break: file=/export/data.bin protocol=smb->nfs type=lease_break count=47 period=60s

Rapid grant-break-grant-break cycles visible in logs.

Cause: Two clients (one NFS, one SMB) are alternately opening the same file with conflicting access modes, causing the LockManager to repeatedly break and re-grant caching state.

Resolution:

1. DittoFS has a built-in anti-storm cache (default 30-second TTL) that prevents re-grants after a break
2. If storms persist, increase the anti-storm TTL:

   adapters:
     smb:
       cross_protocol:
         anti_storm_ttl: 60s  # Increase from 30s default

3. Consider configuring the conflicting clients to use the same protocol for the shared file
4. Read-only access from both protocols does not cause storms (NFS read delegation + SMB Read lease coexist)

SMB Client Cannot Write to NFS-Delegated File

Symptoms:

• SMB WRITE or CREATE (write access) hangs for several seconds before succeeding
• Or fails with STATUS_SHARING_VIOLATION

Cause: An NFS client holds a write delegation on the file. The LockManager must recall the delegation via CB_RECALL and wait for the NFS client to return it before the SMB write can proceed.

Expected behavior: This is correct cross-protocol coordination. The delay is the delegation recall round-trip time (typically under 1 second for responsive NFS clients, up to 90 seconds for the timeout).

Resolution:

1. This is normal behavior — the SMB write will succeed once the NFS delegation is returned
2. If the delay is unacceptable, disable delegations for the share (reduces NFS performance)
3. Monitor recall latency in debug logs

NFS Client Sees Stale Data After SMB Write

Symptoms:

• NFS client reads file and gets old content after an SMB client has written new content

Cause: The NFS client cached the file data under a read delegation, and the delegation recall + cache invalidation has not completed yet.

Resolution:

1. DittoFS automatically sends CB_RECALL to NFS clients when an SMB client modifies a file
2. After the recall, the NFS client invalidates its cache and re-reads from the server
3. If the NFS client is not responding to recalls:
   • Check NFS client backchannel connectivity
   • After delegation recall timeout (90s), the delegation is force-revoked
   • NFS client’s next request will fail with EXPIRED or ADMIN_REVOKED, forcing a refresh
4. Force a cache refresh on the NFS client:

   # Remount with noac to disable attribute caching
   sudo mount -t nfs4 -o noac server:/export /mnt/nfs

Directory Listing Inconsistency Across Protocols

Symptoms:

• Creating a file via SMB, but NFS ls does not show it (or vice versa)

Cause: Directory change notifications have not yet broken the other protocol’s directory lease/delegation.

Resolution:

1. DittoFS automatically breaks directory leases and delegations when CreateFile, RemoveFile, or Rename modifies a directory
2. The break is processed through the LockManager and dispatched to both protocols
3. If inconsistency persists:
   • Verify directory leases are enabled: check leases.directory_leases: true
   • Check for notification queue overflow (1024 events/directory capacity)
   • Force a directory re-read: ls -la /mnt/nfs/dir/ or refresh the SMB directory listing

Diagnostic Commands

# Enable cross-protocol debug logging
DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start

# Key log patterns to search for:
# "cross_protocol_break" - Break initiated between protocols
# "delegation_recall" - NFS delegation being recalled
# "lease_break" - SMB lease being broken
# "anti_storm" - Grant suppressed by anti-storm cache
# "dir_change_notify" - Directory change notification dispatched
# "break_timeout" - Break acknowledgment timed out

Log Messages Reference

Log Message	Level	Meaning
cross_protocol_break	INFO	A caching break was initiated across protocols
delegation_recall	DEBUG	CB_RECALL sent to NFS client
delegation_returned	DEBUG	NFS client returned delegation voluntarily
delegation_force_revoked	WARN	Delegation revoked after timeout
lease_break	DEBUG	Lease break notification sent to SMB client
lease_break_ack	DEBUG	SMB client acknowledged lease break
anti_storm_suppressed	DEBUG	Grant suppressed by anti-storm cache
dir_change_notify	DEBUG	Directory change event queued
notification_overflow	WARN	Directory notification queue overflowed

Common Error Messages

”export not found”

Cause: The share name in the mount command doesn’t match configuration.

Solution: Check share names in config and use exact match:

# If config has "name: /export"
sudo mount -t nfs -o nfsvers=3,tcp,port=12049,mountport=12049 localhost:/export /mnt/test

“authentication failed”

Cause: Server requires authentication but client isn’t providing it.

Solution: Either disable authentication or configure it properly:

shares:
  - name: /export
    require_auth: false
    allowed_auth_methods: [anonymous, unix]

“metadata store not found”

Cause: Share references a non-existent metadata store.

Solution: Ensure stores exist before creating the share:

# Create the stores first
./dfsctl store metadata add --name my-store --type memory
./dfsctl store block local add --name my-blocks --type memory

# Then create the share referencing them
./dfsctl share create --name /export --metadata my-store --local my-blocks

NFSv4 Session Issues

Client cannot establish NFSv4 session

Symptoms:

mount.nfs4: Protocol not supported

Solutions:

1. Verify NFSv4 is enabled in the adapter settings:

   dfsctl adapter settings nfs

2. Check that the client supports NFSv4. Some older clients default to NFSv3.
3. Try mounting with explicit version:

   sudo mount -t nfs4 localhost:/export /mnt/test

Session expired or lost

Symptoms:

NFS4ERR_EXPIRED or NFS4ERR_STALE_CLIENTID

Solutions:

1. The server may have restarted, causing session loss. Remount the share.
2. Check the grace period status:

   dfsctl grace status

3. If in grace period, wait for it to complete before retrying.

Kerberos Authentication Issues

Kerberos authentication fails

Symptoms:

mount.nfs4: access denied by server

with RPCSEC_GSS configured.

Solutions:

1. Verify the keytab file is accessible:

   klist -k /etc/krb5.keytab

2. Check that the service principal matches the server hostname
3. Verify clock synchronization between client and KDC (Kerberos requires clocks within 5 minutes)
4. Enable debug logging to see the RPCSEC_GSS negotiation:

   DITTOFS_LOGGING_LEVEL=DEBUG ./dfs start

ACL Issues

ACL operations fail with NFS3ERR_NOTSUPP

Symptoms:

setfacl: Operation not supported

Solutions:

1. ACLs require NFSv4. Ensure you are mounting with NFSv4:

   sudo mount -t nfs4 localhost:/export /mnt/test

2. NFSv3 does not support ACLs - only standard POSIX mode bits.

Getting More Help

If you’re still experiencing issues:

1. Check existing issues: GitHub Issues
2. Enable debug logging and capture relevant output
3. Open a new issue with:
   • DittoFS version
   • Operating system and version
   • Configuration file (redact sensitive info)
   • Full error messages
   • Debug logs showing the problem
   • Steps to reproduce