Immediate Fix
To stabilize AWS S3 transfers on flaky WiFi or high-latency networks, the fastest solution is to tune the AWS CLI retry logic and multipart upload settings. These adjustments allow the CLI to handle temporary drops without restarting the entire transfer.
Run the following commands to increase the retry attempts and optimize the concurrency for unstable connections:
# Increase retry attempts for network timeouts
aws configure set default.s3.max_attempts 10
# Reduce concurrent requests to prevent bandwidth saturation
aws configure set default.s3.max_concurrent_requests 5
# Lower the multipart threshold to 16MB for more granular recovery
aws configure set default.s3.multipart_threshold 16MBBy lowering the multipart_threshold, the AWS CLI breaks large files into smaller chunks earlier. If a WiFi drop occurs, only the specific failed 16MB chunk needs to be re-uploaded rather than a massive 5GB file.
Technical Explanation
Unstable networks usually suffer from two main issues: packet loss and high jitter. When your WiFi drops, the TCP connection underlying the S3 transfer breaks. Without proper configuration, the AWS CLI might reach its default retry limit too quickly and terminate the process.
AWS S3 uses “Multipart Uploads” to handle large objects. This process splits a file into parts, uploads them independently, and reassembles them at the destination. On an unstable network, smaller parts are more likely to succeed during “bursts” of connectivity.
| Parameter | Standard Network | Unstable/WiFi |
|---|---|---|
| max_concurrent_requests | 10 | 3 – 5 |
| max_attempts | 5 | 10 – 20 |
| multipart_chunksize | 8MB | 16MB – 32MB |
Using max_concurrent_requests with a lower value prevents “Bufferbloat,” where too many simultaneous connections saturate a weak upload link, causing even more packets to drop. It creates a more “conservative” but resilient data stream.
Alternative Methods
1. S3 Transfer Acceleration
If your local network is stable but the route to the AWS Region is poor, enable S3 Transfer Acceleration. This utilizes Amazon’s global backbone network. You can enable it via the console and then use the --use-accelerate-endpoint flag in your commands.
aws s3 cp large-file.zip s3://my-bucket/ --endpoint-url https://s3-accelerate.amazonaws.com2. AWS DataSync
For persistent network issues or massive datasets, AWS DataSync is superior to the CLI. It includes built-in retry mechanisms, data validation via checksums, and bandwidth throttling to ensure it doesn’t kill your local internet connection.
3. Using S3 Sync instead of CP
Always use aws s3 sync instead of aws s3 cp for directories. If the transfer fails halfway through, running the sync command again will only upload the files that are missing or have changed, saving you from re-uploading completed data.