feat: Add configurable S3Object deletion batch size #662
Conversation
|
@AMZN-bradhawk thanks for this contribution. Question, should we just set the default to be 1000 and omit the setting all together? Is there a reason to not just set to 1000 by default? If there is we can leave the setting even if we change the default to 1000. Thoughts? |
|
I believe the most efficient batch size would be 1000. I thought it might be something that users may want to configure. We could also merge the 2 thoughts - making the default 1000, but also allowing it to be configured for a lower value. This is just my testing data with batch size in the columns and total objects to delete on the left. Total seconds to delete the objects and the bucket containing them are the results. With larger quantities of objects, the benefit of a larger batch size grows. The simplest change would be to use S3's maximum batch size of 1000. This would also limit the number of API calls no matter the number of S3Objects being deleted. 
|
|
@AMZN-bradhawk I would say let's make the default 1000. I can't think of a reason to make it configurable, but since you already did the work, I would say let's just leave it. |
|
🎉 This PR is included in version 3.54.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
S3 has a maximum batch deletion for S3Objects of 1000. When clearing buckets with large quantities of S3Objects, a larger batch size improves speeds. In personal testing I've found that using a 1000 count batch size has improved the deletion of a S3 bucket containing 50,000 objects was reduced from about 8min to 2min.
This change makes the batch size for S3Object deletion configurable