Boto3 download file to io

Background I am using the following Boto3 code to download file from S3. This code works fine when uploaded outside any folder. Query Any thoughts on how to resolve this error? David Wesby 41 3 3 silver badges 6 6 bronze badges. Rohan Rohan 1 1 gold badge 6 6 silver badges 18 18 bronze badges. Add a comment. Active Oldest Votes. So code above could be rewritten to Andriy Ivaneyko Andriy Ivaneyko 17k 4 4 gold badges 48 48 silver badges 68 68 bronze badges.

Is this causing the error now AndriyIvaneyko I am facing the same issue. The link you have provided are old. Can you help me in solving the issue? Seems it is a boto3 thing — Tinkerbell.

Show 1 more comment. Yankee Yankee 1, 2 2 gold badges 21 21 silver badges 42 42 bronze badges. Using this following code i am able to accomplish my task. Apoorv Maheshwari Apoorv Maheshwari This was the answer I was trying to find all along!

Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Who owns this outage? Building intelligent escalation chains for modern SRE. Podcast Who is building clouds for the independent developer? Featured on Meta. Now live: A fully responsive profile. Reducing the weight of our footer.

Based on that explanation, I understood that I had to create a new session and client in each thread. Doing that ends up being quite slow, since creating a session has some overhead.

Actually you can create only one session and one client and pass that client to each thread, see the note. Bonus: Have a progress bar using tqdm that updated on each download. I hope this is useful for somebody out there having this also pretty standard problem. I hope you now understood which features of Boto3 are threadsafe and which are not, and most importantly, that you learned how to download multiple files from S3 in parallel using Python.

If you have any doubts or comments, reach out on Twitter or by email. Solution with tqdm working with multiple threads Boto3 in a nutshell: clients, sessions, and resources Boto3 is the official Python SDK for accessing and managing all AWS resources. Its 3 most used features are: sessions, clients, and resources.

TL;DR clients are threadsafe but sessions and resources are not. TL;DR instantiate one client and pass it to each thread.