Python Boto3 Lambda Upload Temp File
I'm working on a Lambda that compresses image files in an S3 bucket. I'm able to download the image in the Lambda, compress it as a new file. I'm trying to upload the new file to the same S3 bucket and I keep on getting hit with the following error:
module initialization error: expected string or bytes-like object
Here's the code to upload:
s3 = boto3.client('s3')
s3.upload_file(filename,my_bucket,basename)
Here are the logs from one of the test uploads:
Getting ready to download Giggidy.png
This is what we're calling our temp file: /tmp/tmp6i7fvb6z.png
Let's compress /tmp/tmp6i7fvb6z.png
Compressed /tmp/tmp6i7fvb6z.png to /tmp/tmpmq23jj5c.png
Getting ready to upload /tmp/tmpmq23jj5c.png
File to Upload, filename: /tmp/tmpmq23jj5c.png
Mime Type: image/png
Name in Bucket, basename: tmpmq23jj5c.png
START RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d Version: $LATEST
module initialization error: expected string or bytes-like object
END RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d
How can I upload a file within the context of a Lambda?
UPDATE: I've uploaded my code to a gist for review: https://gist.github.com/kjenney/068531ffe01e14bb7a2351dc55592551
I also moved the boto3 client connection up in my script thinking that might be hosing the upload but I still get the same error in the same order. 'process' is my handler function.
amazon-s3 aws-lambda boto3
asked Nov 21 at 1:42
Ken J
9919
add a comment |
I'm working on a Lambda that compresses image files in an S3 bucket. I'm able to download the image in the Lambda, compress it as a new file. I'm trying to upload the new file to the same S3 bucket and I keep on getting hit with the following error:
module initialization error: expected string or bytes-like object
Here's the code to upload:
s3 = boto3.client('s3')
s3.upload_file(filename,my_bucket,basename)
Here are the logs from one of the test uploads:
Getting ready to download Giggidy.png
This is what we're calling our temp file: /tmp/tmp6i7fvb6z.png
Let's compress /tmp/tmp6i7fvb6z.png
Compressed /tmp/tmp6i7fvb6z.png to /tmp/tmpmq23jj5c.png
Getting ready to upload /tmp/tmpmq23jj5c.png
File to Upload, filename: /tmp/tmpmq23jj5c.png
Mime Type: image/png
Name in Bucket, basename: tmpmq23jj5c.png
START RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d Version: $LATEST
module initialization error: expected string or bytes-like object
END RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d
How can I upload a file within the context of a Lambda?
UPDATE: I've uploaded my code to a gist for review: https://gist.github.com/kjenney/068531ffe01e14bb7a2351dc55592551
I also moved the boto3 client connection up in my script thinking that might be hosing the upload but I still get the same error in the same order. 'process' is my handler function.
amazon-s3 aws-lambda boto3
asked Nov 21 at 1:42
Ken J
9919
Could you please show more of your code, especially your Lambda handler? How much of those logs are coming from Lambda? Where does everything beforeSTARTcome from? I suspect that the function isn't getting started correctly, so it's not even getting near yourupload_file()command.
– John Rotenstein
Nov 21 at 2:03
I ended up with a working solution by using aws-lambda-image. It's written in Node which is not my preferred language but it does work. I'd still love to get something working in Python3. That being said it does seem to be a lot more involved.
– Ken J
Nov 21 at 12:38
It really sounds like the Lambda function has not been configured to match your Handler name. If you want to get the Python version going, please add more details.
– John Rotenstein
Nov 21 at 19:55
@JohnRotenstein If you look at the code you can see that the entrypoint is process. I've got print statements sprinkled thru the code so it's definitely configured correctly. What I'm missing is how/why the upload_file portion is somehow not getting processed correctly even though there's definitely an image there that's getting read as bytes.
– Ken J
Nov 22 at 5:15
add a comment |
I'm working on a Lambda that compresses image files in an S3 bucket. I'm able to download the image in the Lambda, compress it as a new file. I'm trying to upload the new file to the same S3 bucket and I keep on getting hit with the following error:
module initialization error: expected string or bytes-like object
Here's the code to upload:
s3 = boto3.client('s3')
s3.upload_file(filename,my_bucket,basename)
Here are the logs from one of the test uploads:
Getting ready to download Giggidy.png
This is what we're calling our temp file: /tmp/tmp6i7fvb6z.png
Let's compress /tmp/tmp6i7fvb6z.png
Compressed /tmp/tmp6i7fvb6z.png to /tmp/tmpmq23jj5c.png
Getting ready to upload /tmp/tmpmq23jj5c.png
File to Upload, filename: /tmp/tmpmq23jj5c.png
Mime Type: image/png
Name in Bucket, basename: tmpmq23jj5c.png
START RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d Version: $LATEST
module initialization error: expected string or bytes-like object
END RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d
How can I upload a file within the context of a Lambda?
UPDATE: I've uploaded my code to a gist for review: https://gist.github.com/kjenney/068531ffe01e14bb7a2351dc55592551
I also moved the boto3 client connection up in my script thinking that might be hosing the upload but I still get the same error in the same order. 'process' is my handler function.
amazon-s3 aws-lambda boto3
asked Nov 21 at 1:42
Ken J
9919
I'm working on a Lambda that compresses image files in an S3 bucket. I'm able to download the image in the Lambda, compress it as a new file. I'm trying to upload the new file to the same S3 bucket and I keep on getting hit with the following error:
module initialization error: expected string or bytes-like object
Here's the code to upload:
s3 = boto3.client('s3')
s3.upload_file(filename,my_bucket,basename)
Here are the logs from one of the test uploads:
Getting ready to download Giggidy.png
This is what we're calling our temp file: /tmp/tmp6i7fvb6z.png
Let's compress /tmp/tmp6i7fvb6z.png
Compressed /tmp/tmp6i7fvb6z.png to /tmp/tmpmq23jj5c.png
Getting ready to upload /tmp/tmpmq23jj5c.png
File to Upload, filename: /tmp/tmpmq23jj5c.png
Mime Type: image/png
Name in Bucket, basename: tmpmq23jj5c.png
START RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d Version: $LATEST
module initialization error: expected string or bytes-like object
END RequestId: e9062ca9-ed2c-11e9-99ee-e3a40680ga9d
How can I upload a file within the context of a Lambda?
UPDATE: I've uploaded my code to a gist for review: https://gist.github.com/kjenney/068531ffe01e14bb7a2351dc55592551
I also moved the boto3 client connection up in my script thinking that might be hosing the upload but I still get the same error in the same order. 'process' is my handler function.
amazon-s3 aws-lambda boto3
amazon-s3 aws-lambda boto3
asked Nov 21 at 1:42
Ken J
9919
asked Nov 21 at 1:42
Ken J
9919
edited Nov 21 at 2:27
asked Nov 21 at 1:42
Ken J
9919
asked Nov 21 at 1:42
Ken J
9919
asked Nov 21 at 1:42
Ken J
9919
9919
Could you please show more of your code, especially your Lambda handler? How much of those logs are coming from Lambda? Where does everything beforeSTARTcome from? I suspect that the function isn't getting started correctly, so it's not even getting near yourupload_file()command.
– John Rotenstein
Nov 21 at 2:03
I ended up with a working solution by using aws-lambda-image. It's written in Node which is not my preferred language but it does work. I'd still love to get something working in Python3. That being said it does seem to be a lot more involved.
– Ken J
Nov 21 at 12:38
It really sounds like the Lambda function has not been configured to match your Handler name. If you want to get the Python version going, please add more details.
– John Rotenstein
Nov 21 at 19:55
@JohnRotenstein If you look at the code you can see that the entrypoint is process. I've got print statements sprinkled thru the code so it's definitely configured correctly. What I'm missing is how/why the upload_file portion is somehow not getting processed correctly even though there's definitely an image there that's getting read as bytes.
– Ken J
Nov 22 at 5:15
add a comment |
Could you please show more of your code, especially your Lambda handler? How much of those logs are coming from Lambda? Where does everything beforeSTARTcome from? I suspect that the function isn't getting started correctly, so it's not even getting near yourupload_file()command.
– John Rotenstein
Nov 21 at 2:03
I ended up with a working solution by using aws-lambda-image. It's written in Node which is not my preferred language but it does work. I'd still love to get something working in Python3. That being said it does seem to be a lot more involved.
– Ken J
Nov 21 at 12:38
It really sounds like the Lambda function has not been configured to match your Handler name. If you want to get the Python version going, please add more details.
– John Rotenstein
Nov 21 at 19:55
@JohnRotenstein If you look at the code you can see that the entrypoint is process. I've got print statements sprinkled thru the code so it's definitely configured correctly. What I'm missing is how/why the upload_file portion is somehow not getting processed correctly even though there's definitely an image there that's getting read as bytes.
– Ken J
Nov 22 at 5:15
Could you please show more of your code, especially your Lambda handler? How much of those logs are coming from Lambda? Where does everything before
START come from? I suspect that the function isn't getting started correctly, so it's not even getting near your upload_file() command.– John Rotenstein
Nov 21 at 2:03
Could you please show more of your code, especially your Lambda handler? How much of those logs are coming from Lambda? Where does everything before
START come from? I suspect that the function isn't getting started correctly, so it's not even getting near your upload_file() command.– John Rotenstein
Nov 21 at 2:03
I ended up with a working solution by using aws-lambda-image. It's written in Node which is not my preferred language but it does work. I'd still love to get something working in Python3. That being said it does seem to be a lot more involved.
– Ken J
Nov 21 at 12:38
I ended up with a working solution by using aws-lambda-image. It's written in Node which is not my preferred language but it does work. I'd still love to get something working in Python3. That being said it does seem to be a lot more involved.
– Ken J
Nov 21 at 12:38
It really sounds like the Lambda function has not been configured to match your Handler name. If you want to get the Python version going, please add more details.
– John Rotenstein
Nov 21 at 19:55
It really sounds like the Lambda function has not been configured to match your Handler name. If you want to get the Python version going, please add more details.
– John Rotenstein
Nov 21 at 19:55
@JohnRotenstein If you look at the code you can see that the entrypoint is process. I've got print statements sprinkled thru the code so it's definitely configured correctly. What I'm missing is how/why the upload_file portion is somehow not getting processed correctly even though there's definitely an image there that's getting read as bytes.
– Ken J
Nov 22 at 5:15
@JohnRotenstein If you look at the code you can see that the entrypoint is process. I've got print statements sprinkled thru the code so it's definitely configured correctly. What I'm missing is how/why the upload_file portion is somehow not getting processed correctly even though there's definitely an image there that's getting read as bytes.
– Ken J
Nov 22 at 5:15
add a comment |
1 Answer
1
active
oldest
votes
Your problem is this line:
client.upload_file(filename,my_bucket,basename)
From the documentation, the format is:
client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
Note that the bucket name is a string. That's why the error says expected string.
However, your code sets my_bucket as:
my_bucket = s3.Bucket(bucket)
You should use the name of the bucket rather than the bucket object.
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404172%2fpython-boto3-lambda-upload-temp-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Your problem is this line:
client.upload_file(filename,my_bucket,basename)
From the documentation, the format is:
client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
Note that the bucket name is a string. That's why the error says expected string.
However, your code sets my_bucket as:
my_bucket = s3.Bucket(bucket)
You should use the name of the bucket rather than the bucket object.
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
add a comment |
Your problem is this line:
client.upload_file(filename,my_bucket,basename)
From the documentation, the format is:
client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
Note that the bucket name is a string. That's why the error says expected string.
However, your code sets my_bucket as:
my_bucket = s3.Bucket(bucket)
You should use the name of the bucket rather than the bucket object.
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
add a comment |
Your problem is this line:
client.upload_file(filename,my_bucket,basename)
From the documentation, the format is:
client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
Note that the bucket name is a string. That's why the error says expected string.
However, your code sets my_bucket as:
my_bucket = s3.Bucket(bucket)
You should use the name of the bucket rather than the bucket object.
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
Your problem is this line:
client.upload_file(filename,my_bucket,basename)
From the documentation, the format is:
client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
Note that the bucket name is a string. That's why the error says expected string.
However, your code sets my_bucket as:
my_bucket = s3.Bucket(bucket)
You should use the name of the bucket rather than the bucket object.
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
answered Nov 22 at 5:28
John Rotenstein
67.1k774118
67.1k774118
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404172%2fpython-boto3-lambda-upload-temp-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Could you please show more of your code, especially your Lambda handler? How much of those logs are coming from Lambda? Where does everything before
STARTcome from? I suspect that the function isn't getting started correctly, so it's not even getting near yourupload_file()command.– John Rotenstein
Nov 21 at 2:03
I ended up with a working solution by using aws-lambda-image. It's written in Node which is not my preferred language but it does work. I'd still love to get something working in Python3. That being said it does seem to be a lot more involved.
– Ken J
Nov 21 at 12:38
It really sounds like the Lambda function has not been configured to match your Handler name. If you want to get the Python version going, please add more details.
– John Rotenstein
Nov 21 at 19:55
@JohnRotenstein If you look at the code you can see that the entrypoint is process. I've got print statements sprinkled thru the code so it's definitely configured correctly. What I'm missing is how/why the upload_file portion is somehow not getting processed correctly even though there's definitely an image there that's getting read as bytes.
– Ken J
Nov 22 at 5:15