pytorch CUDA out of memory when training on big image
When I train a model with 512*512 images with batch_size=4, it works,
but when I train it with 1024*1024 images with batch_size=1, the cuda out of memory error happens.
I am confused, the total number of pixels are equal. Why one 1024*1024 image uses more memory than four 512*512 images?
pytorch
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
add a comment |
When I train a model with 512*512 images with batch_size=4, it works,
but when I train it with 1024*1024 images with batch_size=1, the cuda out of memory error happens.
I am confused, the total number of pixels are equal. Why one 1024*1024 image uses more memory than four 512*512 images?
pytorch
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
I had the same problem I couldn't find any solution. Actually, I am trying to implement a paper, and even in that paper, author mentioned we cannot train using 1024*1024 or above. They tested their implementation on 2 Nvidia 1080 Ti with 2*12GB GPU memory. If you find any solution, please tell me. Thank you
– M. Doosti Lakhani
Nov 25 '18 at 9:13
add a comment |
When I train a model with 512*512 images with batch_size=4, it works,
but when I train it with 1024*1024 images with batch_size=1, the cuda out of memory error happens.
I am confused, the total number of pixels are equal. Why one 1024*1024 image uses more memory than four 512*512 images?
pytorch
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
When I train a model with 512*512 images with batch_size=4, it works,
but when I train it with 1024*1024 images with batch_size=1, the cuda out of memory error happens.
I am confused, the total number of pixels are equal. Why one 1024*1024 image uses more memory than four 512*512 images?
pytorch
pytorch
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
edited Nov 26 '18 at 4:30
dwjohnston
2,829114890
2,829114890
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
asked Nov 25 '18 at 7:20
ILINLINILINLIN
61
61
I had the same problem I couldn't find any solution. Actually, I am trying to implement a paper, and even in that paper, author mentioned we cannot train using 1024*1024 or above. They tested their implementation on 2 Nvidia 1080 Ti with 2*12GB GPU memory. If you find any solution, please tell me. Thank you
– M. Doosti Lakhani
Nov 25 '18 at 9:13
add a comment |
I had the same problem I couldn't find any solution. Actually, I am trying to implement a paper, and even in that paper, author mentioned we cannot train using 1024*1024 or above. They tested their implementation on 2 Nvidia 1080 Ti with 2*12GB GPU memory. If you find any solution, please tell me. Thank you
– M. Doosti Lakhani
Nov 25 '18 at 9:13
I had the same problem I couldn't find any solution. Actually, I am trying to implement a paper, and even in that paper, author mentioned we cannot train using 1024*1024 or above. They tested their implementation on 2 Nvidia 1080 Ti with 2*12GB GPU memory. If you find any solution, please tell me. Thank you
– M. Doosti Lakhani
Nov 25 '18 at 9:13
I had the same problem I couldn't find any solution. Actually, I am trying to implement a paper, and even in that paper, author mentioned we cannot train using 1024*1024 or above. They tested their implementation on 2 Nvidia 1080 Ti with 2*12GB GPU memory. If you find any solution, please tell me. Thank you
– M. Doosti Lakhani
Nov 25 '18 at 9:13
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53465463%2fpytorch-cuda-out-of-memory-when-training-on-big-image%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53465463%2fpytorch-cuda-out-of-memory-when-training-on-big-image%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I had the same problem I couldn't find any solution. Actually, I am trying to implement a paper, and even in that paper, author mentioned we cannot train using 1024*1024 or above. They tested their implementation on 2 Nvidia 1080 Ti with 2*12GB GPU memory. If you find any solution, please tell me. Thank you
– M. Doosti Lakhani
Nov 25 '18 at 9:13