Getting specific data from a large table pandas
I'm trying to process a very large CSV file (4.2 GB) using Pandas, scanning through the file for instances of a specific value. Given the large size of this file, I have tried processing in chunks, but I'm having trouble coming up with an algorithm to find instances of the value.
Essentially, I have a set of 33 parameters, several of which can be found in the 4.2 GB table. I have a list of 2,000-odd patients with pneumonia, and I need to find the first instance of each parameter for each patient. The end goal is to create a new table, with one row per patient and each column representing a different parameter. To fill this table, I first have to process the very large table. The large table consists of "Chart Events" for every patient - a patient can have hundreds of events, and the total database includes about 40,000 patients.
I hope to use the new table to train a machine learning algorithm to predict the length of ICU stay for the patients.
My idea so far has been this:
for each parameter:
for every row in chartevents:
for every patient:
if the row contains the parameter for that patient:
update the new table value for that patient and parameter
Obviously this is super inefficient, so I was hoping someone might know better way. For more information about the data, check out this website.
python pandas
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
asked Nov 25 '18 at 4:42
mrajumraju
1
add a comment |
I'm trying to process a very large CSV file (4.2 GB) using Pandas, scanning through the file for instances of a specific value. Given the large size of this file, I have tried processing in chunks, but I'm having trouble coming up with an algorithm to find instances of the value.
Essentially, I have a set of 33 parameters, several of which can be found in the 4.2 GB table. I have a list of 2,000-odd patients with pneumonia, and I need to find the first instance of each parameter for each patient. The end goal is to create a new table, with one row per patient and each column representing a different parameter. To fill this table, I first have to process the very large table. The large table consists of "Chart Events" for every patient - a patient can have hundreds of events, and the total database includes about 40,000 patients.
I hope to use the new table to train a machine learning algorithm to predict the length of ICU stay for the patients.
My idea so far has been this:
for each parameter:
for every row in chartevents:
for every patient:
if the row contains the parameter for that patient:
update the new table value for that patient and parameter
Obviously this is super inefficient, so I was hoping someone might know better way. For more information about the data, check out this website.
python pandas
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
asked Nov 25 '18 at 4:42
mrajumraju
1
Give a try toDaskdask.org
– dataLeo
Nov 25 '18 at 5:03
Question has nothing to do withmachine-learning- kindly do not spam the tag (removed).
– desertnaut
Nov 25 '18 at 9:31
add a comment |
I'm trying to process a very large CSV file (4.2 GB) using Pandas, scanning through the file for instances of a specific value. Given the large size of this file, I have tried processing in chunks, but I'm having trouble coming up with an algorithm to find instances of the value.
Essentially, I have a set of 33 parameters, several of which can be found in the 4.2 GB table. I have a list of 2,000-odd patients with pneumonia, and I need to find the first instance of each parameter for each patient. The end goal is to create a new table, with one row per patient and each column representing a different parameter. To fill this table, I first have to process the very large table. The large table consists of "Chart Events" for every patient - a patient can have hundreds of events, and the total database includes about 40,000 patients.
I hope to use the new table to train a machine learning algorithm to predict the length of ICU stay for the patients.
My idea so far has been this:
for each parameter:
for every row in chartevents:
for every patient:
if the row contains the parameter for that patient:
update the new table value for that patient and parameter
Obviously this is super inefficient, so I was hoping someone might know better way. For more information about the data, check out this website.
python pandas
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
asked Nov 25 '18 at 4:42
mrajumraju
1
I'm trying to process a very large CSV file (4.2 GB) using Pandas, scanning through the file for instances of a specific value. Given the large size of this file, I have tried processing in chunks, but I'm having trouble coming up with an algorithm to find instances of the value.
Essentially, I have a set of 33 parameters, several of which can be found in the 4.2 GB table. I have a list of 2,000-odd patients with pneumonia, and I need to find the first instance of each parameter for each patient. The end goal is to create a new table, with one row per patient and each column representing a different parameter. To fill this table, I first have to process the very large table. The large table consists of "Chart Events" for every patient - a patient can have hundreds of events, and the total database includes about 40,000 patients.
I hope to use the new table to train a machine learning algorithm to predict the length of ICU stay for the patients.
My idea so far has been this:
for each parameter:
for every row in chartevents:
for every patient:
if the row contains the parameter for that patient:
update the new table value for that patient and parameter
Obviously this is super inefficient, so I was hoping someone might know better way. For more information about the data, check out this website.
python pandas
python pandas
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
asked Nov 25 '18 at 4:42
mrajumraju
1
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
asked Nov 25 '18 at 4:42
mrajumraju
1
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
edited Nov 25 '18 at 9:31
desertnaut
18.2k73872
18.2k73872
asked Nov 25 '18 at 4:42
mrajumraju
1
asked Nov 25 '18 at 4:42
mrajumraju
1
asked Nov 25 '18 at 4:42
mrajumraju
1
1
Give a try toDaskdask.org
– dataLeo
Nov 25 '18 at 5:03
Question has nothing to do withmachine-learning- kindly do not spam the tag (removed).
– desertnaut
Nov 25 '18 at 9:31
add a comment |
Give a try toDaskdask.org
– dataLeo
Nov 25 '18 at 5:03
Question has nothing to do withmachine-learning- kindly do not spam the tag (removed).
– desertnaut
Nov 25 '18 at 9:31
Give a try to
Dask dask.org– dataLeo
Nov 25 '18 at 5:03
Give a try to
Dask dask.org– dataLeo
Nov 25 '18 at 5:03
Question has nothing to do with
machine-learning - kindly do not spam the tag (removed).– desertnaut
Nov 25 '18 at 9:31
Question has nothing to do with
machine-learning - kindly do not spam the tag (removed).– desertnaut
Nov 25 '18 at 9:31
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53464709%2fgetting-specific-data-from-a-large-table-pandas%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53464709%2fgetting-specific-data-from-a-large-table-pandas%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Give a try to
Daskdask.org– dataLeo
Nov 25 '18 at 5:03
Question has nothing to do with
machine-learning- kindly do not spam the tag (removed).– desertnaut
Nov 25 '18 at 9:31