Compare 2 big source by Python 3
I have 11 big files (text file with raw data) on the FTP server.
2016-04-10 00:00:00| 000102840111|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
And also have another source on Google Big Query (11 tables).
And I need to build a process to compare 2 sources (the data in FTP and GBQ) to find out the new records (this process need to run as daily, the FTP server will update files every morning). And this is the path I used:
- I export all the data ( 11 tables) on GBQ and download it to my local, and create historical data by MySQL.
- After that, I read the data in the FTP server and compare with historical data to find out the new records (use a simple query).
- Send it to GBQ, also insert to historical data.
Is this way is the best solution? Could anyone please can recommend me another solution?
python-3.x
asked 7 mins ago
Han Van Pham
254
add a comment |
I have 11 big files (text file with raw data) on the FTP server.
2016-04-10 00:00:00| 000102840111|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
And also have another source on Google Big Query (11 tables).
And I need to build a process to compare 2 sources (the data in FTP and GBQ) to find out the new records (this process need to run as daily, the FTP server will update files every morning). And this is the path I used:
- I export all the data ( 11 tables) on GBQ and download it to my local, and create historical data by MySQL.
- After that, I read the data in the FTP server and compare with historical data to find out the new records (use a simple query).
- Send it to GBQ, also insert to historical data.
Is this way is the best solution? Could anyone please can recommend me another solution?
python-3.x
asked 7 mins ago
Han Van Pham
254
add a comment |
I have 11 big files (text file with raw data) on the FTP server.
2016-04-10 00:00:00| 000102840111|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
And also have another source on Google Big Query (11 tables).
And I need to build a process to compare 2 sources (the data in FTP and GBQ) to find out the new records (this process need to run as daily, the FTP server will update files every morning). And this is the path I used:
- I export all the data ( 11 tables) on GBQ and download it to my local, and create historical data by MySQL.
- After that, I read the data in the FTP server and compare with historical data to find out the new records (use a simple query).
- Send it to GBQ, also insert to historical data.
Is this way is the best solution? Could anyone please can recommend me another solution?
python-3.x
asked 7 mins ago
Han Van Pham
254
I have 11 big files (text file with raw data) on the FTP server.
2016-04-10 00:00:00| 000102840111|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
2018-04-10 00:00:00| 000102840687|4987043|4845485
And also have another source on Google Big Query (11 tables).
And I need to build a process to compare 2 sources (the data in FTP and GBQ) to find out the new records (this process need to run as daily, the FTP server will update files every morning). And this is the path I used:
- I export all the data ( 11 tables) on GBQ and download it to my local, and create historical data by MySQL.
- After that, I read the data in the FTP server and compare with historical data to find out the new records (use a simple query).
- Send it to GBQ, also insert to historical data.
Is this way is the best solution? Could anyone please can recommend me another solution?
python-3.x
python-3.x
asked 7 mins ago
Han Van Pham
254
asked 7 mins ago
Han Van Pham
254
asked 7 mins ago
Han Van Pham
254
asked 7 mins ago
Han Van Pham
254
asked 7 mins ago
Han Van Pham
254
254
add a comment |
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
});
});
}, "mathjax-editing");
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "196"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f210346%2fcompare-2-big-source-by-python-3%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Code Review Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f210346%2fcompare-2-big-source-by-python-3%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
