How can I combine a sequence of JSON with jq without using the slurp flag?
I have a ton of records (~4,500) that I've processed (using jq) down to a sequence of JSON grouped by hourly UTC time (~680 groups, all unique).
{
"2018-10-09T19:00:00.000Z":
}
{
"2018-10-09T20:00:00.000Z":
}
{
"2018-10-09T21:00:00.000Z":
}
I'm pretty sure you can see where this is going, but I want to combine all these into a single JSON object to hand over to another system for more fun.
{
"2018-10-09T19:00:00.000Z": ,
"2018-10-09T20:00:00.000Z": ,
"2018-10-09T21:00:00.000Z":
}
The last two things I'm doing before I get to the sequence of objects is:
group_by(.day) | { (.[0].day): . }
Where .day is the ISO Date you see referenced above.
I've tried a few things around map and reduce functions, but can't seem to massage the data the way I want. I've spent a few hours on this and need to take a break, so any help or direction you can point me would be great!
javascript node.js json jq
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
add a comment |
I have a ton of records (~4,500) that I've processed (using jq) down to a sequence of JSON grouped by hourly UTC time (~680 groups, all unique).
{
"2018-10-09T19:00:00.000Z":
}
{
"2018-10-09T20:00:00.000Z":
}
{
"2018-10-09T21:00:00.000Z":
}
I'm pretty sure you can see where this is going, but I want to combine all these into a single JSON object to hand over to another system for more fun.
{
"2018-10-09T19:00:00.000Z": ,
"2018-10-09T20:00:00.000Z": ,
"2018-10-09T21:00:00.000Z":
}
The last two things I'm doing before I get to the sequence of objects is:
group_by(.day) | { (.[0].day): . }
Where .day is the ISO Date you see referenced above.
I've tried a few things around map and reduce functions, but can't seem to massage the data the way I want. I've spent a few hours on this and need to take a break, so any help or direction you can point me would be great!
javascript node.js json jq
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
add a comment |
I have a ton of records (~4,500) that I've processed (using jq) down to a sequence of JSON grouped by hourly UTC time (~680 groups, all unique).
{
"2018-10-09T19:00:00.000Z":
}
{
"2018-10-09T20:00:00.000Z":
}
{
"2018-10-09T21:00:00.000Z":
}
I'm pretty sure you can see where this is going, but I want to combine all these into a single JSON object to hand over to another system for more fun.
{
"2018-10-09T19:00:00.000Z": ,
"2018-10-09T20:00:00.000Z": ,
"2018-10-09T21:00:00.000Z":
}
The last two things I'm doing before I get to the sequence of objects is:
group_by(.day) | { (.[0].day): . }
Where .day is the ISO Date you see referenced above.
I've tried a few things around map and reduce functions, but can't seem to massage the data the way I want. I've spent a few hours on this and need to take a break, so any help or direction you can point me would be great!
javascript node.js json jq
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
I have a ton of records (~4,500) that I've processed (using jq) down to a sequence of JSON grouped by hourly UTC time (~680 groups, all unique).
{
"2018-10-09T19:00:00.000Z":
}
{
"2018-10-09T20:00:00.000Z":
}
{
"2018-10-09T21:00:00.000Z":
}
I'm pretty sure you can see where this is going, but I want to combine all these into a single JSON object to hand over to another system for more fun.
{
"2018-10-09T19:00:00.000Z": ,
"2018-10-09T20:00:00.000Z": ,
"2018-10-09T21:00:00.000Z":
}
The last two things I'm doing before I get to the sequence of objects is:
group_by(.day) | { (.[0].day): . }
Where .day is the ISO Date you see referenced above.
I've tried a few things around map and reduce functions, but can't seem to massage the data the way I want. I've spent a few hours on this and need to take a break, so any help or direction you can point me would be great!
javascript node.js json jq
javascript node.js json jq
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
asked Nov 21 '18 at 21:33
Sam BantnerSam Bantner
9018
9018
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
If everything is already in memory, you could modify the group_by line as follows:
reduce group_by(.day) as $in ({}; . + { ($in[0].day): $in }
Alternatives to group_by
Since group_by entails a sort, it may be unnecessarily inefficient. You might like to consider using a variant such as the following:
# sort-free variant of group_by/1
# f must always evaluate to an integer or always to a string.
# Output: an array in the former case, or an object in the latter case
def GROUP_BY(f): reduce . as $x ({}; .[$x|f] += [$x] );
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
Everything is in memory and this is a very self-explanatory answer. Thanks for the help! I didn't think to bounce reducing up one level, which completely makes sense as my brain is clear :)
– Sam Bantner
Nov 23 '18 at 13:11
add a comment |
If the stream of objects is already in a file, use inputs with the -n command-line option.
This will avoid the overhead of "slurping" but will still require enough RAM for the entire result to fit into memory. If that doesn't work for you, then you will have to resort to desperate measures :-)
This might be a useful starting point:
jq -n 'reduce inputs as $in ({}; . + $in)'
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
I think you could also pipe the output as an array intoadd.
– mustachioed
Nov 21 '18 at 22:00
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53420766%2fhow-can-i-combine-a-sequence-of-json-with-jq-without-using-the-slurp-flag%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
If everything is already in memory, you could modify the group_by line as follows:
reduce group_by(.day) as $in ({}; . + { ($in[0].day): $in }
Alternatives to group_by
Since group_by entails a sort, it may be unnecessarily inefficient. You might like to consider using a variant such as the following:
# sort-free variant of group_by/1
# f must always evaluate to an integer or always to a string.
# Output: an array in the former case, or an object in the latter case
def GROUP_BY(f): reduce . as $x ({}; .[$x|f] += [$x] );
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
Everything is in memory and this is a very self-explanatory answer. Thanks for the help! I didn't think to bounce reducing up one level, which completely makes sense as my brain is clear :)
– Sam Bantner
Nov 23 '18 at 13:11
add a comment |
If everything is already in memory, you could modify the group_by line as follows:
reduce group_by(.day) as $in ({}; . + { ($in[0].day): $in }
Alternatives to group_by
Since group_by entails a sort, it may be unnecessarily inefficient. You might like to consider using a variant such as the following:
# sort-free variant of group_by/1
# f must always evaluate to an integer or always to a string.
# Output: an array in the former case, or an object in the latter case
def GROUP_BY(f): reduce . as $x ({}; .[$x|f] += [$x] );
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
Everything is in memory and this is a very self-explanatory answer. Thanks for the help! I didn't think to bounce reducing up one level, which completely makes sense as my brain is clear :)
– Sam Bantner
Nov 23 '18 at 13:11
add a comment |
If everything is already in memory, you could modify the group_by line as follows:
reduce group_by(.day) as $in ({}; . + { ($in[0].day): $in }
Alternatives to group_by
Since group_by entails a sort, it may be unnecessarily inefficient. You might like to consider using a variant such as the following:
# sort-free variant of group_by/1
# f must always evaluate to an integer or always to a string.
# Output: an array in the former case, or an object in the latter case
def GROUP_BY(f): reduce . as $x ({}; .[$x|f] += [$x] );
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
If everything is already in memory, you could modify the group_by line as follows:
reduce group_by(.day) as $in ({}; . + { ($in[0].day): $in }
Alternatives to group_by
Since group_by entails a sort, it may be unnecessarily inefficient. You might like to consider using a variant such as the following:
# sort-free variant of group_by/1
# f must always evaluate to an integer or always to a string.
# Output: an array in the former case, or an object in the latter case
def GROUP_BY(f): reduce . as $x ({}; .[$x|f] += [$x] );
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
edited Nov 21 '18 at 23:00
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
answered Nov 21 '18 at 22:51
peakpeak
30.7k83957
30.7k83957
Everything is in memory and this is a very self-explanatory answer. Thanks for the help! I didn't think to bounce reducing up one level, which completely makes sense as my brain is clear :)
– Sam Bantner
Nov 23 '18 at 13:11
add a comment |
Everything is in memory and this is a very self-explanatory answer. Thanks for the help! I didn't think to bounce reducing up one level, which completely makes sense as my brain is clear :)
– Sam Bantner
Nov 23 '18 at 13:11
Everything is in memory and this is a very self-explanatory answer. Thanks for the help! I didn't think to bounce reducing up one level, which completely makes sense as my brain is clear :)
– Sam Bantner
Nov 23 '18 at 13:11
Everything is in memory and this is a very self-explanatory answer. Thanks for the help! I didn't think to bounce reducing up one level, which completely makes sense as my brain is clear :)
– Sam Bantner
Nov 23 '18 at 13:11
add a comment |
If the stream of objects is already in a file, use inputs with the -n command-line option.
This will avoid the overhead of "slurping" but will still require enough RAM for the entire result to fit into memory. If that doesn't work for you, then you will have to resort to desperate measures :-)
This might be a useful starting point:
jq -n 'reduce inputs as $in ({}; . + $in)'
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
I think you could also pipe the output as an array intoadd.
– mustachioed
Nov 21 '18 at 22:00
add a comment |
If the stream of objects is already in a file, use inputs with the -n command-line option.
This will avoid the overhead of "slurping" but will still require enough RAM for the entire result to fit into memory. If that doesn't work for you, then you will have to resort to desperate measures :-)
This might be a useful starting point:
jq -n 'reduce inputs as $in ({}; . + $in)'
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
I think you could also pipe the output as an array intoadd.
– mustachioed
Nov 21 '18 at 22:00
add a comment |
If the stream of objects is already in a file, use inputs with the -n command-line option.
This will avoid the overhead of "slurping" but will still require enough RAM for the entire result to fit into memory. If that doesn't work for you, then you will have to resort to desperate measures :-)
This might be a useful starting point:
jq -n 'reduce inputs as $in ({}; . + $in)'
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
If the stream of objects is already in a file, use inputs with the -n command-line option.
This will avoid the overhead of "slurping" but will still require enough RAM for the entire result to fit into memory. If that doesn't work for you, then you will have to resort to desperate measures :-)
This might be a useful starting point:
jq -n 'reduce inputs as $in ({}; . + $in)'
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
edited Nov 22 '18 at 6:38
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
answered Nov 21 '18 at 21:59
peakpeak
30.7k83957
30.7k83957
I think you could also pipe the output as an array intoadd.
– mustachioed
Nov 21 '18 at 22:00
add a comment |
I think you could also pipe the output as an array intoadd.
– mustachioed
Nov 21 '18 at 22:00
I think you could also pipe the output as an array into
add.– mustachioed
Nov 21 '18 at 22:00
I think you could also pipe the output as an array into
add.– mustachioed
Nov 21 '18 at 22:00
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53420766%2fhow-can-i-combine-a-sequence-of-json-with-jq-without-using-the-slurp-flag%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
