Apache Nifi manipulate nested json with Execute Script
I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.
…
"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",
"amp": "32",
"vol": "400",
"vpower": 22
}
]
}
As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code"
has to be renamed to"adrID"
and "sckt"
should map to the values "bike"
.
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
def flowFile = session.get()
if (!flowFile) return
try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)
// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5
}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}
json apache nested apache-nifi
add a comment |
I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.
…
"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",
"amp": "32",
"vol": "400",
"vpower": 22
}
]
}
As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code"
has to be renamed to"adrID"
and "sckt"
should map to the values "bike"
.
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
def flowFile = session.get()
if (!flowFile) return
try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)
// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5
}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}
json apache nested apache-nifi
what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37
add a comment |
I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.
…
"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",
"amp": "32",
"vol": "400",
"vpower": 22
}
]
}
As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code"
has to be renamed to"adrID"
and "sckt"
should map to the values "bike"
.
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
def flowFile = session.get()
if (!flowFile) return
try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)
// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5
}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}
json apache nested apache-nifi
I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.
…
"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",
"amp": "32",
"vol": "400",
"vpower": 22
}
]
}
As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code"
has to be renamed to"adrID"
and "sckt"
should map to the values "bike"
.
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
def flowFile = session.get()
if (!flowFile) return
try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)
// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5
}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}
json apache nested apache-nifi
json apache nested apache-nifi
edited Nov 21 '18 at 15:39
asked Nov 21 '18 at 11:11
Steve
83
83
what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37
add a comment |
what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37
what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23
what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37
add a comment |
1 Answer
1
active
oldest
votes
...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }
I tried using your Reply, but I faced a few new Problems. My Code is as follows :flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value }
Problem: postalCode is not generated and "value" Shows "null"
– Steve
Nov 22 '18 at 8:56
ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58
I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53410877%2fapache-nifi-manipulate-nested-json-with-execute-script%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }
I tried using your Reply, but I faced a few new Problems. My Code is as follows :flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value }
Problem: postalCode is not generated and "value" Shows "null"
– Steve
Nov 22 '18 at 8:56
ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58
I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38
add a comment |
...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }
I tried using your Reply, but I faced a few new Problems. My Code is as follows :flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value }
Problem: postalCode is not generated and "value" Shows "null"
– Steve
Nov 22 '18 at 8:56
ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58
I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38
add a comment |
...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }
...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }
answered Nov 21 '18 at 16:26
daggett
7,72721529
7,72721529
I tried using your Reply, but I faced a few new Problems. My Code is as follows :flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value }
Problem: postalCode is not generated and "value" Shows "null"
– Steve
Nov 22 '18 at 8:56
ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58
I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38
add a comment |
I tried using your Reply, but I faced a few new Problems. My Code is as follows :flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value }
Problem: postalCode is not generated and "value" Shows "null"
– Steve
Nov 22 '18 at 8:56
ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58
I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38
I tried using your Reply, but I faced a few new Problems. My Code is as follows :
flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value }
Problem: postalCode is not generated and "value" Shows "null"– Steve
Nov 22 '18 at 8:56
I tried using your Reply, but I faced a few new Problems. My Code is as follows :
flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value }
Problem: postalCode is not generated and "value" Shows "null"– Steve
Nov 22 '18 at 8:56
ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58
ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58
I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38
I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53410877%2fapache-nifi-manipulate-nested-json-with-execute-script%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23
I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37