Apache Nifi manipulate nested json with Execute Script












1














I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.





"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",

"amp": "32",
"vol": "400",
"vpower": 22

}
]
}


As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code" has to be renamed to"adrID" and "sckt" should map to the values "bike".



I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example



def flowFile = session.get()
if (!flowFile) return

try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)

// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5

}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)

} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}









share|improve this question
























  • what have you tried so far? what's your question? please provide input and output json examples....
    – daggett
    Nov 21 '18 at 13:23










  • I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
    – Steve
    Nov 21 '18 at 15:37


















1














I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.





"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",

"amp": "32",
"vol": "400",
"vpower": 22

}
]
}


As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code" has to be renamed to"adrID" and "sckt" should map to the values "bike".



I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example



def flowFile = session.get()
if (!flowFile) return

try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)

// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5

}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)

} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}









share|improve this question
























  • what have you tried so far? what's your question? please provide input and output json examples....
    – daggett
    Nov 21 '18 at 13:23










  • I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
    – Steve
    Nov 21 '18 at 15:37
















1












1








1







I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.





"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",

"amp": "32",
"vol": "400",
"vpower": 22

}
]
}


As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code" has to be renamed to"adrID" and "sckt" should map to the values "bike".



I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example



def flowFile = session.get()
if (!flowFile) return

try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)

// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5

}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)

} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}









share|improve this question















I am looking for a way to update the given Json's values and keys in a dynamic way. The way the Json is delivered is Always the same(in Terms of structure). The only Thing that differs is the amount of Data that is provided. So for example there could sometimes be 30, sometimes only 10 nestings etc.





"ampdata": [
{
"nr": "303",
"code": "JGJGh4958GH",
"status": "AVAILABLE",
"ability": [ "" ],
"type": "wheeled",
"conns": [
{
"nr": "447",
"status": "",
"version": "3",
"format": "sckt",

"amp": "32",
"vol": "400",
"vpower": 22

}
]
}


As Json uses other keys/values than I in my DB, I Need to convert them. Additionally I Need to Change some values if they match explicit strings.
So for example: "Code" has to be renamed to"adrID" and "sckt" should map to the values "bike".



I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example



def flowFile = session.get()
if (!flowFile) return

try {
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = new JsonSlurper().parseText(text)
def objBuilder = new JsonBuilder(obj)

// Update ingestionDate field with today's date
for(i in 0..obj.data.size()-1){
obj.data[0].remove("postal_code")
objBuilder.data[0].postal_code=5

}
// Output updated JSON
def json = JsonOutput.toJson(obj)
outputStream.write(JsonOutput.prettyPrint(json).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').tokenize('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)

} catch(Exception e) {
log.error('Error during JSON operations', e)
session.transfer(flowFile, REL_FAILURE)
}






json apache nested apache-nifi






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 21 '18 at 15:39

























asked Nov 21 '18 at 11:11









Steve

83




83












  • what have you tried so far? what's your question? please provide input and output json examples....
    – daggett
    Nov 21 '18 at 13:23










  • I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
    – Steve
    Nov 21 '18 at 15:37




















  • what have you tried so far? what's your question? please provide input and output json examples....
    – daggett
    Nov 21 '18 at 13:23










  • I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
    – Steve
    Nov 21 '18 at 15:37


















what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23




what have you tried so far? what's your question? please provide input and output json examples....
– daggett
Nov 21 '18 at 13:23












I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37






I tried a simple Groovy-Script to remove the key and or Change the value. There is no Problem in changing values, but in changing the key itself. So I tried removing the key and adding a new key. Unfortunately I could not figure out how to add a new key:value to the given json. So how can I add a new pair of key:value or rename the key, if that´s possible. Have a look at my code-example
– Steve
Nov 21 '18 at 15:37














1 Answer
1






active

oldest

votes


















0














...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }





share|improve this answer





















  • I tried using your Reply, but I faced a few new Problems. My Code is as follows : flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value } Problem: postalCode is not generated and "value" Shows "null"
    – Steve
    Nov 22 '18 at 8:56












  • ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
    – Steve
    Nov 22 '18 at 8:58










  • I don't see the write to output in your code
    – daggett
    Nov 22 '18 at 10:38











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53410877%2fapache-nifi-manipulate-nested-json-with-execute-script%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }





share|improve this answer





















  • I tried using your Reply, but I faced a few new Problems. My Code is as follows : flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value } Problem: postalCode is not generated and "value" Shows "null"
    – Steve
    Nov 22 '18 at 8:56












  • ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
    – Steve
    Nov 22 '18 at 8:58










  • I don't see the write to output in your code
    – daggett
    Nov 22 '18 at 10:38
















0














...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }





share|improve this answer





















  • I tried using your Reply, but I faced a few new Problems. My Code is as follows : flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value } Problem: postalCode is not generated and "value" Shows "null"
    – Steve
    Nov 22 '18 at 8:56












  • ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
    – Steve
    Nov 22 '18 at 8:58










  • I don't see the write to output in your code
    – daggett
    Nov 22 '18 at 10:38














0












0








0






...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }





share|improve this answer












...
def obj = new JsonSlurper().parse(inputStream, "UTF-8")
obj.data.each{e->
def value = e.remove("postal_code")
//set old value with a new key into object
e["postalCode"] = value
}
//write to output
def builder = new JsonBuilder(obj)
outputStream.withWriter("UTF-8"){ it << builder.toPrettyString() }






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 21 '18 at 16:26









daggett

7,72721529




7,72721529












  • I tried using your Reply, but I faced a few new Problems. My Code is as follows : flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value } Problem: postalCode is not generated and "value" Shows "null"
    – Steve
    Nov 22 '18 at 8:56












  • ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
    – Steve
    Nov 22 '18 at 8:58










  • I don't see the write to output in your code
    – daggett
    Nov 22 '18 at 10:38


















  • I tried using your Reply, but I faced a few new Problems. My Code is as follows : flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value } Problem: postalCode is not generated and "value" Shows "null"
    – Steve
    Nov 22 '18 at 8:56












  • ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
    – Steve
    Nov 22 '18 at 8:58










  • I don't see the write to output in your code
    – daggett
    Nov 22 '18 at 10:38
















I tried using your Reply, but I faced a few new Problems. My Code is as follows : flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value } Problem: postalCode is not generated and "value" Shows "null"
– Steve
Nov 22 '18 at 8:56






I tried using your Reply, but I faced a few new Problems. My Code is as follows : flowFile = session.write(flowFile, { inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def obj = new JsonSlurper().parseText(text) // Update ingestionDate field with today's date obj.data.each{e-> def value = obj.data[0].remove("postal_code") obj.data[0].name = value //set old value with a new key into object e["postalCode"] = value } Problem: postalCode is not generated and "value" Shows "null"
– Steve
Nov 22 '18 at 8:56














ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58




ps: the removal was successfull, i just could not figure out how to remove and update key/value. my Output is still the same as shown above, as I could not get it to work with your Version.
– Steve
Nov 22 '18 at 8:58












I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38




I don't see the write to output in your code
– daggett
Nov 22 '18 at 10:38


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53410877%2fapache-nifi-manipulate-nested-json-with-execute-script%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

404 Error Contact Form 7 ajax form submitting

How to know if a Active Directory user can login interactively

TypeError: fit_transform() missing 1 required positional argument: 'X'