Deserialize hits memory limit












6














We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:



{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}


Taking the above example we could assume that we could deserialize the above JSON string into the following classes:



public class MainModel {

String x;
String y;
String z;
LinesModel lines;
}


public class LineModel {

String a;
String b;
String c;
Account record;
}


As we can see the LineModel states that the record attribute has an Account sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.



Therefore we could deserialize the JSON the following way:



MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);


Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts children tree for each Account.



In order to remove it, we are doing the following:



Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');

Map<String, Object> itemObj;
Map<String, Object> record;

for (Object item : lines) {

itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');

if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);




The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.



Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.



We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:



String textPattern = '\"Contacts\":.*records.*\[.*]},';

String resultText = jsonString.replaceAll(textPattern, '');

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);


But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.



Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?



Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.










share|improve this question




















  • 5




    This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
    – Derek F
    4 hours ago






  • 1




    I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
    – Keith C
    1 hour ago
















6














We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:



{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}


Taking the above example we could assume that we could deserialize the above JSON string into the following classes:



public class MainModel {

String x;
String y;
String z;
LinesModel lines;
}


public class LineModel {

String a;
String b;
String c;
Account record;
}


As we can see the LineModel states that the record attribute has an Account sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.



Therefore we could deserialize the JSON the following way:



MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);


Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts children tree for each Account.



In order to remove it, we are doing the following:



Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');

Map<String, Object> itemObj;
Map<String, Object> record;

for (Object item : lines) {

itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');

if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);




The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.



Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.



We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:



String textPattern = '\"Contacts\":.*records.*\[.*]},';

String resultText = jsonString.replaceAll(textPattern, '');

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);


But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.



Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?



Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.










share|improve this question




















  • 5




    This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
    – Derek F
    4 hours ago






  • 1




    I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
    – Keith C
    1 hour ago














6












6








6







We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:



{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}


Taking the above example we could assume that we could deserialize the above JSON string into the following classes:



public class MainModel {

String x;
String y;
String z;
LinesModel lines;
}


public class LineModel {

String a;
String b;
String c;
Account record;
}


As we can see the LineModel states that the record attribute has an Account sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.



Therefore we could deserialize the JSON the following way:



MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);


Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts children tree for each Account.



In order to remove it, we are doing the following:



Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');

Map<String, Object> itemObj;
Map<String, Object> record;

for (Object item : lines) {

itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');

if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);




The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.



Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.



We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:



String textPattern = '\"Contacts\":.*records.*\[.*]},';

String resultText = jsonString.replaceAll(textPattern, '');

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);


But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.



Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?



Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.










share|improve this question















We have a custom apex webservice in which we do certain operations and later call a managed package apex method that returns a JSON string which we deserialize and send back to the client. For example purposes let's assume the following JSON structure is returned from the managed package method:



{
"x": "xxx",
"y": "yyy",
"z": "zzz",
"lines": [{
"a": "aaa",
"b": "bbb",
"c": "ccc",
"record": {
"attributes": {
"type": "Account",
"url": "/services/data/v44.0/sobjects/Account/001000000000000000"
},
"Id": "001000000000000000",
"AccountNumber": "12345",
"Contacts": {
"totalSize": 1,
"done": true,
"records": [{
"attributes": {
"type": "Contact",
"url": "/services/data/v44.0/sobjects/Contact/003S00000000000000"
},
"Id": "003S00000000000000",
"Phone": "22323423423",
"MobilePhone": "345345345345"
}]
}
}
}]
}


Taking the above example we could assume that we could deserialize the above JSON string into the following classes:



public class MainModel {

String x;
String y;
String z;
LinesModel lines;
}


public class LineModel {

String a;
String b;
String c;
Account record;
}


As we can see the LineModel states that the record attribute has an Account sobject type, therefore it can span children relationships of Contacts, Opportunities and many other children relationships.



Therefore we could deserialize the JSON the following way:



MainModel model = (MainModel)JSON.deserialize(jsonString, MainModel.class);


Before returning the model to our webservice caller we want to get rid of some information that the manage package includes but we don't want, for example the Contacts children tree for each Account.



In order to remove it, we are doing the following:



Map<String, Object> genericModel = (Map<String, Object>)JSON.deserializeUntyped(jsonString);
Object lines = (Object)genericModel.get('lines');

Map<String, Object> itemObj;
Map<String, Object> record;

for (Object item : lines) {

itemObj = (Map<String, Object>)item;
record = (Map<String, Object>)itemObj.get('record');

if (record.containsKey('Contacts')) {
record.remove('Contacts')
}
}

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(genericModel), MainModel.class);




The issue we're facing is that when the response from the manage package has many nested records the desarialization process (the deserializeUntyped call) fails with an Apex heap memory limit. When I say nested records, in the context of the above example there could maybe be only 20 Accounts but each account maybe having 60 contacts.



Since we don't have control on what the manage package returns in the JSON we cannot avoid having those unnecessary children.



We've tried removing the children using regex directly on the JSON string the following way to avoid unnecessary deserialization for cleaning:



String textPattern = '\"Contacts\":.*records.*\[.*]},';

String resultText = jsonString.replaceAll(textPattern, '');

MainModel model = (MainModel)JSON.deserialize(JSON.serialize(resultText), MainModel.class);


But this results on a System.LimitException: Regex too complicated error because the JSON string is too large.



Any ideas on how we can clean the unwanted JSON data before returning it to the caller that could be viable?



Please take into account that the above is an example very simplified for the sake of understanding the real issue, the real JSON has many other relationships and objects that need cleaning and is more complex but the scenario is pretty much the same.







json deserialize heap






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 1 hour ago









Keith C

94.6k1089204




94.6k1089204










asked 4 hours ago









jonathanwieseljonathanwiesel

252114




252114








  • 5




    This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
    – Derek F
    4 hours ago






  • 1




    I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
    – Keith C
    1 hour ago














  • 5




    This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
    – Derek F
    4 hours ago






  • 1




    I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
    – Keith C
    1 hour ago








5




5




This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
4 hours ago




This doesn't sound like a memory leak to me, just a very large json string. The synchronous limit on heap size is ~6MB, and the async limit is ~12MB. If you're trying to manipulate that JSON, I'd expect that you'd only be able to handle a JSON string less than half that size (the string remains in memory, and you need at least as much memory to deserialize). Do you happen to have any idea of the size of your JSON string(s)?
– Derek F
4 hours ago




1




1




I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
1 hour ago




I've edited the question title to reflect Derek's point that this isn't a leak but rather hitting the limit.
– Keith C
1 hour ago










1 Answer
1






active

oldest

votes


















6














In your wrapper class LineModel, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.



public class LineModel {

String a;
String b;
String c;
MyAccount record;
}

public Class MyAccount{
String Id;
String AccountNumber;
}

public class MainModel {

String x;
String y;
String z;
List<LineModel> lines;
}





share|improve this answer



















  • 1




    +1. This is indeed a very neat approach to deal with any such limitations.
    – Jayant Das
    31 mins ago











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "459"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f246040%2fdeserialize-hits-memory-limit%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









6














In your wrapper class LineModel, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.



public class LineModel {

String a;
String b;
String c;
MyAccount record;
}

public Class MyAccount{
String Id;
String AccountNumber;
}

public class MainModel {

String x;
String y;
String z;
List<LineModel> lines;
}





share|improve this answer



















  • 1




    +1. This is indeed a very neat approach to deal with any such limitations.
    – Jayant Das
    31 mins ago
















6














In your wrapper class LineModel, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.



public class LineModel {

String a;
String b;
String c;
MyAccount record;
}

public Class MyAccount{
String Id;
String AccountNumber;
}

public class MainModel {

String x;
String y;
String z;
List<LineModel> lines;
}





share|improve this answer



















  • 1




    +1. This is indeed a very neat approach to deal with any such limitations.
    – Jayant Das
    31 mins ago














6












6








6






In your wrapper class LineModel, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.



public class LineModel {

String a;
String b;
String c;
MyAccount record;
}

public Class MyAccount{
String Id;
String AccountNumber;
}

public class MainModel {

String x;
String y;
String z;
List<LineModel> lines;
}





share|improve this answer














In your wrapper class LineModel, don't use Standard Account. If you do that you will also deserialize children. Instead, create Apex Wrapper for Account with fields you need. This will be more memory efficient.



public class LineModel {

String a;
String b;
String c;
MyAccount record;
}

public Class MyAccount{
String Id;
String AccountNumber;
}

public class MainModel {

String x;
String y;
String z;
List<LineModel> lines;
}






share|improve this answer














share|improve this answer



share|improve this answer








edited 29 mins ago









Jayant Das

12.5k2723




12.5k2723










answered 3 hours ago









Pranay JaiswalPranay Jaiswal

14k32351




14k32351








  • 1




    +1. This is indeed a very neat approach to deal with any such limitations.
    – Jayant Das
    31 mins ago














  • 1




    +1. This is indeed a very neat approach to deal with any such limitations.
    – Jayant Das
    31 mins ago








1




1




+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
31 mins ago




+1. This is indeed a very neat approach to deal with any such limitations.
– Jayant Das
31 mins ago


















draft saved

draft discarded




















































Thanks for contributing an answer to Salesforce Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f246040%2fdeserialize-hits-memory-limit%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

404 Error Contact Form 7 ajax form submitting

How to know if a Active Directory user can login interactively

TypeError: fit_transform() missing 1 required positional argument: 'X'