why do SortedMap and TreeMap have different impacts on performance in spark cluster?
up vote
0
down vote
favorite
I have recently run across a weird phenomenon:
var timePosSortedMap = TreeMap.empty[String, String]
var timePosSortedMap1 = SortedMap.empty[String, String]
If i try using timePosSortedMap1 in a dataset, then it tends to give worse performance and to throw some exceptions related to insufficient memory in spark cluster; but when i try timePosSortedMap, there is no such issue.
then i am confused. because we can see SortedMap.empty[A, B] = TreeMap.empty[A, B] in scala's source code.
so can anyone help to shed some light on that? Thanks a ton.
scala apache-spark
asked 13 hours ago
kenneth.sun
61
add a comment |
up vote
0
down vote
favorite
I have recently run across a weird phenomenon:
var timePosSortedMap = TreeMap.empty[String, String]
var timePosSortedMap1 = SortedMap.empty[String, String]
If i try using timePosSortedMap1 in a dataset, then it tends to give worse performance and to throw some exceptions related to insufficient memory in spark cluster; but when i try timePosSortedMap, there is no such issue.
then i am confused. because we can see SortedMap.empty[A, B] = TreeMap.empty[A, B] in scala's source code.
so can anyone help to shed some light on that? Thanks a ton.
scala apache-spark
asked 13 hours ago
kenneth.sun
61
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have recently run across a weird phenomenon:
var timePosSortedMap = TreeMap.empty[String, String]
var timePosSortedMap1 = SortedMap.empty[String, String]
If i try using timePosSortedMap1 in a dataset, then it tends to give worse performance and to throw some exceptions related to insufficient memory in spark cluster; but when i try timePosSortedMap, there is no such issue.
then i am confused. because we can see SortedMap.empty[A, B] = TreeMap.empty[A, B] in scala's source code.
so can anyone help to shed some light on that? Thanks a ton.
scala apache-spark
asked 13 hours ago
kenneth.sun
61
I have recently run across a weird phenomenon:
var timePosSortedMap = TreeMap.empty[String, String]
var timePosSortedMap1 = SortedMap.empty[String, String]
If i try using timePosSortedMap1 in a dataset, then it tends to give worse performance and to throw some exceptions related to insufficient memory in spark cluster; but when i try timePosSortedMap, there is no such issue.
then i am confused. because we can see SortedMap.empty[A, B] = TreeMap.empty[A, B] in scala's source code.
so can anyone help to shed some light on that? Thanks a ton.
scala apache-spark
scala apache-spark
asked 13 hours ago
kenneth.sun
61
asked 13 hours ago
kenneth.sun
61
edited 13 hours ago
asked 13 hours ago
kenneth.sun
61
asked 13 hours ago
kenneth.sun
61
asked 13 hours ago
kenneth.sun
61
61
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53370791%2fwhy-do-sortedmap-and-treemap-have-different-impacts-on-performance-in-spark-clus%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
