python numba list of list of tuples












0















I'm trying to speed up my algorithm with numba, after no further improvements where left by using numpy and optimizing.



I have a function that does some computations in a large 2-fold nested loop:



import random
from numba import njit

@njit()
def decide_if_vaild():
return bool(random.getrandbits(1))

@njit()
def decide_what_bin(bins):
return random.randint(0, bins-1)

@njit()
def foo(bins, loops):
results = [ for _ in range(bins)]

for i in range(loops):
for j in range(i+1, loops):
happy = decide_if_vaild()
bin = decide_what_bin(bins)
if happy:
results[bin].append( (i,j) )
# or
# results[bin].append( [i,j] )
return results

if __name__ == '__main__':
x = foo(3,100)


If I run this minimal example above I (as expected) get a typing error:



  File "C:UsersxxxAppDataLocalProgramsPythonPython36libsite-packagesnumbatypeinfer.py", line 104, in getone
assert self.type is not None
numba.errors.InternalError:
[1] During: typing of call at C:/Users/xxx/minimal_example.py (21)
--%<-----------------------------------------------------------------

File "minimal_example.py", line 21


The problem is the: "results[bin].append( (i,j) )" where I try to add a tuple (does not work with lists either) to the list.



The number of bins is kown in advance, but how many elements (2-tuple or list or np.array) depends on how often decide_if_vaild evaluates to True, and as I don't know how often this will be and the computation is quite expensive I don't know any other workaround.



Any good ideas how I could generate the result in the jitted function and return it, or pass a global container that this function could fill?



This probably falls back to:



numba.errors.LoweringError: Failed at nopython (nopython mode backend)
list(list(list(int64))): unsupported nested memory-managed object


where a similiar issue for list(list(int64)) (https://github.com/numba/numba/issues/2560) was resolved in numba 0.39.0 as of https://github.com/numba/numba/pull/2840










share|improve this question




















  • 1





    This code can easily be rewritten with numpy using no for loops. Furthermore with results = [ for _ in range(bins)] you are creating an empty and untyped list. Numba does not like this. And last but not least you should not overwrite internals such as bin.

    – Scotty1-
    Nov 28 '18 at 10:11











  • @Scotty1- How can this be rewritten with numpy? Afaik numpy is not too happy working with dynamic sized vectors. As I dont know the size of the result bins bevor... Regarding the internal bin, this only happend to quickly show a reproducable minimal example.

    – iR0Nic
    Nov 28 '18 at 10:31











  • I don't really know what variable size is, furthermore in your example foo only receives one argument. Thus I can't run your code to test it. But you could get happy with happy = np.random.randint(0, 2, size, dtype=np.bool) and then create results depending on the size of happy. Just try to "rethink" your code. It takes some time at the beginning but once you get used to vectorized code, you'll see it is much easier to read and understand (and faster and much more efficient...).

    – Scotty1-
    Nov 28 '18 at 10:52











  • You are completely right, I'm sorry. I've corrected the example. I see what you mean, I'll give it a tought and a try, thank you!

    – iR0Nic
    Nov 28 '18 at 11:14


















0















I'm trying to speed up my algorithm with numba, after no further improvements where left by using numpy and optimizing.



I have a function that does some computations in a large 2-fold nested loop:



import random
from numba import njit

@njit()
def decide_if_vaild():
return bool(random.getrandbits(1))

@njit()
def decide_what_bin(bins):
return random.randint(0, bins-1)

@njit()
def foo(bins, loops):
results = [ for _ in range(bins)]

for i in range(loops):
for j in range(i+1, loops):
happy = decide_if_vaild()
bin = decide_what_bin(bins)
if happy:
results[bin].append( (i,j) )
# or
# results[bin].append( [i,j] )
return results

if __name__ == '__main__':
x = foo(3,100)


If I run this minimal example above I (as expected) get a typing error:



  File "C:UsersxxxAppDataLocalProgramsPythonPython36libsite-packagesnumbatypeinfer.py", line 104, in getone
assert self.type is not None
numba.errors.InternalError:
[1] During: typing of call at C:/Users/xxx/minimal_example.py (21)
--%<-----------------------------------------------------------------

File "minimal_example.py", line 21


The problem is the: "results[bin].append( (i,j) )" where I try to add a tuple (does not work with lists either) to the list.



The number of bins is kown in advance, but how many elements (2-tuple or list or np.array) depends on how often decide_if_vaild evaluates to True, and as I don't know how often this will be and the computation is quite expensive I don't know any other workaround.



Any good ideas how I could generate the result in the jitted function and return it, or pass a global container that this function could fill?



This probably falls back to:



numba.errors.LoweringError: Failed at nopython (nopython mode backend)
list(list(list(int64))): unsupported nested memory-managed object


where a similiar issue for list(list(int64)) (https://github.com/numba/numba/issues/2560) was resolved in numba 0.39.0 as of https://github.com/numba/numba/pull/2840










share|improve this question




















  • 1





    This code can easily be rewritten with numpy using no for loops. Furthermore with results = [ for _ in range(bins)] you are creating an empty and untyped list. Numba does not like this. And last but not least you should not overwrite internals such as bin.

    – Scotty1-
    Nov 28 '18 at 10:11











  • @Scotty1- How can this be rewritten with numpy? Afaik numpy is not too happy working with dynamic sized vectors. As I dont know the size of the result bins bevor... Regarding the internal bin, this only happend to quickly show a reproducable minimal example.

    – iR0Nic
    Nov 28 '18 at 10:31











  • I don't really know what variable size is, furthermore in your example foo only receives one argument. Thus I can't run your code to test it. But you could get happy with happy = np.random.randint(0, 2, size, dtype=np.bool) and then create results depending on the size of happy. Just try to "rethink" your code. It takes some time at the beginning but once you get used to vectorized code, you'll see it is much easier to read and understand (and faster and much more efficient...).

    – Scotty1-
    Nov 28 '18 at 10:52











  • You are completely right, I'm sorry. I've corrected the example. I see what you mean, I'll give it a tought and a try, thank you!

    – iR0Nic
    Nov 28 '18 at 11:14
















0












0








0








I'm trying to speed up my algorithm with numba, after no further improvements where left by using numpy and optimizing.



I have a function that does some computations in a large 2-fold nested loop:



import random
from numba import njit

@njit()
def decide_if_vaild():
return bool(random.getrandbits(1))

@njit()
def decide_what_bin(bins):
return random.randint(0, bins-1)

@njit()
def foo(bins, loops):
results = [ for _ in range(bins)]

for i in range(loops):
for j in range(i+1, loops):
happy = decide_if_vaild()
bin = decide_what_bin(bins)
if happy:
results[bin].append( (i,j) )
# or
# results[bin].append( [i,j] )
return results

if __name__ == '__main__':
x = foo(3,100)


If I run this minimal example above I (as expected) get a typing error:



  File "C:UsersxxxAppDataLocalProgramsPythonPython36libsite-packagesnumbatypeinfer.py", line 104, in getone
assert self.type is not None
numba.errors.InternalError:
[1] During: typing of call at C:/Users/xxx/minimal_example.py (21)
--%<-----------------------------------------------------------------

File "minimal_example.py", line 21


The problem is the: "results[bin].append( (i,j) )" where I try to add a tuple (does not work with lists either) to the list.



The number of bins is kown in advance, but how many elements (2-tuple or list or np.array) depends on how often decide_if_vaild evaluates to True, and as I don't know how often this will be and the computation is quite expensive I don't know any other workaround.



Any good ideas how I could generate the result in the jitted function and return it, or pass a global container that this function could fill?



This probably falls back to:



numba.errors.LoweringError: Failed at nopython (nopython mode backend)
list(list(list(int64))): unsupported nested memory-managed object


where a similiar issue for list(list(int64)) (https://github.com/numba/numba/issues/2560) was resolved in numba 0.39.0 as of https://github.com/numba/numba/pull/2840










share|improve this question
















I'm trying to speed up my algorithm with numba, after no further improvements where left by using numpy and optimizing.



I have a function that does some computations in a large 2-fold nested loop:



import random
from numba import njit

@njit()
def decide_if_vaild():
return bool(random.getrandbits(1))

@njit()
def decide_what_bin(bins):
return random.randint(0, bins-1)

@njit()
def foo(bins, loops):
results = [ for _ in range(bins)]

for i in range(loops):
for j in range(i+1, loops):
happy = decide_if_vaild()
bin = decide_what_bin(bins)
if happy:
results[bin].append( (i,j) )
# or
# results[bin].append( [i,j] )
return results

if __name__ == '__main__':
x = foo(3,100)


If I run this minimal example above I (as expected) get a typing error:



  File "C:UsersxxxAppDataLocalProgramsPythonPython36libsite-packagesnumbatypeinfer.py", line 104, in getone
assert self.type is not None
numba.errors.InternalError:
[1] During: typing of call at C:/Users/xxx/minimal_example.py (21)
--%<-----------------------------------------------------------------

File "minimal_example.py", line 21


The problem is the: "results[bin].append( (i,j) )" where I try to add a tuple (does not work with lists either) to the list.



The number of bins is kown in advance, but how many elements (2-tuple or list or np.array) depends on how often decide_if_vaild evaluates to True, and as I don't know how often this will be and the computation is quite expensive I don't know any other workaround.



Any good ideas how I could generate the result in the jitted function and return it, or pass a global container that this function could fill?



This probably falls back to:



numba.errors.LoweringError: Failed at nopython (nopython mode backend)
list(list(list(int64))): unsupported nested memory-managed object


where a similiar issue for list(list(int64)) (https://github.com/numba/numba/issues/2560) was resolved in numba 0.39.0 as of https://github.com/numba/numba/pull/2840







python numba






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 28 '18 at 11:06







iR0Nic

















asked Nov 26 '18 at 0:16









iR0NiciR0Nic

7814




7814








  • 1





    This code can easily be rewritten with numpy using no for loops. Furthermore with results = [ for _ in range(bins)] you are creating an empty and untyped list. Numba does not like this. And last but not least you should not overwrite internals such as bin.

    – Scotty1-
    Nov 28 '18 at 10:11











  • @Scotty1- How can this be rewritten with numpy? Afaik numpy is not too happy working with dynamic sized vectors. As I dont know the size of the result bins bevor... Regarding the internal bin, this only happend to quickly show a reproducable minimal example.

    – iR0Nic
    Nov 28 '18 at 10:31











  • I don't really know what variable size is, furthermore in your example foo only receives one argument. Thus I can't run your code to test it. But you could get happy with happy = np.random.randint(0, 2, size, dtype=np.bool) and then create results depending on the size of happy. Just try to "rethink" your code. It takes some time at the beginning but once you get used to vectorized code, you'll see it is much easier to read and understand (and faster and much more efficient...).

    – Scotty1-
    Nov 28 '18 at 10:52











  • You are completely right, I'm sorry. I've corrected the example. I see what you mean, I'll give it a tought and a try, thank you!

    – iR0Nic
    Nov 28 '18 at 11:14
















  • 1





    This code can easily be rewritten with numpy using no for loops. Furthermore with results = [ for _ in range(bins)] you are creating an empty and untyped list. Numba does not like this. And last but not least you should not overwrite internals such as bin.

    – Scotty1-
    Nov 28 '18 at 10:11











  • @Scotty1- How can this be rewritten with numpy? Afaik numpy is not too happy working with dynamic sized vectors. As I dont know the size of the result bins bevor... Regarding the internal bin, this only happend to quickly show a reproducable minimal example.

    – iR0Nic
    Nov 28 '18 at 10:31











  • I don't really know what variable size is, furthermore in your example foo only receives one argument. Thus I can't run your code to test it. But you could get happy with happy = np.random.randint(0, 2, size, dtype=np.bool) and then create results depending on the size of happy. Just try to "rethink" your code. It takes some time at the beginning but once you get used to vectorized code, you'll see it is much easier to read and understand (and faster and much more efficient...).

    – Scotty1-
    Nov 28 '18 at 10:52











  • You are completely right, I'm sorry. I've corrected the example. I see what you mean, I'll give it a tought and a try, thank you!

    – iR0Nic
    Nov 28 '18 at 11:14










1




1





This code can easily be rewritten with numpy using no for loops. Furthermore with results = [ for _ in range(bins)] you are creating an empty and untyped list. Numba does not like this. And last but not least you should not overwrite internals such as bin.

– Scotty1-
Nov 28 '18 at 10:11





This code can easily be rewritten with numpy using no for loops. Furthermore with results = [ for _ in range(bins)] you are creating an empty and untyped list. Numba does not like this. And last but not least you should not overwrite internals such as bin.

– Scotty1-
Nov 28 '18 at 10:11













@Scotty1- How can this be rewritten with numpy? Afaik numpy is not too happy working with dynamic sized vectors. As I dont know the size of the result bins bevor... Regarding the internal bin, this only happend to quickly show a reproducable minimal example.

– iR0Nic
Nov 28 '18 at 10:31





@Scotty1- How can this be rewritten with numpy? Afaik numpy is not too happy working with dynamic sized vectors. As I dont know the size of the result bins bevor... Regarding the internal bin, this only happend to quickly show a reproducable minimal example.

– iR0Nic
Nov 28 '18 at 10:31













I don't really know what variable size is, furthermore in your example foo only receives one argument. Thus I can't run your code to test it. But you could get happy with happy = np.random.randint(0, 2, size, dtype=np.bool) and then create results depending on the size of happy. Just try to "rethink" your code. It takes some time at the beginning but once you get used to vectorized code, you'll see it is much easier to read and understand (and faster and much more efficient...).

– Scotty1-
Nov 28 '18 at 10:52





I don't really know what variable size is, furthermore in your example foo only receives one argument. Thus I can't run your code to test it. But you could get happy with happy = np.random.randint(0, 2, size, dtype=np.bool) and then create results depending on the size of happy. Just try to "rethink" your code. It takes some time at the beginning but once you get used to vectorized code, you'll see it is much easier to read and understand (and faster and much more efficient...).

– Scotty1-
Nov 28 '18 at 10:52













You are completely right, I'm sorry. I've corrected the example. I see what you mean, I'll give it a tought and a try, thank you!

– iR0Nic
Nov 28 '18 at 11:14







You are completely right, I'm sorry. I've corrected the example. I see what you mean, I'll give it a tought and a try, thank you!

– iR0Nic
Nov 28 '18 at 11:14














1 Answer
1






active

oldest

votes


















0














I have now implemented following workaround, even though it does not answer the question completly, it might be a suitable approach for others struggeling with this issue:



@njit()
def foo(bins, loops):
results =
mapping =

for i in range(loops):
for j in range(loops+1, size):
happy = decide_if_vaild()
bin = decide_what_bin(bins)
if happy:
results.append( (i,j) )
mapping.append( bin )
return results, mapping


This will return a list of tuples (supported as of numba 0.39.0) and a list of mappings, where mapping[i] contains the bin for results[i]. Now the jit-compiler works smoothly and I can unpack the results outside of jit.






share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473319%2fpython-numba-list-of-list-of-tuples%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    I have now implemented following workaround, even though it does not answer the question completly, it might be a suitable approach for others struggeling with this issue:



    @njit()
    def foo(bins, loops):
    results =
    mapping =

    for i in range(loops):
    for j in range(loops+1, size):
    happy = decide_if_vaild()
    bin = decide_what_bin(bins)
    if happy:
    results.append( (i,j) )
    mapping.append( bin )
    return results, mapping


    This will return a list of tuples (supported as of numba 0.39.0) and a list of mappings, where mapping[i] contains the bin for results[i]. Now the jit-compiler works smoothly and I can unpack the results outside of jit.






    share|improve this answer




























      0














      I have now implemented following workaround, even though it does not answer the question completly, it might be a suitable approach for others struggeling with this issue:



      @njit()
      def foo(bins, loops):
      results =
      mapping =

      for i in range(loops):
      for j in range(loops+1, size):
      happy = decide_if_vaild()
      bin = decide_what_bin(bins)
      if happy:
      results.append( (i,j) )
      mapping.append( bin )
      return results, mapping


      This will return a list of tuples (supported as of numba 0.39.0) and a list of mappings, where mapping[i] contains the bin for results[i]. Now the jit-compiler works smoothly and I can unpack the results outside of jit.






      share|improve this answer


























        0












        0








        0







        I have now implemented following workaround, even though it does not answer the question completly, it might be a suitable approach for others struggeling with this issue:



        @njit()
        def foo(bins, loops):
        results =
        mapping =

        for i in range(loops):
        for j in range(loops+1, size):
        happy = decide_if_vaild()
        bin = decide_what_bin(bins)
        if happy:
        results.append( (i,j) )
        mapping.append( bin )
        return results, mapping


        This will return a list of tuples (supported as of numba 0.39.0) and a list of mappings, where mapping[i] contains the bin for results[i]. Now the jit-compiler works smoothly and I can unpack the results outside of jit.






        share|improve this answer













        I have now implemented following workaround, even though it does not answer the question completly, it might be a suitable approach for others struggeling with this issue:



        @njit()
        def foo(bins, loops):
        results =
        mapping =

        for i in range(loops):
        for j in range(loops+1, size):
        happy = decide_if_vaild()
        bin = decide_what_bin(bins)
        if happy:
        results.append( (i,j) )
        mapping.append( bin )
        return results, mapping


        This will return a list of tuples (supported as of numba 0.39.0) and a list of mappings, where mapping[i] contains the bin for results[i]. Now the jit-compiler works smoothly and I can unpack the results outside of jit.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 26 '18 at 12:38









        iR0NiciR0Nic

        7814




        7814
































            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53473319%2fpython-numba-list-of-list-of-tuples%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            404 Error Contact Form 7 ajax form submitting

            How to know if a Active Directory user can login interactively

            TypeError: fit_transform() missing 1 required positional argument: 'X'