Java how to build a stream on top of an existing interface

I have an existing interface that allows me to access a theoretically infinite collection as follows:

List<Element> retrieve(int start, int end);



//example

retrieve(5, 10); // retrieves the elements 5 through 10.

Now I would like to build a Java stream on top of this existing interface so that I can stream as many elements as I need without requesting a large list at once.

How would I go about doing this?

I looked at examples of Java streams and all I can find are examples of how to create stream from collections that are completely in memory. I currently load in 30 elements at a time and do the necessary processing but it would be cleaner if I could abstract that logic away and just return a stream instead.

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

how would you implement that retrieve btw? It's obvious if your Collection implements RandomAccess, but if it does not...

– Eugene
Nov 24 '18 at 21:52

also if you really want to return a Stream, why not simply skip(5).limit(10)?

– Eugene
Nov 24 '18 at 21:56

You probably want one of the static factory methods of Stream, like Stream.generate.

– VGR
Nov 24 '18 at 23:08

add a comment |

I have an existing interface that allows me to access a theoretically infinite collection as follows:

List<Element> retrieve(int start, int end);



//example

retrieve(5, 10); // retrieves the elements 5 through 10.

Now I would like to build a Java stream on top of this existing interface so that I can stream as many elements as I need without requesting a large list at once.

How would I go about doing this?

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

how would you implement that retrieve btw? It's obvious if your Collection implements RandomAccess, but if it does not...

– Eugene
Nov 24 '18 at 21:52

also if you really want to return a Stream, why not simply skip(5).limit(10)?

– Eugene
Nov 24 '18 at 21:56

You probably want one of the static factory methods of Stream, like Stream.generate.

– VGR
Nov 24 '18 at 23:08

add a comment |

I have an existing interface that allows me to access a theoretically infinite collection as follows:

List<Element> retrieve(int start, int end);



//example

retrieve(5, 10); // retrieves the elements 5 through 10.

Now I would like to build a Java stream on top of this existing interface so that I can stream as many elements as I need without requesting a large list at once.

How would I go about doing this?

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

I have an existing interface that allows me to access a theoretically infinite collection as follows:

List<Element> retrieve(int start, int end);



//example

retrieve(5, 10); // retrieves the elements 5 through 10.

Now I would like to build a Java stream on top of this existing interface so that I can stream as many elements as I need without requesting a large list at once.

How would I go about doing this?

java java-stream

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

edited Nov 24 '18 at 23:54

Stefan Zobel

2,44031828

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

asked Nov 24 '18 at 21:30

Lasse Jacobs

407610

how would you implement that retrieve btw? It's obvious if your Collection implements RandomAccess, but if it does not...

– Eugene
Nov 24 '18 at 21:52

also if you really want to return a Stream, why not simply skip(5).limit(10)?

– Eugene
Nov 24 '18 at 21:56

You probably want one of the static factory methods of Stream, like Stream.generate.

– VGR
Nov 24 '18 at 23:08

add a comment |

how would you implement that retrieve btw? It's obvious if your Collection implements RandomAccess, but if it does not...

– Eugene
Nov 24 '18 at 21:52

also if you really want to return a Stream, why not simply skip(5).limit(10)?

– Eugene
Nov 24 '18 at 21:56

You probably want one of the static factory methods of Stream, like Stream.generate.

– VGR
Nov 24 '18 at 23:08

how would you implement that retrieve btw? It's obvious if your Collection implements RandomAccess, but if it does not...

– Eugene
Nov 24 '18 at 21:52

also if you really want to return a Stream, why not simply skip(5).limit(10)?

– Eugene
Nov 24 '18 at 21:56

You probably want one of the static factory methods of Stream, like Stream.generate.

– VGR
Nov 24 '18 at 23:08

add a comment |

2 Answers
2

active

oldest

votes

class Chunk implements Supplier<Element> {

    private final Generator generator;

    private final int chunkSize;

    private List<Element> list = Collections.emptyList();

    private int index = 0;



    public Chunk(Generator generator, int chunkSize) {

        assert chunkSize > 0;

        this.generator = generator;

        this.chunkSize = chunkSize;

    }



    @Override

    public Element get() {

        if (list.isEmpty()) {

            list = generator.retrieve(index, index + chunkSize);

            index += chunkSize;

        }

        return list.remove(0);

    }

}

Here I'm assuming retrieve returns a mutable list. If not then you'd need to create a new ArrayList or equivalent at this point.

This can be used as Stream.generate(new Chuck(generator, 30)). It generates an infinite stream starting at index 0. You could add a constructor that allows the starting index to be set if that would be useful.

edited Nov 25 '18 at 4:19

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

add a comment |

I assume you can't edit retrieve method.

You can do this:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(x, x).get(0))

If one term of the sequence depends on the previous term, this would mean recalculating every term up to n if you want the nth term.

This slightly solves the problem by getting it in chunks of 100:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(1 + (x - 1) * 100, x * 100)).flatMap(List::stream)

If you can edit what's behind that interface, you can just make that return a Stream<Element>, using IntStream.iterate as above.

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

Wouldn't this be an infinite stream without a short-circuiting operation? Basically, the chunks range should be the parameters for short-circuiting somehow.

– nullpointer
Nov 25 '18 at 3:36

@nullpointer doesn’t OP want exactly an infinite stream? OP can simply add a limit after that and specify how many he wants.

– Sweeper
Nov 25 '18 at 8:08

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53462529%2fjava-how-to-build-a-stream-on-top-of-an-existing-interface%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

class Chunk implements Supplier<Element> {

    private final Generator generator;

    private final int chunkSize;

    private List<Element> list = Collections.emptyList();

    private int index = 0;



    public Chunk(Generator generator, int chunkSize) {

        assert chunkSize > 0;

        this.generator = generator;

        this.chunkSize = chunkSize;

    }



    @Override

    public Element get() {

        if (list.isEmpty()) {

            list = generator.retrieve(index, index + chunkSize);

            index += chunkSize;

        }

        return list.remove(0);

    }

}

Here I'm assuming retrieve returns a mutable list. If not then you'd need to create a new ArrayList or equivalent at this point.

edited Nov 25 '18 at 4:19

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

add a comment |

class Chunk implements Supplier<Element> {

    private final Generator generator;

    private final int chunkSize;

    private List<Element> list = Collections.emptyList();

    private int index = 0;



    public Chunk(Generator generator, int chunkSize) {

        assert chunkSize > 0;

        this.generator = generator;

        this.chunkSize = chunkSize;

    }



    @Override

    public Element get() {

        if (list.isEmpty()) {

            list = generator.retrieve(index, index + chunkSize);

            index += chunkSize;

        }

        return list.remove(0);

    }

}

Here I'm assuming retrieve returns a mutable list. If not then you'd need to create a new ArrayList or equivalent at this point.

edited Nov 25 '18 at 4:19

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

add a comment |

class Chunk implements Supplier<Element> {

    private final Generator generator;

    private final int chunkSize;

    private List<Element> list = Collections.emptyList();

    private int index = 0;



    public Chunk(Generator generator, int chunkSize) {

        assert chunkSize > 0;

        this.generator = generator;

        this.chunkSize = chunkSize;

    }



    @Override

    public Element get() {

        if (list.isEmpty()) {

            list = generator.retrieve(index, index + chunkSize);

            index += chunkSize;

        }

        return list.remove(0);

    }

}

Here I'm assuming retrieve returns a mutable list. If not then you'd need to create a new ArrayList or equivalent at this point.

edited Nov 25 '18 at 4:19

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

class Chunk implements Supplier<Element> {

    private final Generator generator;

    private final int chunkSize;

    private List<Element> list = Collections.emptyList();

    private int index = 0;



    public Chunk(Generator generator, int chunkSize) {

        assert chunkSize > 0;

        this.generator = generator;

        this.chunkSize = chunkSize;

    }



    @Override

    public Element get() {

        if (list.isEmpty()) {

            list = generator.retrieve(index, index + chunkSize);

            index += chunkSize;

        }

        return list.remove(0);

    }

}

Here I'm assuming retrieve returns a mutable list. If not then you'd need to create a new ArrayList or equivalent at this point.

edited Nov 25 '18 at 4:19

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

edited Nov 25 '18 at 4:19

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

answered Nov 24 '18 at 21:53

sprinter

16.7k32860

add a comment |

I assume you can't edit retrieve method.

You can do this:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(x, x).get(0))

If one term of the sequence depends on the previous term, this would mean recalculating every term up to n if you want the nth term.

This slightly solves the problem by getting it in chunks of 100:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(1 + (x - 1) * 100, x * 100)).flatMap(List::stream)

If you can edit what's behind that interface, you can just make that return a Stream<Element>, using IntStream.iterate as above.

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

Wouldn't this be an infinite stream without a short-circuiting operation? Basically, the chunks range should be the parameters for short-circuiting somehow.

– nullpointer
Nov 25 '18 at 3:36

@nullpointer doesn’t OP want exactly an infinite stream? OP can simply add a limit after that and specify how many he wants.

– Sweeper
Nov 25 '18 at 8:08

add a comment |

I assume you can't edit retrieve method.

You can do this:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(x, x).get(0))

If one term of the sequence depends on the previous term, this would mean recalculating every term up to n if you want the nth term.

This slightly solves the problem by getting it in chunks of 100:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(1 + (x - 1) * 100, x * 100)).flatMap(List::stream)

If you can edit what's behind that interface, you can just make that return a Stream<Element>, using IntStream.iterate as above.

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

Wouldn't this be an infinite stream without a short-circuiting operation? Basically, the chunks range should be the parameters for short-circuiting somehow.

– nullpointer
Nov 25 '18 at 3:36

@nullpointer doesn’t OP want exactly an infinite stream? OP can simply add a limit after that and specify how many he wants.

– Sweeper
Nov 25 '18 at 8:08

add a comment |

I assume you can't edit retrieve method.

You can do this:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(x, x).get(0))

If one term of the sequence depends on the previous term, this would mean recalculating every term up to n if you want the nth term.

This slightly solves the problem by getting it in chunks of 100:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(1 + (x - 1) * 100, x * 100)).flatMap(List::stream)

If you can edit what's behind that interface, you can just make that return a Stream<Element>, using IntStream.iterate as above.

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

I assume you can't edit retrieve method.

You can do this:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(x, x).get(0))

If one term of the sequence depends on the previous term, this would mean recalculating every term up to n if you want the nth term.

This slightly solves the problem by getting it in chunks of 100:

IntStream.iterate(1, x -> x + 1).mapToObj(x -> retrieve(1 + (x - 1) * 100, x * 100)).flatMap(List::stream)

If you can edit what's behind that interface, you can just make that return a Stream<Element>, using IntStream.iterate as above.

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

answered Nov 24 '18 at 21:39

Sweeper

68.2k1073140

Wouldn't this be an infinite stream without a short-circuiting operation? Basically, the chunks range should be the parameters for short-circuiting somehow.

– nullpointer
Nov 25 '18 at 3:36

@nullpointer doesn’t OP want exactly an infinite stream? OP can simply add a limit after that and specify how many he wants.

– Sweeper
Nov 25 '18 at 8:08

add a comment |

Wouldn't this be an infinite stream without a short-circuiting operation? Basically, the chunks range should be the parameters for short-circuiting somehow.

– nullpointer
Nov 25 '18 at 3:36

@nullpointer doesn’t OP want exactly an infinite stream? OP can simply add a limit after that and specify how many he wants.

– Sweeper
Nov 25 '18 at 8:08

Wouldn't this be an infinite stream without a short-circuiting operation? Basically, the chunks range should be the parameters for short-circuiting somehow.

– nullpointer
Nov 25 '18 at 3:36

@nullpointer doesn’t OP want exactly an infinite stream? OP can simply add a limit after that and specify how many he wants.

– Sweeper
Nov 25 '18 at 8:08

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Tukukkk