Return distribution over set of action space from Neural Network
up vote
0
down vote
favorite
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 at 15:52
Siddhant Tandon
253
add a comment |
up vote
0
down vote
favorite
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 at 15:52
Siddhant Tandon
253
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 at 15:52
Siddhant Tandon
253
I am trying to build a neural network to output a probabilistic distribution over set of whole action space.
My action space is a vector of 3 individual actions : [a,b,c]
a can have 3 possible actions within itself a1,a2,a3 and similarly b has b1,b2,b3, c has c1,c2,c3. So In total i can have 27 different combinations of these actions 3^3 = 27. Ultimately the neural network should output 27 combinations of these actions (which is a matrix of 27 x 3) : [[a1,b1,c1],[a2,b2,c2],[a3,b3,c3],[a1,b1,c2],[a1,b1,c3],.....] and so on for all 27 combinations. Just to mention the input to my network is a state which is a vector of 5 elements.
I want a probability associated to each of these 27 combinations.
I know I can associate probability by using a softmax with 27 outputs but I don't understand how the network can output a matrix in this case where every row has a probability associated to it.
python tensorflow neural-network probability reinforcement-learning
python tensorflow neural-network probability reinforcement-learning
asked Nov 19 at 15:52
Siddhant Tandon
253
asked Nov 19 at 15:52
Siddhant Tandon
253
asked Nov 19 at 15:52
Siddhant Tandon
253
asked Nov 19 at 15:52
Siddhant Tandon
253
asked Nov 19 at 15:52
Siddhant Tandon
253
253
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 at 16:04
rvd
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 at 16:41
|
show 1 more comment
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 at 16:04
rvd
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 at 16:41
|
show 1 more comment
up vote
0
down vote
accepted
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 at 16:04
rvd
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 at 16:41
|
show 1 more comment
up vote
0
down vote
accepted
up vote
0
down vote
accepted
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 at 16:04
rvd
43117
Is there any reason you want it to return a matrix of these actions? Why not just map each of the 27 combinations to integers 0-26? So your architecture could look like [Linear(5, n), ReLU, Linear(n, .) ... Softmax(Linear(., 27))]. Then when you need to evaluate, you can just map it back to the action sequence. This is similar to how in NLP tasks you map multidimensional word vectors to integers via stoi for training and bring them back via itos.
I should point out that if your training paradigm involves some further use of these discrete choices (say you use the argmax of this upstream of another net), then the nondifferentiability of argmax means that this architechture will not learn anything. I only mention this because you use the phrase "action space", which is typical in DRL. If this is the case, you may want to consider algorithms like REINFORCE where action sequences can be learned discretely and used upstream via policy gradient.
answered Nov 19 at 16:04
rvd
43117
answered Nov 19 at 16:04
rvd
43117
answered Nov 19 at 16:04
rvd
43117
answered Nov 19 at 16:04
rvd
43117
43117
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 at 16:41
|
show 1 more comment
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more theargmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here
– Siddhant Tandon
Nov 19 at 16:15
So you want the Q-values of each of the three values fora,b, andc? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.
– rvd
Nov 19 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elementsa,b,cas output and repeat this 27 times ?
– Siddhant Tandon
Nov 19 at 16:28
Not quite. I mean make the network emit a sequence like[a2, b1, c3]and strongly penalize making invalid sequences like 2as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.
– rvd
Nov 19 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 at 16:41
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more the
argmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here– Siddhant Tandon
Nov 19 at 16:15
actually what i would ultimately want is a Q-value associated to each of the elements in the matrix. Can you explain a bit more the
argmax, i ask this because in DQN you take the argmax to select the Q values and then just put that in a loss functio, & everything works fine. This is what im trying to do here– Siddhant Tandon
Nov 19 at 16:15
So you want the Q-values of each of the three values for
a, b, and c? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.– rvd
Nov 19 at 16:19
So you want the Q-values of each of the three values for
a, b, and c? Why not just do the learning one step at a time (make one action, evaluate, make next action, etc.) and set your action space as all 27 options? You can also evaluate after all three steps are taken, you just need to store a small action history list. Yes for a DQN it's fine, just wanted to be sure that you aren't using this discrete choice to then inform some further trainable construction; it's fine to feed this directly into the loss.– rvd
Nov 19 at 16:19
ah you mean something like feed the input in the network get a vector of 3 elements
a,b,c as output and repeat this 27 times ?– Siddhant Tandon
Nov 19 at 16:28
ah you mean something like feed the input in the network get a vector of 3 elements
a,b,c as output and repeat this 27 times ?– Siddhant Tandon
Nov 19 at 16:28
Not quite. I mean make the network emit a sequence like
[a2, b1, c3] and strongly penalize making invalid sequences like 2 as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.– rvd
Nov 19 at 16:31
Not quite. I mean make the network emit a sequence like
[a2, b1, c3] and strongly penalize making invalid sequences like 2 as in a row (or simply make it impossible). After the three actions are emitted, assign Q values and backprop the loss gradient. Your Q table is of size 27.– rvd
Nov 19 at 16:31
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 at 16:41
I still have issues understanding it, but I think I will try the approach of assigning integers. I might comment again on this thread, thank you very much.
– Siddhant Tandon
Nov 19 at 16:41
|
show 1 more comment
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53378284%2freturn-distribution-over-set-of-action-space-from-neural-network%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
