I think you need to do something like this:

model = Sequential()  
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))  
model.add(RepeatVector(10))
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))  
model.add(TimeDistributed(Dense(1)))
model.add(Activation('linear'))   
model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])  

otherwise you are just repeating the the last Dense layer and getting a constant value.

Thank you very much. I tried your suggestion and the predictions now look like this:

The number of epochs and hidden neurons is the same as in the other testcases, but the prediction is worse for 10 steps compared to 50. Is there a (simple) explanation why it gets worse with more layers? Or does it just need to train longer because it has more parameters to adjust?

I would say that the modeling assumptions of both approaches are different. In the later model, it is assumed that the model sees the complete input sequence (first 50 steps), somehow creates a summary and uses this summary to generate a new signal (last 10 steps).

On the other hand, your initial model estimated the last 50 steps while reading the input signal, no summarisation of the original signal was used.

That's a perfect and clear answer, thank you very much.

HI, I have been studying how to use the many to many model of lstm to predict time series data, and now I have the same problem that you once had, could you share your demo py files about predicting a simple sine wave to me ? I mean i want to learn your code and replace your data with mine just to have a try. it will be very nice of you if you could do me a favor! thanks first !
my email : zhangping16@mails.ucas.ac.cn

thank you !

Here you go!
test_sine.txt

HI there!
It seems in new versions of Keras the input_dim and output_dim arguments are replaced with input_shape() function. So may you edit this parts of code to match to the new version:

model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))  
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))

I also have another question. what is the reason of using model.add(Activation('linear')) ?
Thanks in advanced!

Hi @bestazad ,

You can obtain the same result using input_dim() or input_shape(), to my knowledge both these two "alternatives" has been used for quite some time

https://stackoverflow.com/questions/53106111/in-keras-when-should-i-use-input-shape-instead-of-input-dim

The reasoning the why model.add(Activation('linear')) is used is most likely because this is (only) tentative example, other activation functions can probably give similar results here.

How would you train the model on variable input length?

Hi @gustavz

Two options/suggestions:

1. Padding https://machinelearningmastery.com/data-preparation-variable-length-input-sequences-sequence-prediction/
2. Sequence bucketing https://arxiv.org/ftp/arxiv/papers/1708/1708.05604.pdf

Padding looks easier but I would guess that this method also decreases the usefulness of the model.

If you (or anybode else) could help me with a good explanation of what RepeatVector() does here I would be happy, this is the best reference https://stackoverflow.com/questions/51749404/how-to-connect-lstm-layers-in-keras-repeatvector-or-return-sequence-true , however, this is for a Encode/Decoder network and I'm not sure if this is the same for a LSTM network. E.g., does RepeatVector() that the original input (from the very first layer) or does RepeatVector() work with the inputs/outputs between hidden layers?

what is the difference between:

model = Sequential()  
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))  
model.add(RepeatVector(10))
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))  
model.add(TimeDistributed(Dense(1)))
model.add(Activation('linear')) 

and

model = Sequential()  
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=True))  
model.add(LSTM(output_dim=hidden_neurons, return_sequences=False))  
model.add(Dense(10))

maybe best explained with this image

Thanks for this; which is which? I've added some numbers to your image to better reference the variants. I assume that the code that contains RepeatVector() is represented by variant 4 and that the code that does not contains RepeatVector() is represented by variant 5. Is this correct?

Thanks! :-)

Option 1 is an Encoder-Decoder, Option 2 is a Vanilla LSTM

Option 1 is part 4 of the image?

Thanks for this; which is which? I've added some numbers to your image to better reference the variants. I assume that the code that contains RepeatVector() is represented by variant 4 and that the code that does not contains RepeatVector() is represented by variant 5. Is this correct?

Thanks! :-)

that the code that does not contains RepeatVector() is a many-to-one architecture (variant 3). To have a many-to-many architecture you have to mdofy the code that does not contain RepeatVector() to have return_sequences=True in both LSTM layer and not only only the first layer.

Can anybody will help me in how to write the code for 5th case of the above image?
Specifically in keras.

Anyone to help me with forecasting Time Series using CNN-LSTM, I tried but it doesn't attach the forecasting part to the testing Data.
You can reach me via godjose70@yahoo.com