In order to understand a coherent text, humans infer semantic or logical relations between textual units that may or may not be explicitly stated. For example, in "I am hungry. I did not have lunch today." the reader infers a "causality" relation, even if it is not explicitly stated via a term such as "because". This linguistic mechanism is called "implicit discourse relations". The current state of the art demonstrates that automatically identifying implicit relations is much more difficult than for explicit relations. In this talk, we will present our work on the use of various traditional machine learning models and Convolutional Neural Networks with pretrained word embeddings to improve discourse relation recognition (DRR). Results at the recent CoNLL-2016 shared task show early promise with deep neural networks, but no added gain with pretrained embeddings. As recent research has also shown how correlation mechanism in neural networks can increase DRR, we will then introduce our recent focus on Encoder-Decoder type networks with attention mechanism for DRR.