Long-text machine reading comprehension (LT-MRC) requires the machine to answer questions based on a lengthy text, such as stories, academic papers, and books. Although machine reading comprehension (MRC) has been one of the cutting-edge research fields in natural language processing, MRC on long documents poses several extra challenges to the model, which requires a more effective discovery and modeling of distant concept dependencies and more efficient training methods and architectures to decrease Transformer's $O(n^2)$ time complexity for longer inputs. In this talk, we present three directions toward building a better LT-MRC system. For more effective concept dependency modeling, we introduce HEF and QEN, which are state-of-the-art taxonomy expansion and completion models that can build hierarchical knowledge graphs based on concept-describing text pieces to assist graph-based MRC approaches. We also introduce two ongoing projects which improve PLM's focus on both the document and question to generate answers more related to the given context, and an improvement to the latest global convolutional models to allow learnable, flexible, and adaptive distant dependency modeling with lower time complexity than Transformers. For future works, we would like to investigate more efficient architectures for longer inputs, and better utilizing layouts to improve LT-MRC models.