Millions of kilometers of rails need to be inspected throughout the world. Based on vision inspection, rail surface defect detection in a large number of rail images becomes an efficient way to assess rail health status. However there are not only common stock (straight) rails but also many different types associated with rail crossing and connecting, collectively called multi-type rails. Dark rusty regions of rail surfaces and rail bottoms are hard to differentiate in the multi-type rail region segmentation. We discuss how to extract more reliable features to solve this problem by introducing attention-based context information fusion. The experimental results show that the attention-based context information fusion has a positive effect on multi-type rail region segmentation.