To address the issues of complex backgrounds and poor segmentation performance for small ship objects in sea–land port areas, we propose a sea–land port segmentation algorithm based on spatial and semantic alignment fusion. The algorithm utilizes parallel Transformer–convolutional-neural-network (CNN) dual-branch encoders for feature extraction and introduces two modules: spatial alignment fusion and semantic alignment fusion. By the collaborative work of four submodules: spatial feature alignment, spatial feature fusion, semantic feature alignment, and semantic feature fusion, the dual-branch network achieves feature alignment and fusion. The spatial and semantic alignment fusion module efficiently combines local details extracted by the Transformer–CNN dual-branch with global semantic information. This enhances the model's ability to understand and analyze complex sea–land port scenes, effectively addressing low segmentation accuracy of port ship objects and the overlapping and occlusion of port objects. Experimental results demonstrate that the proposed sea–land port segmentation algorithm achieves optimal segmentation accuracy on two publicly available sea–land port segmentation datasets, ISDSD and HRSC2016-SL.