Photodynamic therapy (PDT) has attracted tremendous attention in the antitumor and antimicrobial areas. To enhance the water solubility of photosensitizers and facilitate their accumulation in the tumor/infection site, polymeric materials are frequently explored as delivery systems, which are expected to show target and controllable activation of photosensitizers. This review introduces the smart polymeric delivery systems for the PDT of tumor and bacterial infections. In particular, strategies that are tumor/bacteria targeted or activatable by the tumor/bacteria microenvironment such as enzyme/pH/reactive oxygen species (ROS) are summarized. The similarities and differences of polymeric delivery systems in antitumor and antimicrobial PDT are compared. Finally, the potential challenges and perspectives of those polymeric delivery systems are discussed.