Basal-like breast cancers (BLBC) are heterogenous, poorly differentiated and present with limited therapeutic opportunities. We investigated the role of the HLH transcription factor ID4 in breast stem cells and BLBC aetiology. Id4 defines a subset of Cd24+Cd29hi basal mammary epithelial cells enriched for stem/progenitor cell gene expression and mammary repopulating capacity. Id4 is required for mammary ductal morphogenesis in vivo and epithelial cell proliferation in vitro. Id4 overexpression maintains the stem/progenitor cell pool by restricting luminal differentiation, mediated via suppression of BrCa1 and the Notch-Elf5 axis. ID4 is highly expressed by approximately half of all BLBCs, which possess a transcriptional signature resembling mammary stem cells. ID4-positive BLBCs have a very poor prognosis across multiple clinical cohorts. This study identifies ID4 as a key regulator of mammary stem/progenitors and identifies a subset of BLBCs with poor prognosis and a basal cell-like phenotype.