Ultracold gases of highly magnetic lanthanide atoms have enabled the realization of dipolar quantum droplets and supersolids. However, future studies could be limited by the achievable atom numbers and hindered by high three-body loss rates. Here we study density-dependent atom loss in an ultracold gas of 166Er for magnetic fields below 4 G, identifying six previously unreported, strongly temperature-dependent features. We find that their positions and widths show a linear temperature dependence up to at least 15 µK. In addition, we observe a weak, polarization-dependent shift of the loss features with the intensity of the light used to optically trap the atoms. This detailed knowledge of the loss landscape allows us to optimize the production of dipolar Bose-Einstein condensates with more than 2 × 105 atoms and points towards optimal strategies for the study of large-atom-number dipolar gases in the droplet and supersolid regimes.