Yes I know, your least-favorite idea goes here. But seriously, someone must have come up with the concept before. Like a bad get-rich-quick scheme could fall into this category, where joining the scheme makes people lose money and become more desperate, so they become more likely to do desperate things like invest more in the scheme. But it can apply to a number of other bad ideas.
I don’t think it is a subcategory I think it’s the term you’re looking for.
The actual phrase has its origins in a financial sense but the way it’s used nowadays is much more broad. You can invest time, money, emotion, identity etc and it’s still the “sunk cost fallacy” if it keeps failing and you keep going.