I also had no - zero - interest in racing. At ~ 50 years old, I neither bounce nor heal well.
Riding my street bike at track days has absolutely, definitely made me a better - safer - street rider.
I ride slower on the street - because I have experienced the thrill-of-faster on the track. And have the sense to keep it there.
I ride safer on the street - because I have better skills (braking, cornering, throttle control, etc) -AND- because I trust the bike more (and trust that it's lmits are well above what I thought).
Figure it this way - with less skill, you are capable of taking a turn at maybe 60 MPH, but with more skill you could take it at 80 MPH. When street riding at 50 MPH, the less-skilled rider is riding at 83% of ability - only 17% in reserve. The more skilled rider is riding at only 63% of their ability - they have 37% in reserve "capacity." That is over twice the reserve - safety margin.
Finally, I ride more comfortably on the street, and can devote more attention to street-stuff (oncoming traffic, dogs, oil spills, whatever) because i am able to devote less attention to the basics / mechanics of riding / operating the bike. The ride is more fun.
Yes, there are crashes at track days. My experience is that they are mostly in the level 2 groups - the level 1 guys (beginners) are tightly controlled and many are nervous, theny KNOW they are slow, and so they crash less. The level 3 guys (experts) are seriously skilled and fast, and experienced enough that crashes are pretty rare (but often spectacular when they occur!). The guys in the middle - Level 2 - include the street squids who THINK they deserve to be in Level 3, and the guys who just moved up from Level 1 and are pushing too hard to keep up. So, again - I have seen the most crashes in Level 2.
Summary: There is no - zero - downside / disadvantage to track days. The upsides are real. Knowledge, skill, confidence.