Found the actual issue why it was breaking and properly fixed it
The vote system we use that emulates the TF2 F1-F5 menu had a very rare yet possible server crash bug related to memory handling that has been fixed.
