Ousted Men's Wearhouse exec. chairman quits board