Conservative activists want new speaker