A group of Congress MLAs seeking the dismissal of Madhya Pradesh minister Vijay Shah for his objectionable remarks against Colonel Sofiya Qureshi were arrested on Friday before being let off later.