I tried to search using CreatedBy managed property in advanced search webpart but no results were returned.
The Author managed property however returned correct results. Both these managed properties are created by SharePoint itself. A closer look at these properties revealed that Author managed property is mapped to Office:4(Text) crawled property.
whereas CreatedBy managed property is mapped to ows_Created_x0020_By(Text) crawled propery. I added Office:4(Text) to CreatedBy managed property and it started working.