The palatial building of Mitras came into existence during the times of Durgacharan Mitra who had several business including being court jeweller of Nawab Siraj -ud -Daulah. His Nephew Nilmani Mitra continued the legacy. Later the family’s condition declined. but Nilmani Mitra’s grandson Radhakrishna bought back fortune in the family. It was Radhakrishna who started Durga Puja in 1807.