Is it permissible to take books confiscated by customs and official institutions after notifying the department, and what is the expiation if it is not permissible, especially given that there is no owner for these books?

Smuggling, if it does not result in harm, then people should not be prevented from selling their goods. Therefore, customs prohibition would be illegitimate. And assuming the prohibition is permissible, then the financial penalty should be for the general welfare of Muslims, not for individuals. Accordingly, what some institutions do by reviewing customs warehouses to withdraw some books for their own benefit is considered an injustice. Therefore, it is not permissible for you to take the books with the permission of your department.