You said:
I tried to make classes so that I follow Single Responsibility principle.
See Wikipedia pages, Single responsibility principle and Separation of concerns.
Your first class Admin extends Person violates this principle. You have methods addBook and removeBook that then passes on the workload to a Library object. The Admin class should contain state and logic that represents a particular administrator. That class should not know about the bureaucracy of book inventory.
Look at it this way: Suppose you were to expand your software to encompass more of a library's work. Besides adding/removing books from the shelves, an administrator of a library also is involved in daily opening & closing procedures, hiring/firing staff, building maintenance, legal compliance, fire safety inspections, budget planning, generating reports to executives, and so on. Should all that code be added to your Admin class? Obviously, no, that would be overwhelming and confusing, making your software difficult to change or maintain.
To add and remove books from the library's inventory, create an Inventory class with add & remove methods. Those methods might take an Admin object as an argument if the requirements say we need to track which person authorized or executed the addition/removal.