There are 2 issues here 1) the time it takes to build the file index and the scalability of the device doing the indexing. A 500GB Disk will take half the time a 1 Gig disk to build the index. 2) the file index takes room - memory in the router. Your amp was not really designed as a file server. The designers undoubtedly allocated some resources to support the USB port, but I'm pretty sure supporting Multi-Terabyte disk stores was out of scope. The solution is either NAS Ethernet storage which is designed to accommodate large storage volumes or a dedicated media server.
Either way, I hope you have taken steps to protect your investment in time and effort in building that library, by mirroring the disks or by configuring a RAID (Redundant Array of Independent Disks) so that when a disk fails, the array can rebuild itself.