You're actually doing pretty good. Affiliate marketing is a 90/10 game. You have more than 10% active. you can always work to help more affiliates get active, but deleting inactives is something I wouldnt do if I still managed programs.
Here are some blogs I've written on the subject that should help you understand why I feel that way.
Booting Inactive Affiliates? Affiliate Activation Campaigns Gone Bad
http://affiliate-blogs.5staraffiliat...ion-campaigns/
Top 7 Reasons Your Affiliates Could be Inactive - Don’t Expire Them
http://affiliate-blogs.5staraffiliat...ates-inactive/
Ultimate Affiliate Activation Email from zZounds Affiliate Manager
http://affiliate-blogs.5staraffiliat...ivation-email/