This totally depends on your goals!
First you'll want to think about how many worms you want total.
1 Pound of worms can eat about half their weight a day, and will take up about 1 square foot of space.
Need to vermicompost 7 pounds of food a week? You'll eventually want at least 2 pounds of worms.
Want to fill a 3 square foot bin up with worms? You'll eventually want 3 pounds of worms.
Second, you need to decide how quickly you want to get there, and at what cost.
For example, if you want to have 4 pound of worms eventually, you can either buy all 4 of those pounds at once, or start with a smaller amount like half a pound.
If you buy all 4 pounds at once, you'll reach high efficiency a lot quicker, but it will cost more money up front.
Or you can start with less worms (like half a pound), save money, and ease into worm composting, but it will take longer to get to top performance as the worms need to reproduce.
Another thing to consider is that the more worms you start with, the more you'll need to make sure you're providing them a good home. 4,000 worms will be somewhat more demanding than only 500.
If its your first time and you're still unsure, a half pound up to 2 pounds is a good place to start!