Obviously price is an important factor in choosing a provider - but it should not be the only factor. Customer support can be essential - problems in credit card processing can quickly impact your bottom line. The best way to learn about a provider's level of customer service is to obtain customer referrals from current clients.
Request referrals to merchants that are comparable to your company in size and industry. Then ask these important questions: do they have to wait several minutes before reaching a customer support rep? Are their needs serviced quickly? How does the provider deal with charge backs?
Also ask the provider about their level of support - do they have phones staffed 24 x 7? Do they charge per incident?
Your business will be selling over the Internet and you do not have a secure server that can encrypt credit card information, make sure your provider can offer secure ordering through SSL (secure sockets layer, a widely-used web standard for security). Setting up your own secure site is expensive and technically challenging.
Finally, if you are not using a bank or financial company you recognize, make sure you verify that the company you are investigating is legitimate - there are con artists and scammers who set up fake processing companies just to collect setup fees then vanish. Contact the Better Business Bureau to check the company's status if you are unsure, and if you find a provider on the Web, make sure you get a physical address and phone number.