The software selection matrix will help you compare software capabilities.

However, there are other factors which go into selecting a vendor and software.

If you choose open source software, you may have a choice of several vendors who can support that software.  This will give you more options for the present and also for the future where the initial vendor may not be performing well.  You can then replace the vendor with another who is also familiar with the software but may be more responsive to your needs.

Vendor strengths and capabilities

Clearly you want a vendor who has the people experience to support your application.  You should get an idea of the size of the vendor from the number of employees, the number of projects they have under contract, the length of time they have been in business, their experience with this specific application (it will be difficult if this is their first installation of the application).  Vendor experience with the software platform (operating system and programming language and database) is important.

It is important to obtain references from the vendor of current and former clients.  These should be checked carefully with questions that probe the performance and responsiveness of the vendor.  Inquire specifically about any problems and how these were resolved.  Be suspicious if there were no issues or if the answers are vague.

Finally, all software will require some degree of customization or modification to meet your needs.  You should carefully evaluate the vendor's ability to modify software.  This is where speaking with prior clients of the vendor can be valuable.  They can give you a good idea of how capable and responsive to requests for customization the vendor will be.  Here again, using open source software can give you more options in this area.