Magento uses EAV concept so consumes lots of resources. To overcome this if Magento standard practices & tips are applied Magento is fast and can run smoothly on VPS Server or some low level hosting plans.
We have optimized more than 20 Magento website. Proper application of

• Configuration
• Indexing
• Cache
• FPC
• Standard Templating
• Html based optimization

is what is needed to optimize Magento site.
Also check following link : https://stackoverflow.com/questions/9216743/tweaking-magento-for-performance/15458875#15458875.
We do optimize at following levels:
1) Configuration : By checking Flat Catalog settings and other configuration parameters so as to reduce required resources.
2) Indexing.
3) Cache.
4) FPC.
5) Analyzing site from html perspective and reducing html related request and download time.
6) Use of Solr to move search from Mysql to Solr search which in turn will reduce Mysql server load.
7) Standard templating so most of the blocks are cached and fewer sql queries are needed.
Here is indepth explanation:
Configuration :
Make sure flat catalog is used.
compiler mode to be set on.
Merge css and js(Fooman Speedster ).
Your cron should be run in offpeak hours.
Delete additional stores if not in use.
Delete cart rules if not in use.
Minimize customer group as suggested in comment.
Remove staging stores if on EE.
Use of Solr in EE in highly recommended for all features support provided by EE. Bypassing solar and loading on Mysql is no recommended.
Html Level:
Use image sprites to reduce number of request.
optimize image for size.
Use Ajax( Lazy loading of images) where ever possible.
Make your output W3C compliant
Use of image sprites
Use some web test tool to analyse number of requests and other html related parameters responsible for download time and act accordingly.
Server Configuration:
Use query cache but avoid size greater then 64 MB.
Session to be on Ram
Use of APC recommended
Enable Gzip Compression in .htaccess
Load cache on RAM if memory permits.
Reduce hard disc file reads and try reads from ram as this is faster.
Instead of using Apache use nginx or litespeed.
Use Apache mod_expires and be sure to set how long files should be cached.In case you are on Apache server.
Use a Content Delivery Network (CDN).
Enable Apache KeepAlives.
Make sure cron is run so as to clean logs stored in data base.
Number of days log should be minimized as per requirement.
Upgrade PHP version to above 5.3
Code/XML/Concept level:
Remove all modules not in use by removing there xml.Just disabling will not do.
Use Ajax( Lazy loading of images) where ever possible.
CMS blocks take more time then a magento block so unless you want a block to be modified do not use CMS blocks.
Do not use collection count use collection getSize to get what is the collection size.
Minimize number of searchable attributes as these result in columns in flat catalog table and will slow down your search.
If on EE make sure that most pages are delivered without application initialization.Even if one container needs application initialization its going to effect execution speed as apart form URL rewrites most of the other code will get executed.
Check XML for blocks placed in default handle and if those blocks not on specific page then move those XML values from default handle to specific handles.It has been observed that lots of blocks are executed that are not displayed.
If using FPC make sure your containers are cached and repeat request for container is delivered via cache.Improper placeholder definition results in container cache not being used but each time new container content getting generated.
Analyze page blocks and variables and if possible add those variables/blocks to cache.
Solr Level:
Use of Solr for search and layered navigation is recommended.
Standard templating:
Check XML for blocks placed in default handle and if those blocks not on specific page then move those XML values from default handle to specific handles(using Local.xml).It has been observed that lots of blocks are executed that are not displayed.
Use of getChildHtml(‘childName’) is recommended as this will cache block against direct use of block code in .phtml file.
Might help but not recommended:
Switch off Logs writing in Magento.
Remove Admin notification module.