That's a question I was asked for a long time in the past and I have a good answer now.
First of all, it depends on what you want to achieve. If you are a hobbyist and want to pinch some pennies then shared web hosting will work quite well for you.
If you run a profitable business or just want to bring a better experience to your customers then I would recommend to go with a cloud solution such as Azure.
Latency is a key factor here. See, shared hosting providers tend to squeeze the resources up to the maximum. They put lots of customers on a limited set of machines. And it hurts badly at the end. A bold example from my own experience is spontaneous 8 second delays before serving the first byte of a page to website visitors.
A 1 second delay in page response can result in a 7% reduction in conversions.
Just take a look at interesting read about latency impact.
Please note that not all shared hosters are that bad. But most of them are.
So here's the deal. Use a cloud solution instead. It is just more reliable in terms of latency and availability. Better latency => more conversions. A simple math that works for every business.
How much would it cost? A quick answer: just a couple of bucks per month. Azure shared website instance is $9.68/mo. That's quite affordable especially when you have a business that depends on website conversions. The exact cost depends on your needs and perfectly explained in Scott's Hanselman article.
Technologies and Caching
ASP.NET is a native technology for Azure. While you can use PHP, Node.js and others, ASP.NET is the most performant and adapted one. So stick with it, this will cut your costs. JitBit published an insightful read on that topic.
Yet another important aspect is caching. I see a lot of websites everyday and most of them have HTTP caching off. Why is that important? It puts a pressure on your server and degrades user experience. Please make sure your website leverages HTTP caching because it reduces the costs and makes your customers happier. A quick hint for ASP.NET: OutputCacheAttribute.
Tools
I suggest to avoid Content Management Systems (CMS) if you want to make a really fast website. CMS tend to be sluggish for a number of reasons. Instead, use Visual Studio as it brings great productivity and flexibility you need to gain speed.
Put it that way: Visual Studio and ASP.NET already form a nearly perfect system for website authoring.
Visual Studio Express for Web is what you need to get started. This requires programming skills though.
Another tool you may consider is Cloud Combine. It allows to access the site files via FTP as well as to scratch Azure blobs and tables.