As with everything, keeping your browser windows open while you run all of your tests will not work in every instance.
Sometimes, you may have a site that sets server-side cookies that Selenium is unaware of. In this case, clearing out your cookies may have no effect and you may find that closing down the browser is the only way to ensure a clean environment for each test.
If you use InternetExplorerDriver, you will probably find that when you use slightly older versions of Internet Explorer (for example, Internet Explorer 8 and Internet Explorer 9), your tests will get slower and slower until they grind to a halt. Unfortunately, older versions of IE are not perfect and they do have some memory leak issues.
Using InternetExplorerDriver does exacerbate these issues because it is really stressing the browser. As a result, it does get a lot of unfair press. It's an excellent bit of code that deals with an awful lot of crap that gets thrown at it.
This is not to say that you can't use this method; you may not see any issues with the application that you are testing. You can of course use a mix of strategies; you can have multiple phases of testing. You can put tests that are able to reuse the browser in the first phase. You can then put tests that need a browser restart in your second phase.
Removing the browser shutdown and startup time after each test really does make a massive difference to the speed of your test runs. From personal experience, I would suggest that you should always try to keep the browser open whenever realistically possible to keep your test times down.
At the end of the day, the only way to be sure if it will work for you is experimentation and hard data. Just remember to do that investigation first. Once you are done, you should then tailor your thread usage to each browser/machine combination or, you should set a baseline that works with everything in your environment.