Any very high traffic can slow a site down or even the server. If, you site is going to very high traffic then if you are using a shared server plan you may need to upgrade to a package to compensate for the high traffic. There too many variables to say exactly what is slowing your site down. I tend to point out the factor of high traffic on a shared server plan being a key more than a VPS or dedicated account. Even though you are allowed a generous amount of bandwidth of a shared server account depending on what functions the site performs rather it would be the issue. Constant connections to sql database providing that there is more than 25 connections at a time being used can cause latency.

I do not see the .htaccess file causing latency unless you have called out directives that generally negate the purpose of the operation of the site. The .htaccess file can cause many things if not properly coded with the right directives.

i have .htaccess directives that deny most of the world, and it doesn't seem to slow anything down.

Yes you can, just have a deny,allow statement in your .htaccess, then have your usual Deny From statement, then after all of those have an Allow From and the IPs/ranges you want to allow.

As a sidenote: depending upon your forum software, it may simply be easier to ban IPs/subnets in there than through .htaccess. vBulletin definitely lets you do this, other forum softwares do as well but I don't know how they do it.