IF you have valuable content, or a group of viewers then you should install a forum if it will add value to your website.  Numbers aren't that important.

I have two forums for a couple of clients. 

One caters to about a 1,000 people total.  They don't post much but it is used to disseminate information to the members. 

The second has exactly 50 users, with some custom functions that they use to run a small organization.. 

A forum will generate traffic. Don't wait for traffic to build up and then install a forum. You're going about it backwards.

Don't forget to install various add-ons and to enable features to make life difficult for spammers. Any good forum software will have such things. You want features that will make it difficult for "bots" to register (e.g., graphical "captcha" puzzles), you don't want to allow "guests" to post (only registered members), and you may want to limit newcomers' ability to post links to other sites or to start new topics/threads until they've got a certain number of posts under their belt. Of course, you need to keep on top of forum activity to swiftly execute spammers -- they will attempt to register and to post junk, no matter what your forum is about or how big it is.