“Business casual” is a nebulous purgatory. Nobody really knows what it means, except that it doesn't include the increasingly rare suit and tie, and it doesn't descend quite to the depths of gym wear. This leaves the typical working man in a state of confusion and frustration. The most common strategy is to take either the least or the most formal outfit you own and move it towards a middle ground. Thus mongrels such as the suit and tie worn with neither jacket nor tie, or the golf outfit without the gloves and spikes.
This strategy is bound to fail. The result looks like a compromise assembled by a vacationing light packer for an unexpected work situation. If you work in a business casual setting, be prepared for it.
While some suit jackets can be worn as sport coats, suit pants should only be worn with suits. Likewise, anything made by Under Armour should not be worn in an office.
If you're starting your business casual wardrobe, there are two kinds of pants you should buy: those in the tan/khaki family, and those in the light to mid grey family in flannel. These go well with just about anything.
Next you'll want button-down (this refers to the buttons on the collar, not the shirt front) shirts. The buttons help the collar stay up, framing your face, rather than flopping around your collarbones as unbuttoned spread collars tend to do. The sporty origins of the button-down collar also help to informalize it.
From this base, you have a range of formality available to you. You could simply add shoes, socks, and a belt, and be done with it. But to me this looks unfinished.
A cardigan gives you some of the advantages of a jacket – in particular the V shape on your chest that will make you look broader shouldered and slimmer waisted.
Put a sportcoat in place of the cardigan and the outfit becomes slightly dressier. Sportcoats should be of textured fabric – tweeds in the winter, linen in the summer, for instance - to differentiate them from suit jackets. Casual details like contrasting buttons or patch pockets also help to establish the sport coat's identity.
Finally, you can add a knit tie to reach the absolute height of formality within business casual. The only downside is that you'll have to change before your golf meetings.