You won't be penalized for it by the search engines and here is why - what would stop your competition from paying someone to submit your site to thousands of directories? If they penalized you for it, this would be happening every day.
If you put your link into popular directories you might not get much
seo rank weight from it, but the inbound links - AKA - visitors, is what makes it worth it. Traffic is king, but don't fall into the trap of putting your site on link farms.