I want functionality I can dress up the way I want, something I can stick into a table so that it doesn't take over the look of my website.
Ah yes, therein lies the rub. Most, if not all, carts I've seen are built to be complete (sub)sites in and of themselves. They're meant to be turnkey systems, but the tradeoff is that they tend to insist on owning the layout and everything. You have to go diving through the code and substantially modify them in order to get a certain "look and feel" compatible with the rest of your site. The cart can be a standalone site, so it may bring in all sorts of extra baggage that's not wanted and may even interfere with integrating the cart into a larger site.
I've been studying Dreamweaver CS3 and CSS and want to push the site into a slicker direction, and _at the same time_ organize what's sold into a database, add more shopping functionality, and create a system that's easier for me to update. I think most of the features shopping carts offer won't interest me. I'm at just as interested in maintaining the aesthetic of the site as I am to generating more sales.
It appears that "drop-in" cart functionality like PayPal's is rather limited compared to "all-in-one" solutions. You keep the look and feel of the rest of your site, at the cost of having very limited cart capability.
is there a shopping cart out there that's ideal for me? Or am I better off learning more advanced features, dynamic development, javascript, etc. Or is that very quixotic?
I would recommend against trying to re-invent the wheel -- lots of people have put lots of effort into making commercial-grade shopping carts very robust and very capable. Customers and (worse,) hackers will find all sorts of holes in any home-brew cart.
I have spent quite a few months going over the code in osCommerce to further modularize it, make it MySQL 5 compatible, and in general streamline it. My intent is to split it up into modules and to have some number of "templates" or frameworks into which all these modules are plugged in, rather than the existing large number of monolithic pages. The look and feel of each page would be in its template, while most of the functionality would be the plug-ins. The idea is that osCommerce would ship with a vanilla page layout (header, footer, three columns) so that it continues to be a turnkey system. However, you could replace the templates with your own page templates to match the rest of your site. They might include all or only part of current available function. Furthermore, you could have custom content for each item, by pointing to your own code files, rather than accepting their "one size fits all" one image and and a bunch of text for each item. Something like that. You would also be able to select the template or style used based on category, subcategory, item, etc.
My current job (50-60 hour weeks) leaves me mentally exhausted by the end of the day, so I'm not making much headway on this right now. However, if there's sufficient interest, maybe I'll stir myself to take another crack at it. I've always intended to offer it back to osCommerce as a new framework for their version 3.0 or 3.1 (which I have no idea what will look like). If they don't want it, maybe I'll fork it off into a new product?