For irregular (non rectangular) images, you use a photo editor like Photoshop or GIMP to erase all the unwanted parts and replace them with transparency. The overall image will still be rectangular, but much of it will now be transparent, and whatever other image is under it will show through. You'll probably need to use CSS to position the images so that they overlay each other. Note that in the example you gave, the branch, mountain range, wood panel, and some background were all composited into a single image, rather than overlaying multiple partial images at the browser. However, the principle of erasing / transparency is still useful here.

Note that transparency is a tricky subject, because of limited support by some browsers. All browsers support transparency in GIF files, but 1) GIFs aren't really good for photos and 2) GIF transparency is all or nothing (completely transparent or completely opaque). The PNG file format is good for photos and allows variable amounts of transparency (good for transitions and blending at edges), but  IE6 and earlier browsers don't support PNG transparency. There are some tricks involving certain filters that can be applied to at least IE6, but it's a clumsy fix. Note that in the example you gave, the end result has no transparency -- it's 100% opaque, so this is not an issue.

Rounded corners and/or shadow effects are usually done by breaking up the frames into 8 or so edge and corner images. If you use a div for each, and the image is set in the background, I think you can define thin slices of image and use CSS to repeat them in one direction. This permits a wide range of side lengths. I would suggest looking at the HTML and CSS code behind a page that has this effect, to see how it's done (browser View > Page Source).