See the Nvidia CUDA download page for download and documentation.
The information below was current as of late 2007. Since then Nvidia has added support for Mac OS X, and a Vista version is in the works.
CUDA currently works on either 32-bit Windows XP or 32- or 64-bit linux. Only certain linux distributions are officially supported, and the only free one is openSUSE, which made the choice straightforward for me. I have the 64-bit version at work and the 32-bit version at home. As far as CUDA performance goes there is very little difference between the two. Apparently CUDA can work on other distributions but this is not guaranteed.
A key point to note though is that in 32-bit mode ints, longs and pointers are 32 bits long and long longs are 64 bits long, whereas in 64-bit mode only ints are 32 bits long and the rest are 64 bits long.
For 32-bit linux and multiple graphics cards you may have to edit your boot options in the grub menu.lst file as detailed in the Nvidia driver notes; I had to add the line "uppermem 524288" before the "kernel ..." line and had to append "vmalloc=192MB" to the end of the kernel line.
You may have to mess around a bit to get the primary graphics card to be the one you want it to be if you're able to have a separate card for graphics only. On my home machine I had to make sure the onboard GPU was set to "always on" and in the pci options that the onboard GPU was primary.
Particularly when running in emulation mode, you may need to increase the stack size; in bash a suitable command is "ulimit -s unlimited".
If you have dedicated card for CUDA then you don't want it mentioned in your xorg.conf file at all. Your other nvidia graphics card (or onboard GPU) should be there and have the "nvidia" driver for it. With all the autodetection that can go on now I found it simplest to install the operating system and Nvidia driver before putting the CUDA card(s) into the machine at all.
I did need to add 'Option "HWCursor" "Off"' to the device section of my xorg.conf to get the Nvidia driver to display the mouse arrow properly with my 6100 onboard graphics.
For runtime linking you either have to set your LD_LIBRARY_PATH variable to have "/usr/local/cuda/lib" in it, or, perhaps better, add the same to /etc/ld.so.conf and run ldconfig as root.
Also don't forget to add "/usr/local/cuda/bin" to your path.
For a while I was a bit concerned about various diagnostic messages of the form "NVRM: Xid ..." that seeed to crop up in the system error log as revealed by the dmesg command. While messages of this nature can spell bad news, I traced mine to a benign cause: whenever I terminated a CUDA program prematurely using Ctrl-c one of those messages popped up.