Don't cheap out on your life jacket. They vary greatly in quality. But the biggest issue is comfort. That may sound odd but if its comfortable you will wear it and the best life jacket is the one you wear. Try one on in the store and then look online for a possible better deal to ensure sizing and fit.
There are lifejackets designed for kayaking that don't ride up when you sit down and are cut in a way that allows free movement of your arms without rubbing and chafing.
Some people will suggest a paddle leash, and while they may sound like a good idea, I'd advise against a paddle leash, especially in moving water - where one of the main hazards is entrapment. Getting entangled in anything in moving water is pretty much a death sentence.
As far as which kayak, which is your biggest piece of equipment, it really depends on your budget and the water you'll be on.
If you intend to fish a lot, or get in and out of the boat frequently, you may consider a sit on top, rather than a sit in.
There are numerous brands and designs, and I would recommend you find a local store where someone knows what the water is like and can recommend a good starter boat based on that. Do a couple a kayak trips, rent a kayak or kayaking test trips. My local shop does test days where you can try over 30 different kayaks.
Try out quite a few before you buy one so that you understand the different feel that the avialable styles have to offer. Generally speaking boats that are sold in big-box stores are only suitable for lakes and slow rivers. They are designed to be easy to get in and out of and easy to handle without training, but that comes with a huge cost in terms of control and capability.
