Most of the time, the armor is under the jersey and pants.
Not long ago, gear was either for dirt or street and there wasn't much crossover. That has changed a lot just in the last ten years or so. I now have one set of gear that I wear for both street and dual sport. But, it might be a bit much for full on dirt bike riding, like single track trails in the woods.
I have the KLIM Badland Pro jacket and pants. It works well in temps from the high 20s up to the 110 F range. I actually find it more comfortable on hot days than my mesh gear. It is Gore-Tex so there is no messing with liners or over suits. If it gets wet or cold, I just close the vents and keep going. I really like the suit, but it is pricey up front. However, I plan on wearing it for a good 7-8 years at least.
There are all manner of good jackets or armored over skins now. The latter will be a mesh suit with armor all over the place. This is what would typically be worn under a jersey. They are definitely cooler than a full blown street biased jacket. For a long time, I just wore a Joe Rocket Phoenix mesh jacket. I later switched to a Firstgear Kilamanjaro jacket for the weather proofing, but that did not survive a street accident. After that accident, I got the Motoport AirMesh II Kevlar suit.
This stuff is great for protection. It has armor all over the place. I had mine custom sized with some additional custom features. All up I think it was around $1100. I crash tested it numerous times, including hitting a deer and being slammed to the ground at 40mph or so. Not a scratch on me, but my left ankle broke because I had on my street boots instead of my dirt boots. I still have it and wear it sometimes instead of the KLIM if I am not worried about weather issues. It looks crazy bulky, but when riding, I never even notice it. It is quite comfortable.
Pants for dirt only usually have no armor at all. In fact, they usually don't even have any serious abrasion protection. They are just pants. This is why you will often see armored shorts or leggings. These are worn under the pants. This is pretty much the only way you will get any hip protection. Even a lot of street gear lacks good hip protection. That was one of the custom additions I added to the Motoport stuff.
Speaking of boots... You can either go with a full blown dirt boot that offers maximum protection all the way to a full on street boot. The big difference will be the ankle/shin protection. Dirt boots tend to only allow the ankle to move up/down, and not side/side. This is a BIG deal. The only drawback I see is that they are less comfy for walking. On the bike, it is a non issue to me. I have the SIDI Adventure Gore-Tex boots. They are VERY comfy right out of the box. I've tested them several times in scenarios where they kept my feet uninjured despite the presence of rocks and the full weight of the bike on them. They could use a steel cover over the toe though just to protect the leather. They are 100% water proof unless it comes in over the top of the boot. Again, pricey, but well worth it. My SIDI dirt boots are 11 years old and still perfectly good. They just aren't water proof.
Helmets are no biggie. Just find one you like and run with it. I wore a Shoei RF1100 for a long time. Then I got a dirt bike helmet with the goggles and wore that for a few years. I now have the ARAI XD-4 DS helmet. I love it with two exceptions. There is no quick release for the face shield. Mine sometimes has an annoying boiling tea kettle whistle. Most of the time, neither of those are serious issues. It is very comfortable. I find that I never wear goggles with it, but I am usually leading the rides so dust is not as big an issue for me as it would be for those behind me.