One typically uses DFP (DoubleClick for Publishers) for this. One can also use the Ad-Balancer feature in AdSense but my understanding is that it isn't ideal. Here is a thread discussing that:
[webmasterworld.com...]

Note that the major difference between the two is that Ad-Balancer uses a relative price floor, that is you only show the top XX% of the highest paying ads. So depending on the market for any given day one could be blocking all ads that pay say $0.10 or and then the next it could be $0.25 or less. But XX% of all ads will always be AdSense. DFP lets you set an actual price floor. Say $0.15, so then the number of Adsense ad will vary constantly depending on the market.

Yes you can use passbacks with adsense. Create a new HTML file on the domain, and put third party vendor codes on that html page. Make sure that the sizes match and make sure that they load. On the ad units in adsense interface: Find the option Show Other URL from "If no ads available" (It's toward the end of ad unit setting). Set the URL to the HTML file that you have created. I would keep track of third party vendor's stats to make sure that the passbacks are triggered correctly. (Responsive ads get a little tricky, I will not use default responsive or natives to this purpose).

This is what I do personally. I actually do not know if this is the best method. But it worked wonders for me together with aggressive ad balance setting. (I have not started ad balancer again since it's too deep into Q4 when it was reintroduced, I don't want to mess with it at this stage.) I probably will reorganize my ad stack around End of Q1.

Another side note, be careful when you are designing passbacks and stacks. Watch out for update delays and cache. You want to avoid infinite looping passbacks. This is a common error. (Especially when you first start playing with passbacks). Watch for out of ordinary number of ad requests.

And yes, adsense doesn't have "hard floor" like some of the newer vendors, but passback can surely help when the coverage is low in some cases.

My stack is usually - Third party vendor with a hard floor -> Pass to adsense -> Pass to a final vendor (Typically affiliate). And I stop at the third or fourth hop.

When I had ad balancer aggressively, my stack was Adsense -> Third Party Vendor with Hard Floor -> Final Hops. But this required a bit more tweaking throughout the quarter (at least weekly or even daily) as fillrates / CPM for all the agencies fluctuate. It's a ton of guesswork and experience.