Hi @ebell,
Welcome to FTC! 
As @PorkyBelly said, definitely consider Animal for Saturday night dinner instead of HLAY, especially since you want to visit it (don’t miss out on the amazing Foie Gras Loco Moco).
Sunday Brunch? Gjusta is pretty amazing, but it has limited seating and isn’t a place you can hang for a few hours like you’re asking about.
I second @PorkyBelly’s rec for Connie & Ted’s, since you’re asking about a place with seafood & oysters, and Connie & Ted’s does it well. Here’s a post with some pics from a few visits we had if it helps - Connie & Ted’s.
But Nobu Malibu seems like a swell way to spend the afternoon as well.
Bakeries: Republique, Sycamore Kitchen (on the same street!). If you just want to experience fantastic plain Croissants (and Almond Croissants) consider Chaumont.
Hope you enjoy L.A.!