Louisa Matthíasdóttir (1917-2000) was amazing artist. She used her life, family and things around her as inspirations, using clear colors and strong vision of an subject. Louisa moved to New York in 1942 where she painted at home every day. She had many exhibitions and in 1960 she exhibited every year in a Robert Schoelkopf Gallery in New York with a good reviews. I went to see her at Kjarvalstaðir in Reykjavík.
The National Gallery of Iceland is a small gallery but has a good collection of Icelandic art, that includes some of my all time favorites. You can also find a cute small coffee house in there and a small shop with all kinds of designs. The entrance fee is 1500 isl. kr. and with the ticket you can visit two other galleries, so I would say it's worth it!
Marshall húsið is quite new, featuring three small museum; The Living Art Museum, Kling & Bang and studio Ólafur Elíasson. You can also find a restaurant located on the first floor with a view over Reykjavík harbour.