If you want something more minimal rather than a fully-fledged "Harry Potter" space, why not opt for subtle items like bookends? Parenting website What to Expect recommends Hogwarts-shaped bookends for a magic touch. People love to buy babies books as presents as they make sweet memories, so you'll get plenty of use out of your bookends. Bookends can also be used to hold up bottles of baby lotion too, making them a versatile purchase. 

If you want something grander, mom blog ScaryMommy recommends this beautiful three-tier bookcase. Colorful and honoring all houses equally with its four crests, it will brighten up a nursery of any size.

Alternatively, placing a few prints with quotes that feel particularly special to you around your baby's nursery will let people know you like "Harry Potter" but won't be overbearing. You could go for quotes from your favorite character, like these prints that feature beautiful illustrations and quotes from Dumbledore.