Following this blog post from It’s FOSS I decided to make a .fonts folder in my home directory. After that you can download various font’s from Font Squirrel, dafont, Google Fonts, and others and drop the .otf or .ttf files into the .fonts folder.
Then to finally achieve my goal, I made sure I added /home/user/.fonts folder to my fonts as outlined on the GIMP documentation