Rhyme Star uses the PC98 86 soundboard to play the PCM so it cannot be logged as VGM or S98, only way to play the soundtrack files is through PC98 emulation or Hoot (sound driver emulation). If you want only the YM2608 part you can log it yourself with nekoproject.
The sound driver doesn't have support for ADPCM. Other PC98 soundtracks have this limitations to keep it from being logged(i.e. the non-YM2203 version of Eve Burst Error) but usually there's an ADPCM version of the soundtrack if the driver allows it.