I've worked quite a lot with these electronic compasses on mobile phones and its quite possible that there is nothing wrong with your code or sensor.
Instead it could very well be a problem with your environment. There are magnetic fields interfering with the earth's magnetic fields all the time. From electrical equipment interference to the metal structure holding up a building. At the end of the day a compass is just a magnet. If you stand beside a large lump of metal the compass will be attracted to it and point to it rather than the magnetic north pole.
Try this:
Install GPS status
then turn off all filtering (settings... gps & sensors...sensor filtering... no filtering).
Do the calibration (figure of 8 wavy stuff) and then move the phone around your desk.. near monitors, cables, etc. You'll see it go crazy. The information is completely unreliable. I found in the past that moving the phone a few inches to the right completely changed its reading. The same happens with a real compass. Strictly speaking there is no "problem". The device's compass is assigning itself with the strongest magnetic field. Even the magnetic content of nearby rocks can interfere with the compass.
As a further test I've just placed a real (orienteering) compass over my phone which has a compass app installed. The real compass is now pointing everywhere but magnetic North. The two devices are interfering with each other.
So my advice is.. go somewhere in the open, like a park or field, away from any potential interference and power lines, (if you have one bring a real compass to check that the GPS status app is pointing the right way), and see if your compass works as you'd expect.
Extra: The answer from @resus is also important when calibrating. Rotate the phone a few times in each axis. Looks silly but it does calibrate it properly.
Extra 2: Would it be possible/practical to use the compass bearing of your GPS? It would require that the device be moving (walking speed should be fine) but you would not need to worry about any interference. It should give an accurate reading provided your GPS signal is good.
Extra 3: Another thought just occurred to me.. You could try apply a low pass filter to the sensor. This means that the sudden changes in the sensor reading are filtered out .. have a look at this answer. And if that doesn't do a good job there are lots of algorithms on the web for you to choose from.
与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
