Well it was more of a realization of that what he was doing was wrong. He realized that his commander had other motives behind what he was doing and it was clouding his judgment; remember in the final battle he did questioned his commander's order to charge the enemy infantry at the incorrect time against Cornwallis’s orders. He only obeyed the two wrong orders was so he wouldn’t be shot since he valued his own life over both the innocent towns people and his neighbor Mel. (Very Jayne like I think)

I also just realized that when they are at the plantation searching for Mel’s family he just stands there watching his captain when he looks under the table when whoever might have been hiding under there could have had a gun pointed right at him waiting. Again very Jayne like I think.

When did he have a change of heart? I thought he obeyed his orders to burn down the church with all the townsfolk inside?