The 6502 had a couple of unique selling points, and the decimal mode was crucial because it was patent protected.
Decimal mode only affects ADC and SBC instructions, and on the NMOS 6502 only usefully sets the C flag. The N, V and Z flags are set, but don't correspond to what you'd expect from a 10's complement decimal operation.
Nonetheless, all four flags are set, so it's worth understanding how they are set, and why. (Link to Ijor's paper, and Bruce's tutorial)
Many (software) emulators have decimal mode correct, and many have it incorrect or missing. Because the CMOS 6502 and later parts set the flags differently, correctness can only be judged relative to a specific part.
We need some test cases - these may yet be added to the py65 test suite (link), once they have been validated against the transistor-level simulation. At the time of writing, py65 will fail some of these tests, but perhaps not for long.
Need a list of interesting signals to probe to observe the decimal mode adjustments. (The presently released JSSim doesn't have C34 named, but it will on next update)
The two operands, and the carry in, are added as a pair of nibbles. The carry-out from bit3 is adjusted in decimal mode, only for ADC. So the ALU is not a binary byte-wide ALU with a decimal adjustment, it is a pair of binary nibble ALUs with a decimal adjustment. In the tests, we don't specifically need to test that carry-in is used (except to prove that carry-out is changing the carry bit, if we have that freedom)
Tests for ADC
- 00 + 00 and C=0 gives 00 and C=0 N=0 Z=1 V=0 (simulate)
Tests for SBC
- 00 - 00 and C=0 gives 99 and C=0 N=1 Z=0 V=0 (simulate)
- 00 - 00 and C=1 gives 00 and C=1 N=0 Z=1 V=0 (simulate)
One form of test program sets all the input flags using PLP:
lda #$c8 pha lda #$00 plp adc #$00 nop
and to calculate what that initial value of PLP should be, we can use a bit more code
php pla EOR #$f7 // #$f6 if we don't want to invert the carry nop