Overall your code is very reasonable for somebody learning Assembly, I would say quite nice work.
I can produce something what will look more advanced (I hope), but it's more like tiny details improved here and there, and some of them for the price of worse readability of source.
I mean on the syntax and low level. On the algorithmic level your code can be improved considerably, as Paweł already answered, but he missed another opportunity to halve number of divisions.
My variant of code, showing:
how to use the result of
DIV ebx (which has to be done anyway to test remainder) as approximation of suggested sqrt(input) test, for earlier exit.
xor reg,reg used to set zero into register (shorter opcode than
mov reg,0 plus modern CPU recognizes it as idiom of "set to zero" and optimizes for it)
test edx,edx to check if remainder is zero (shorter opcode than
cmp edx,0 plus modern CPU will recognize it as idiom of "test for zero value" and optimizes for it)
resolving even numbers at the beginning (even numbers have zero in least significant bit:
isOdd(n) == (n&1). And the only even number being prime is
2, no need to test other even numbers by
but that means only odd divisors have to be tested in the loop, because
(odd_input % even_divisor) != 0 for sure. So my loop is testing divisors 3, 5, 7, 9, 11, 13, 15, ...
some non-trivial usage of arithmetic instructions and flags to check for input values
[0, 1, 2, 3] ahead of loop, as those would break the loop logic and produce invalid result. It may be nice exercise for you to "decipher" how it works.
you should try to keep things "together" (if possible). For example you set up
DIV at the end of your loop code block, so it's A) not together with
DIV, interleaved by
JMP (hard to read for human, no problem for CPU) B) you have code duplicity (
mov edx,0), because you have to init the register values ahead of very first
DIV separately. => So I do initialize
edx:eax right ahead of
DIV, only single time in code, and it's easier to read IMO.
format PE console
section '.text' code readable executable
xor ecx, ecx ; return value = 0 (0 = not prime, 1 = prime)
mov esi, eax ; copy provided number in esi
shr eax, 1 ; eax = eax div 2, remainder into CF
jz finish ; 0 and 1 are not primes
adc ecx,ecx ; temporarily ecx=1/0 for odd/even input
jz input_was_prime ; 2 and 3 are primes, special test needed
; because they would break loop logic
jecxz finish ; other even numbers (but not 2) are *not* primes
; input number was odd and above 3, test odd divisors in generic loop
mov ebx,ecx ; start with divisor 3 (ebx=1, loop will +2 first)
dec ecx ; result = 0 (not prime)
add ebx,2 ; next odd divisor to test
xor edx,edx ; edx:eax = input number (zero extended to 64b)
div ebx ; eax = quotient, edx = remainder
jz finish ; no prime, when remainder is zero
ja test_divisors_loop ; quotient > divisor -> try next
; when quotient <= divisor, it's safe to end loop, esi is prime number
; it's basically cheap "sqrt" end condition, reusing DIV result
mov cl,1 ; result = 1 (is prime) (ecx was 0 or 1 before)
mov eax, ecx ; result was in ecx, print it
(I did debug it in NASM, so if I'm unlucky, something may fail in FASM due to syntax, but it should be easy to fix. But it's highly unlikely, they should have identical syntax for instructions and I don't use any macros/etc)