Overall your code is very reasonable for somebody learning Assembly, I would say quite nice work.
I can produce something what will look more advanced (I hope), but it's more like tiny details improved here and there, and some of them for the price of worse readability of source.
I mean on the syntax and low level. On the algorithmic level your code can be improved considerably, as Paweł already answered, but he missed another opportunity to halve number of divisions.
My variant of code, showing:
how to use the result of DIV ebx (which has to be done anyway to test remainder) as approximation of suggested sqrt(input) test, for earlier exit.
xor reg,reg used to set zero into register (shorter opcode than mov reg,0 plus modern CPU recognizes it as idiom of "set to zero" and optimizes for it)
test edx,edx to check if remainder is zero (shorter opcode than cmp edx,0 plus modern CPU will recognize it as idiom of "test for zero value" and optimizes for it)
resolving even numbers at the beginning (even numbers have zero in least significant bit: isOdd(n) == (n&1). And the only even number being prime is 2, no need to test other even numbers by DIV.
but that means only odd divisors have to be tested in the loop, because (odd_input % even_divisor) != 0 for sure. So my loop is testing divisors 3, 5, 7, 9, 11, 13, 15, ...
some non-trivial usage of arithmetic instructions and flags to check for input values [0, 1, 2, 3] ahead of loop, as those would break the loop logic and produce invalid result. It may be nice exercise for you to "decipher" how it works.
you should try to keep things "together" (if possible). For example you set up edx:eax for DIV at the end of your loop code block, so it's A) not together with DIV, interleaved by JMP (hard to read for human, no problem for CPU) B) you have code duplicity (mov edx,0), because you have to init the register values ahead of very first DIV separately. => So I do initialize edx:eax right ahead of DIV, only single time in code, and it's easier to read IMO.
format PE console
entry start
include 'win32a.inc'
; ===============================================
section '.text' code readable executable
start:
call read_hex
xor ecx, ecx ; return value = 0 (0 = not prime, 1 = prime)
mov esi, eax ; copy provided number in esi
shr eax, 1 ; eax = eax div 2, remainder into CF
jz finish ; 0 and 1 are not primes
adc ecx,ecx ; temporarily ecx=1/0 for odd/even input
dec eax
jz input_was_prime ; 2 and 3 are primes, special test needed
; because they would break loop logic
jecxz finish ; other even numbers (but not 2) are *not* primes
; input number was odd and above 3, test odd divisors in generic loop
mov ebx,ecx ; start with divisor 3 (ebx=1, loop will +2 first)
dec ecx ; result = 0 (not prime)
test_divisors_loop:
add ebx,2 ; next odd divisor to test
mov eax,esi
xor edx,edx ; edx:eax = input number (zero extended to 64b)
div ebx ; eax = quotient, edx = remainder
test edx,edx
jz finish ; no prime, when remainder is zero
cmp eax,ebx
ja test_divisors_loop ; quotient > divisor -> try next
; when quotient <= divisor, it's safe to end loop, esi is prime number
; it's basically cheap "sqrt" end condition, reusing DIV result
input_was_prime:
mov cl,1 ; result = 1 (is prime) (ecx was 0 or 1 before)
finish:
mov eax, ecx ; result was in ecx, print it
call print_eax
push 0
call [ExitProcess]
include 'training.inc'
(I did debug it in NASM, so if I'm unlucky, something may fail in FASM due to syntax, but it should be easy to fix. But it's highly unlikely, they should have identical syntax for instructions and I don't use any macros/etc)
