Receive byte string (high speed position output) from FANUC robot, parse and append to dated .xml file

Question

This script just opens a TCP server to listen for real-time position data from a FANUC robot to pass to another sensor software we are using (Luna Terahertz sensor).

So far everything is working great, I get an .xml file generated with the data from the FANUC with each datapoint getting an accurate timestamp.

However I don't know what I don't know and would just like some constructive feedback on the way I went about generating the xml file or any flaws you see in my overall formatting or commenting. One thing I'm also interested in is how to better format the incoming data using element

# import socket, datetime and time libraries

import socket
from datetime import datetime
import time


def main():
    #   Define TCP/IP server address and port
    ip_address = '192.168.5.5'
    port = 48200
    
    #   Initialize server socket and begin listening for data
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.bind((ip_address, port))
    server_socket.listen(1)
    
    #   Print message indicating server IP, port and listen status
    print(f"Server listening on {ip_address}:{port}")
    
    # Once a client attempts to send data, receive data, decode bytes into utf-8 string
    # 
    # Lines 39-41 capture current timestamp values to be used in file generation
    #
    # Create xml file in append mode with title "<YearMonthDay>_TPCdata" in .xml format
    # 
    # Write line with granular timestamp, down to microsecond in front of position data
    # 
    # Write position data after timestamp and insert new line for formatting
    #
    # Once the data is received and written, close the client connection 
    # (CAUTION: not closing this leads to FANUC controller locking up)

    while True:
        client_socket, client_address = server_socket.accept()
        print(f"Connection established with {client_address[0]}:{client_address[1]}")
        date_time = datetime.fromtimestamp(time.time())
        timestr = date_time.strftime("%Y%m%d")
        timestrgran = date_time.strftime("%H:%M:%S.%f")
        data = client_socket.recv(1024)
        if data:
            xml_data = data.decode('utf-8')
            with open(timestr+"_TPCdata.xml", mode='a', encoding ='utf-8') as xfile:
                xfile.write(timestrgran)
                xfile.write(" :: ")
                xfile.write(xml_data)
                xfile.write("\n")
                xfile.close
            print(f"Received data: {xml_data}")
            client_socket.close()

if __name__ == "__main__":
main()

The results are in-line with my expectations, just want to know if I did this the most efficient way.

Here is an example of the data being stored in the xml file:

16:09:15.519923 :: <POS><usec>68360821</usec><G1><X>433.619476</X><Y>-801.910278</Y><Z>279.477539</Z><W>-177.122894</W><P>5.610122</P><R>-48.283722</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>
16:09:21.093852 :: <POS><usec>73936913</usec><G1><X>433.619507</X><Y>-801.910278</Y><Z>279.477600</Z><W>-177.122879</W><P>5.610115</P><R>-48.283710</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>
16:09:23.637498 :: <POS><usec>76480955</usec><G1><X>433.619476</X><Y>-801.910339</Y><Z>279.477600</Z><W>-177.122894</W><P>5.610116</P><R>-48.283707</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>
16:09:28.926435 :: <POS><usec>81769041</usec><G1><X>433.619415</X><Y>-801.910217</Y><Z>279.477539</Z><W>-177.122879</W><P>5.610129</P><R>-48.283707</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>
16:09:29.903188 :: <POS><usec>82745057</usec><G1><X>433.619476</X><Y>-801.910278</Y><Z>279.477539</Z><W>-177.122894</W><P>5.610122</P><R>-48.283722</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>
16:10:22.648158 :: <POS><usec>6639596</usec><G1><X>406.542969</X><Y>-805.941467</Y><Z>240.309143</Z><W>-178.196320</W><P>3.798053</P><R>-48.379639</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>
16:10:51.550988 :: <POS><usec>35544047</usec><G1><X>395.019043</X><Y>-792.794189</Y><Z>225.621002</Z><W>-178.700150</W><P>1.136544</P><R>-44.721104</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>

Answer 1

It's good that you have a main, but you need to move some of its contents into subroutines.

It would benefit you to move ip_address and port to argparse parameters of the program.

# Lines 39-41 is an odd way to comment code - better to move that comment up to those actual lines.

It seems that client_address is a 2-tuple of address and port (the documentation is ambiguous and I have not tested this). That being the case,

client_socket, client_address = server_socket.accept()

should become

client_socket, (client_address, client_port) = server_socket.accept()

to use tuple unpacking and avoid the indexing operations.

datetime.fromtimestamp(time.time()) is just datetime.now().

Your socket access pattern is strange and probably incorrect. You accept one recv, and as soon as you get a segment of non-zero length you close the socket! TCP connections are somewhat expensive and intended to be long-lived; so you should keep receiving data as long as the socket is open.

data.decode('utf-8') is a problem for several reasons. First, if these data are coming from a controller, are you sure that the encoding is UTF-8? Very often, embedded systems use ISO-8859 and don't support multi-byte character sets as they're more complex to support, pack, align and serialize. Anyway: you shouldn't decode at all. Just accept the bytes from the socket and write it straight out to the file in binary mode. Bonus: If the controller did use UTF-8, you would eventually encounter an error where packet fragmentation would split one codepoint and the decode would fail.

Speaking of fragmentation: one of the fundamentals of raw socket I/O is that there are no guarantees of message bound. If you're always getting a message that starts with <POS> and ends with </IO>, that's only by luck. Adding other devices on the network, or other conditions like driver congestion will unpredictably break this assumption, at which point you'll be mixing your timestamps in random locations of the XML string. There are various ways to fix this; typically you'd want to build up a buffer queue and only release a message from the queue when it's detected complete.

Don't mix :: free-form text in with XML. Just add an element of your own, something like

<client timestamp="nnn..." /><POS ...

xfile.close doesn't do anything because you forgot parens (). Even so, don't call close() because you already have a with that will perform the close.

Your call to main() needs to be indented.

Suggested

I've only tested this with telnet; please test on your own as well.

import argparse
import logging
import socket
import sys
from datetime import datetime, date
from typing import Iterator, Iterable

logger = logging.getLogger('fanuc')

ENCODING = 'iso-8859-1'


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description='Receive and save data from FANUC')
    parser.add_argument('-i', '--ip-address', default='192.168.5.5', help='IP address to bind')
    parser.add_argument('-p', '--port', default=48200, type=int, help='Port to bind to')
    return parser.parse_args()


def write_lines(get_data: Iterable[bytes]) -> None:
    start_date = None
    file = None

    try:
        while True:
            data = next(get_data)

            new_date = date.today()
            if start_date != new_date:
                start_date = new_date
                filename = f"{start_date:%Y%m%d}_TCPdata.xml"
                if file is not None:
                    file.close()
                file = open(filename, mode='ab')

            file.write(f'<client time="{datetime.now():%H:%M:%S.%f}"/>'.encode(encoding=ENCODING))
            file.write(data)
            file.write(b'\n')
            file.flush()
    finally:
        if file is not None:
            file.close()


def socket_iterator(ip_address: str, port: int) -> Iterator[bytes]:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
        server_socket.bind((ip_address, port))
        server_socket.listen()
        logger.info("Server listening on %s:%d", ip_address, port)

        while True:
            client_socket, (client_address, client_port) = server_socket.accept()
            logger.info("Connection established with %s:%d", client_address, client_port)

            while True:
                data = client_socket.recv(1024)
                if len(data) > 0:
                    logger.debug('Received data: %s', data.decode(ENCODING))
                    yield data
                else:
                    break
            logger.warning('Disconnected')


def main():
    logging.basicConfig(level=logging.INFO)
    args = parse_args()

    try:
        write_lines(socket_iterator(args.ip_address, args.port))
    except KeyboardInterrupt:
        pass


if __name__ == "__main__":
    main()

INFO:fanuc:Server listening on 0.0.0.0:48200
INFO:fanuc:Connection established with 127.0.0.1:52003
WARNING:fanuc:Disconnected

Answer 2

First and foremost, the socket data you retrieve is not actually XML but a markup that resembles XML and your resulting file output is more a .txt than .xml. XML is an industry standard with special rules such as a single root node and no text existing outside of elements or attributes. See W3C 1.0 Specifications. As example, the following format fails both above conditions, so by definition is not XML.

16:09:15.519923 :: <POS><usec>68360821</usec><G1><X>433.619476</X><Y>-801.910278</Y><Z>279.477539</Z><W>-177.122894</W><P>5.610122</P><R>-48.283722</R></G1></POS><CYCLE>1</CYCLE><IO>1</IO>

Consider actually generating XML using conformant DOM libraries like Python's standard library package, xml.etree.ElementTree. Also, avoid treating XML like a text file in IO read/write operations. One of the biggest peeves in programming is treating XML as string.

Specifically, use etree to parse markup to XML tree, build elements and attributes, then write the tree to file for a true XML output. Below will create or append each socket data fragment into a <DATA> element with date and time attributes, then appends each socket data to a running daily XML file with a single root node, <ROOT> to start and close tree.

import os
import xml.etree.ElementTree as ET
...

    while True:
        client_socket, client_address = server_socket.accept()
        print(f"Connection established with {client_address[0]}:{client_address[1]}")
        date_time = datetime.fromtimestamp(time.time())
        timestr = date_time.strftime("%Y-%m-%d")
        timestrgran = date_time.strftime("%H:%M:%S.%f")
        xml_file = f"{timestr}_TPCdata.xml"

        data = client_socket.recv(1024)
        if data:
            if not os.exists(xml_file):
                # CREATE XML FILE WITH EMPTY ROOT
                ET.fromstring("<ROOT/>").write(xml_file, encoding="utf-8")

            # PARSE SOCKET DATA INTO XML TREE
            xml_socket = ET.fromstring(f"<DATA>{data}</DATA>")
            # ADD DATE/TIME ATTRIBUTES
            data_elem = xml_socket.find('DATA')
            data_elem.set('date', timestr)
            data_elem.set('time', timestrgran)

            # APPEND DATA ELEMENT TO DAILY XML
            xml_tree = ET.parse(xml_file)
            xml_tree.find('ROOT').append(xml_socket)

            # SAVE XML TREE
            ET.indent(xml_tree)         # INDENT ADDED IN PYTHON 3.9
            xml_tree.write(xml_file, encoding="utf-8")

            print(f"Received data: {data}")
            client_socket.close()

Output

<?xml version="1.0" encoding="utf-8"?>
<ROOT> 
  <DATA date="2023-11-24" time="16:09:15.519923">
    <POS>
      <usec>68360821</usec>
      <G1>
        <X>433.619476</X>
        <Y>-801.910278</Y>
        <Z>279.477539</Z>
        <W>-177.122894</W>
        <P>5.610122</P>
        <R>-48.283722</R>
      </G1>
    </POS>
    <CYCLE>1</CYCLE>
    <IO>1</IO>
  </DATA>
  <DATA date="2023-11-24" time="16:09:21.093852"/>
    <POS>
      <usec>73936913</usec>
      <G1>
        <X>433.619507</X>
        <Y>-801.910278</Y>
        <Z>279.477600</Z>
        <W>-177.122879</W>
        <P>5.610115</P>
        <R>-48.283710</R>
      </G1>
    </POS>
    <CYCLE>1</CYCLE>
    <IO>1</IO>
  </DATA>
  ...
</ROOT>