VBA - Python - POSTing to a Web Service Message Queue

So previously I've shown VBA calling Python code via a late bound COM interface. COM is not the only interop technology, clearly we have HTTP and REST APIs. So let's write some code to make a Python web service. Initially, I'll write this single-threaded and later I'll use a Multi-threaded Mixin to increase the thread pool.

First Draft - Bounce POST back payload to client (Single Threaded)

There is plenty of code to show a python web server handling GET requests but I want to handle POST requests because I want this Python web service to act like a message queue that receives messages in JSON or XML and immediately writes them to disk (I'll let another different process handle the main processing of these disk-written messages). P.S. I have kept GET code as well so one can browser test the server.

The following is a python executable to be run from the command line so no need for a COM registration code.

# with thanks to https://blog.anvileight.com/posts/simple-python-http-server/#do-get

from http.server import HTTPServer, BaseHTTPRequestHandler
from io import BytesIO

class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):

    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello, world!')

    def do_POST(self):
        content_length = int(self.headers['Content-Length'])
        body = self.rfile.read(content_length)
        self.send_response(200)
        self.end_headers()
        response = BytesIO()
        response.write(b'This is POST request. ')
        response.write(b'Received: ')
        response.write(body)
        self.wfile.write(response.getvalue())

        # finally add to console so we can see it in the command window
        print(body.decode('utf-8'));


httpd = HTTPServer(('localhost', 8000), SimpleHTTPRequestHandler)
print("Serve forever")
httpd.serve_forever()

So above we can see some imports, then the class to handle the requests, then the code to run the HTTP server. This is quite compact, as compact as any Node.js code I have seen. SimpleHTTPRequestHandler inherits from BaseHTTPRequestHandler and we simply override the do_GET and do_POST methods. So open a command window and (assuming you have your environment variables set up correctly) run it with simply PythonHTTPMessageQueue.py ...

C:\Users\Simon\source\repos\PythonHTTPMessageQueue\PythonHTTPMessageQueue>PythonHTTPMessageQueue.py
Serve forever

And we need some VBA client code. One could uses the MSXML XHR classes but this time I'm gonna use WinHTTPRequest.

Option Explicit

Sub TestPythonPost()

    '* Tools -> References -> Microsoft WinHTTP Services, version 5.1
    Dim oWinHttpRequest As WinHttp.WinHttpRequest
    Set oWinHttpRequest = New WinHttp.WinHttpRequest
    
    Dim sURL As String
    sURL = "http://127.0.0.1:8000"
    oWinHttpRequest.Open "POST", sURL, False
    oWinHttpRequest.SetRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
    oWinHttpRequest.SetRequestHeader "Content-type", "application/x-www-form-urlencoded"
    
    '***************************************************************************
    '* Send JSON
    '***************************************************************************
    oWinHttpRequest.Send ("{ ""name"":""John"", ""age"":30, ""car"":null }")

    Debug.Print oWinHttpRequest.ResponseText
    '* outputs: This is POST request. Received: { "name":"John", "age":30, "car":null }
    

End Sub

So a run of the above code prompt some console output in the command window as well as the Immediate window ...

C:\Users\Simon\source\repos\PythonHTTPMessageQueue\PythonHTTPMessageQueue>PythonHTTPMessageQueue.py
Serve forever
127.0.0.1 - - [06/Jun/2018 16:29:03] "POST / HTTP/1.1" 200 -
{ "name":"John", "age":30, "car":null }

Second Draft - write message to file (Single Threaded)

So lets write the message queue logic. We want to create a folder in our temp directory, we do this using tempfile.mkdtemp(prefix='MsgQueue'). For each message in that folder we want a unique filename so we'll use a datetime string formatter function timestamp = datetime.datetime.fromtimestamp(ts).strftime('%Y%m%d_%H%M%S.%f'). Also, we need to introduce how to open a file, write to it and commit using

        with open(msgFName, 'w+') as msg:
            msg.write(body.decode("utf-8"))
            msg.flush()

So here is amended Python program, VBA needs no change. I've added some comments (which begin with #)

# with thanks to https://blog.anvileight.com/posts/simple-python-http-server/#do-get

from http.server import HTTPServer, BaseHTTPRequestHandler
from io import BytesIO
import tempfile

class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):

    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello, world!')
        print(queueDir)

    def do_POST(self):
        content_length = int(self.headers['Content-Length'])
        body = self.rfile.read(content_length)
        self.send_response(200)
        self.end_headers()
        response = BytesIO()
        response.write(b'This is POST request. ')
        response.write(b'Received: ')
        response.write(body)

        # added code to write message to tempfile in temp directory
        msgFName = msgFileName()

        with open(msgFName, 'w+') as msg:
            msg.write(body.decode("utf-8"))
            msg.flush()

        self.wfile.write(response.getvalue())

        # finally add to console so we can see it in the command window
        print(body.decode('utf-8'));

        

def msgFileName():
    # this function uses the date time to generate a filename which hopefully
    # should be unique and allow the files to be sorted
    import datetime
    import time
    ts=time.time()
    timestamp = datetime.datetime.fromtimestamp(ts).strftime('%Y%m%d_%H%M%S.%f')
    fileName = queueDir + '\\' + timestamp + '.txt'
    return fileName


def TempDir():
    #this creates a new directory in the temp folder
    return tempfile.mkdtemp(prefix='MsgQueue')

#Main processing starts here
queueDir =TempDir() #queueDir is in global scope
httpd = HTTPServer(('localhost', 8000), SimpleHTTPRequestHandler)

print("Serve forever, message queue dir:" + queueDir)
httpd.serve_forever()  #code will disappear in here

Final Draft - use mixin to make Multi-threaded

To get multi-threading working requires inheriting from a mixin which is a type of multiple inheritance, see SO for more detail on mixins. So I need some import statements

from SocketServer import ThreadingMixIn
import threading

And I define a new class which multiply inherits

class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
    """Handle requests in a separate thread."""      

And then instantiate that instead, here is the final draft

# with thanks to https://blog.anvileight.com/posts/simple-python-http-server/#do-get

from http.server import HTTPServer, BaseHTTPRequestHandler
from io import BytesIO
import tempfile
from socketserver import ThreadingMixIn
import threading

class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):

    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello, world!')
        print(queueDir)

    def do_POST(self):
        content_length = int(self.headers['Content-Length'])
        body = self.rfile.read(content_length)
        self.send_response(200)
        self.end_headers()
        response = BytesIO()
        response.write(b'This is POST request. ')
        response.write(b'Received: ')
        response.write(body)

        # added code to write message to tempfile in temp directory
        msgFName = msgFileName()

        with open(msgFName, 'w+') as msg:
            msg.write(body.decode("utf-8"))
            msg.flush()

        self.wfile.write(response.getvalue())

        # finally add to console so we can see it in the command window
        print(body.decode('utf-8'));


class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
    """Handle requests in a separate thread."""        

def msgFileName():
    # this function uses the date time to generate a filename which hopefully
    # should be unique and allow the files to be sorted
    import datetime
    import time
    ts=time.time()
    timestamp = datetime.datetime.fromtimestamp(ts).strftime('%Y%m%d_%H%M%S.%f')
    fileName = queueDir + '\\' + timestamp + '.txt'
    return fileName


def TempDir():
    #this creates a new directory in the temp folder
    return tempfile.mkdtemp(prefix='MsgQueue')

#Main processing starts here
queueDir =TempDir() #queueDir is in global scope
httpd = ThreadedHTTPServer(('localhost', 8000), SimpleHTTPRequestHandler)

print("Serve forever, message queue dir:" + queueDir)
httpd.serve_forever()  #code will disappear in here

Final Thoughts

Well adding the multi-threading was just so easy. I can imagine that sometimes one might want to switch back to single threading when debugging so it looks easy to switch.

Also, it turns out that Python already has a Queue that can be passed to a ThreadedHTTPServer as per this SO question.