Showing posts with label XHR. Show all posts
Showing posts with label XHR. Show all posts

Friday, 28 June 2019

VBA - Shell a VBScript if you want multi-tasking

Whilst VBA cannot multi-thread, you can always shell to a script. You have many options for choice of script language. If you have already written some logic in VBA then you can easily convert to VBScript. The following script below will download a binary file using an XHR.

I wrote the code in VBA to begin with and debugged it until I was happy. Then I converted to VBScript. The hallmarks of a VBScript are

  • the absence of type clauses in Dim statements
  • one has to use WScript.CreateObject instead of VBA.CreateObject or New
  • no support for constants, so one has to use the literals themselves, typically we put the name in a comment above

The script also has extra proxy logic for that is the use case I am working on but you can take that out. For the time being it serves as an example of how constants have to be passed as literals.

So save the file below as wget.vbs

Dim xhr
set xhr = WScript.CreateObject("WinHttp.WinHttpRequest.5.1")
Call xhr.SetClientCertificate("LOCAL_MACHINE\Personal\My Certificate")

''WinHttp.WinHttpRequestOption_SslErrorIgnoreFlags=4
''WinHttp.WinHttpRequestSslErrorFlags.SslErrorFlag_Ignore_All=13056
xhr.Option(4) = 13056

'Const HTTPREQUEST_PROXYSETTING_PROXY As Long = 2
xhr.setProxy 2, "127.0.0.1:8888", ""

call xhr.Open("GET", WScript.Arguments(0), False)
xhr.setRequestHeader "Referrer-Policy", "no-referrer"
xhr.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
call xhr.Send()



Dim strm
Set strm = WScript.CreateObject("ADODB.Stream")
strm.Type = 1
Call strm.Open()
Call strm.Write(xhr.ResponseBody)

Call strm.SaveToFile(WScript.Arguments(1))
Call strm.Close()

In the same directory save a new workbook then add the following code to call from VBA. Supply web url as the first argument and destination filename (e.g. somewhere in Temp) and the second argument. You can in the script above how it acquires the arguments using WScript.Arguments(0|1)

Sub TestShell()
    
    Dim sShell As String
    sShell = "cscript " & ThisWorkbook.Path & "\wget.vbs https://duckduckgo.com/ " & Environ("temp") & "\wget_example.txt"
    Debug.Print "Shelling" & vbNewLine & sShell
    VBA.Shell sShell
    
End Sub

The only downside to this is that one cannot return variables easily. Often a script will write to a log where the caller can inspect to determine success/error.

Thursday, 7 June 2018

Python - Javascript - MutationObserver - detecting and POST changes to a page

So in the last post I showed how to write a message queue (I have improved that code, so the latest version is on this page). Next I write code in an HTML+javascript web page which detects changes in the web page and posts those changes to our message queue.

At this point I must confess to using Visual Studio to create new Python projects, it gives me Intellisense, but I still run code from the command window. The VS project is relevant in this post because the Python needs to changed to serve up a web page and also accept POST requests but they must come from the same domain otherwise one gets irritating cross domain errors. So keeping the web page and the Python script in the same project makes sense.

So here is a screenshot of my Visual Studio project explorer window.

Chrome Only Please (No IE)

By the way, I only use Chrome for this project. IE is going away, a fact which prompted me to investigated other ways of web-scraping. So this little project has arisen out of the need to move away from IE.

ClockWithMutationObserver.html

So we need a page, ClockWithMutationObserver.html, that display a clock (with thanks to w3schools.com) . The clock's div has an id of clock. Save it in the same directory as the Python script.

<!DOCTYPE html>
<html>
<head>
    <script>
        function startTime() {
            var today = new Date();
            var h = today.getHours();
            var m = today.getMinutes();
            var s = today.getSeconds();
            m = padZero(m);
            s = padZero(s);
            document.getElementById('clock').innerHTML =
                h + ":" + m + ":" + s;
            var t = setTimeout(startTime, 1000);
        }
        function padZero(i) {
            if (i < 10) { i = "0" + i };  // add zero in front of numbers < 10
            return i;
        }
    </script>

</head>

<body onload="startTime()">

    <div style="font-size:72pt" id="clock"></div>

    <script>

        console.log("entering startObserving");
        var MutationObserver = window.MutationObserver || window.WebKitMutationObserver || window.MozMutationObserver;
        if (MutationObserver == null)
            console.log("MutationObserver not available");

        // mutation observer code from https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver
        var targetNode = document.getElementById('clock');

        // Options for the observer (which mutations to observe)
        var config = { attributes: true, childList: true };

        // Callback function to execute when mutations are observed
        var callback = function (mutationsList) {

            for (var mutation of mutationsList) {
                //debugger;
                //console.log(mutation);  //uncomment to see the full MutationRecord
                var shorterMutationRecord = "{ target: div#clock, newData: " + mutation.addedNodes[0].data + " }"

                console.log(shorterMutationRecord);

                var xhr = new XMLHttpRequest();
                xhr.open("POST", "http://127.0.0.1:8000");
                //xhr.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
                xhr.send(shorterMutationRecord);

            }
        };

        // Create an observer instance linked to the callback function
        var observer = new MutationObserver(callback);

        // Start observing the target node for configured mutations
        observer.observe(targetNode, config);

        // Later, you can stop observing
        //observer.disconnect();

    </script>
</body>
</html>

So the above page gives a nice large clock (in 72pt), something like this

20:37:01

Javascript MutationObserver

So in the world of Javascript the Mozilla Developer Network is a good source of documentation. Thankfully, they have a good page on MutationObserver which allows us to detect changes to the DOM.

In the above web page, there are two blocks of JavaScript, (i) the one on the head drives the clock itself; (ii) and the one at the base is the MutationObserver logic. We find the element we want to observe then we define a callback function for when it changes.

MutationRecords

When our callback function is called, we loop through the changes, for each change there is a detailed MutationRecord and they are worth investigating. In the code, the line //console.log(mutation); is commented out. Uncomment that line if you want to see the rich detail given for each change in the Chrome console. Because of all the detail, I copy across the details I want to a new object, actually a string because that is what I will POST back.

XHR to same domain avoid cross domain errors

We then use an AJAX XHR call to POST the data. It is helpful (but not strictly obligatory) to POST back to the same domain whence the page came; this helps to avoid cross origin domain errors.

PythonHTTPMessageQueue.py

So I have some updated Python web server message queue code here. The main change is that all GET requests serve up the ClockWithMutationObserver.html file.


# with thanks to https://blog.anvileight.com/posts/simple-python-http-server/#do-get

from http.server import HTTPServer, BaseHTTPRequestHandler, SimpleHTTPRequestHandler
from io import BytesIO
import tempfile
from socketserver import ThreadingMixIn
import threading

class MyHTTPRequestHandler(SimpleHTTPRequestHandler):

    def do_GET(self):
        self.path = '/ClockWithMutationObserver.html'
        return SimpleHTTPRequestHandler.do_GET(self)

    def do_POST(self):
        content_length = int(self.headers['Content-Length'])
        body = self.rfile.read(content_length)
        self.send_response(200)
        self.end_headers()
        response = BytesIO()
        response.write(b'This is POST request. ')
        response.write(b'Received: ')
        response.write(body)


        # added code to write message to tempfile in temp directory
        msgFName = msgFileName()

        with open(msgFName, 'w+') as msg:
            msg.write(body.decode("utf-8"))
            msg.flush()

        self.wfile.write(response.getvalue())

        # finally add to console so we can see it in the command window
        print(body.decode('utf-8'));

class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
    """Handle requests in a separate thread."""        

def msgFileName():
    # this function uses the date time to generate a filename which hopefully
    # should be unique and allow the files to be sorted
    import datetime
    import time
    ts=time.time()
    timestamp = datetime.datetime.fromtimestamp(ts).strftime('%Y%m%d_%H%M%S.%f')
    fileName = queueDir + '\\' + timestamp + '.txt'
    return fileName


def TempDir():
    #this creates a new directory in the temp folder
    return tempfile.mkdtemp(prefix='MsgQueue')

#Main processing starts here
queueDir =TempDir() #queueDir is in global scope
httpd = ThreadedHTTPServer(('localhost', 8000), MyHTTPRequestHandler)

print("Serve forever, message queue dir:" + queueDir)
httpd.serve_forever()  #code will disappear in here

Running the code, screen shots

So if we start the Python script and we open Chrome and its console window and browse to the address http://127.0.0.1:8000 we get to watch the clock running but we also see activity in the Chrome console window, the command window and the message queue folder. Here are the screenshots.

Final Thoughts

What have we achieved here? Well we've written code to detect changes in a web page and then POST those changes to a HTTP based message queue. Next step would be to detect changes in someone else's page.

What has this got to do with Excel? This example is a Python web server but in this post I have demonstrated that it is possible to use Excel as a web server and so Excel could easily have replaced the Python web server. But this is Python month!

Monday, 5 February 2018

VBA - ProgID - what is the current version

So a SO question arose about a non-installed version of MSXML2.ServerXMLHTTP. This made me wonder why not poke around in the registry to try and find all instances of MSXML2.ServerXMLHTTP in my registry, the results are given in Appendix A. It showed that version 4 is missing just like for the questioner. The registry sweep shows versions 3.0, 5.0 ,6.0 available.

What is really curious is there is a registry key

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Msxml2.ServerXMLHTTP\CurVer

whose default value is

Msxml2.ServerXMLHTTP.3.0

This means if one writes the following code using late binding and no version in the prog id to instantiate a Msxml2.ServerXMLHTTP then one gets a 3.0 version and not a 6.0 version. The rationale is given in this MSDN blog.

Sub CreateXHR()
    Dim oXHR As Object
    Set oXHR = VBA.CreateObject("Msxml2.ServerXMLHTTP")
End Sub

We can write some code to query the registry to tell us what the non versioned prog id actually returns ...

Sub TestCurVersion()

    Debug.Print CurVersion("Msxml2.ServerXMLHTTP")
    '* for me returns Msxml2.ServerXMLHTTP.3.0
    
    Debug.Print CurVersion("Excel.Application")
    '* for me returns Excel.Application.15
    
End Sub


Function CurVersion(ByVal sClass As String) As String
    Const HKLM As Long = &H80000002
    Dim oWMIReg As Object
    
    Set oWMIReg = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & _
             ".\root\default:StdRegProv")
    Dim sReturnString As String
    oWMIReg.GetStringValue HKLM, "SOFTWARE\Classes\" & sClass & "\CurVer", "", sReturnString
    CurVersion = sReturnString
End Function

So unless one wants to rewrite the registry keys to change the current version to version 6.0 then I recommend supplying the string "Msxml2.ServerXMLHTTP.6.0" thus

Sub CreateXHR60()
    Dim oXHR As Object
    Set oXHR = VBA.CreateObject("Msxml2.ServerXMLHTTP.6.0")
End Sub

Simulating the ProgID resolution

With COM the resolution of the ProgID take places by calling CLSIDFromString in OLE32.dll and we can write code to simulate this and then go lookup in the registry


Option Explicit

Private Type GUID
    Data1 As Long
    Data2 As Integer
    Data3 As Integer
    Data4(7) As Byte
End Type

Private Declare Function OLE32_CLSIDFromString Lib "OLE32" _
    Alias "CLSIDFromString" (ByVal lpszCLSID As String, pclsid As GUID) As Long
    
Public Function VBA_CLSIDFromString(ByVal sClass As String) As String
    
    Dim rclsid As GUID
    Dim hr As Long
    hr = OLE32_CLSIDFromString(StrConv(sClass, vbUnicode), rclsid)
    If hr <> 0 Then Err.Raise hr

    Dim sHexCLSID As String
    
    sHexCLSID = "{" & PadHex(rclsid.Data1, 8) & "-" & PadHex(rclsid.Data2, 4) & "-" & _
                PadHex(rclsid.Data3, 4) & "-"
    
    Dim lData4Loop As Long
    For lData4Loop = 0 To 7
        If lData4Loop = 2 Then sHexCLSID = sHexCLSID & "-"
        sHexCLSID = sHexCLSID & PadHex(rclsid.Data4(lData4Loop), 2)
    
    Next lData4Loop
    
    VBA_CLSIDFromString = sHexCLSID & "}"
    Debug.Assert Len(VBA_CLSIDFromString) = 38
End Function

Private Function PadHex(ByVal lNum As Long, ByVal lDigits As Long) As String
    PadHex = Right(String(lDigits, "0") & Hex$(lNum), lDigits)
End Function

Public Function WMI_COMClassVersion(ByVal sClsId As String) As String
    Dim oWMIReg As Object
    Set oWMIReg = GetObject("winmgmts:{impersonationLevel=impersonate}!\" & _
             ".rootdefault:StdRegProv")
    Dim sVersionString As String
    oWMIReg.GetStringValue &H80000002, "SOFTWAREClassesCLSID" & sClsId & "Version", "", sVersionString
    WMI_COMClassVersion = sVersionString

End Function


Public Function WhatVersionOfProgID(sClass As String) As String
    
    Dim sClsId As String
    sClsId = VBA_CLSIDFromString(sClass)
    
    WhatVersionOfProgID = WMI_COMClassVersion(sClsId)
    Exit Function
End Function

Private Sub TestWhatVersionOfProgID()
    Debug.Assert WhatVersionOfProgID("MSXML2.ServerXMLHTTP") = "3.0"
End Sub


Appendix A

Sweeping my registry for instances of MSXML2.ServerXMLHTTP turned up the following


Computer\HKEY_CLASSES_ROOT\CLSID\{88d96a0b-f192-11d4-a65f-0040963251e5}\ProgID
Computer\HKEY_CLASSES_ROOT\CLSID\{AFB40FFD-B609-40A3-9828-F88BBE11E4E3}\ProgID
Computer\HKEY_CLASSES_ROOT\CLSID\{AFB40FFD-B609-40A3-9828-F88BBE11E4E3}\VersionIndependentProgID
Computer\HKEY_CLASSES_ROOT\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\ProgID
Computer\HKEY_CLASSES_ROOT\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\VersionIndependentProgID
Computer\HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{AFB40FFD-B609-40A3-9828-F88BBE11E4E3}\VersionIndependentProgID
Computer\HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{88d96a0b-f192-11d4-a65f-0040963251e5}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\VersionIndependentProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Msxml2.ServerXMLHTTP
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Msxml2.ServerXMLHTTP.3.0
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Msxml2.ServerXMLHTTP.5.0
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Msxml2.ServerXMLHTTP.6.0
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Msxml2.ServerXMLHTTP\CurVer
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{88D969EB-F192-11D4-A65F-0040963251E5}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{88d96a0b-f192-11d4-a65f-0040963251e5}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{AFB40FFD-B609-40A3-9828-F88BBE11E4E3}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{AFB40FFD-B609-40A3-9828-F88BBE11E4E3}\VersionIndependentProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\WOW6432Node\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\VersionIndependentProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Classes\CLSID\{88D969EB-F192-11D4-A65F-0040963251E5}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Classes\CLSID\{88d96a0b-f192-11d4-a65f-0040963251e5}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Classes\CLSID\{AFB40FFD-B609-40A3-9828-F88BBE11E4E3}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Classes\CLSID\{AFB40FFD-B609-40A3-9828-F88BBE11E4E3}\VersionIndependentProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Classes\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\ProgID
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Classes\CLSID\{AFBA6B42-5692-48EA-8141-DC517DCF0EF1}\VersionIndependentProgID


Links

Saturday, 27 January 2018

VBA - XMLHTTP60 - Tricky event handling

Summary: XMLHTTP60 does not have any standard VBA events but by adding a class and pulling a trick in a text editor we can track events.

So neither MSXML2.XMLHTTP60 nor MSXML2.ServerXMLHTTP60 have any standard VBA events that can be trapped by declaring a variable with the WithEvents keyword. This contrasts with the WinHttp.WinHttpRequest class (see prior blog post for example code). However, we can still trap events but we have to pull a trick or two along the way. The official tutorial from Microsoft is given here, Microsoft - Use the onReadyStateChange Property (Visual Basic)

One needs to create a VBA class to handle the events. I give the source next but this is exported file source to be copied into a text editor such as Notepad, saved and then imported in the VBA IDE. Do not cut and paste directly into the VBA IDE. This is because of a line of hidden source code which is given here

Attribute Item.VB_UserMemId = 0

This line will disappear from view once the class module is imported. [In case you're interested in what it does, setting UserMemId=0 makes it the default method which means you can call it whatever you want because the caller will ask for it by its Dispatch Id (0), this is an IDispatch trick]. The top 9 lines also disappear but that is standard behaviour.

VERSION 1.0 CLASS
BEGIN
  MultiUse = -1  'True
END
Attribute VB_Name = "XHRSink"
Attribute VB_GlobalNameSpace = False
Attribute VB_Creatable = False
Attribute VB_PredeclaredId = False
Attribute VB_Exposed = False
Option Explicit

' In text editor, need to put "Attribute Item.VB_UserMemId = 0" on line underneath next line, 
' save and then import into VBA IDE, this sets the default method of this class
' Then we set an instance of this class to the OnReadyStateChange property of
' MSXML2.XMLHTTP60 or MSXML2.ServerXMLHTTP60 to get events
' The line "Attribute Item.VB_UserMemId = 0" will DISAPPEAR from view once imported
Sub OnReadyStateChange()
 Attribute Item.VB_UserMemId = 0

    Debug.Print goXHR.readyState
    If goXHR.readyState = 4 Then
        Debug.Print "sink code handling result"
        Debug.Print goXHR.responseText
    End If
End Sub

So now that the above class is imported, it should read XHRSink in the project folder, we can use it when setting the OnReadyStateChange [N.B. we don't use the Set keyword, this is not a typo!] of XMLHTTP60 (or ServerXMLHTTP60) . We are calling a slow and chunky web service built in a prior blog post.

Option Explicit

'* Tools->References
'MSXML2      Microsoft XML, v6.0      C:\Windows\SysWOW64\msxml6.dll


Global goXHR As MSXML2.XMLHTTP60

'https://msdn.microsoft.com/en-us/library/ms757030(v=vs.85).aspx


Public Sub HttpGet()
    On Error GoTo ErrHandler

    Randomize
    Debug.Print String(10, vbNewLine)

    Dim bAsync As Boolean
    'bAsync = True
    bAsync = False

    Set goXHR = New MSXML2.XMLHTTP60
    
    '* need random number in query parameters to make url unique and stop caching
    goXHR.Open bstrMethod:="GET", bstrURL:="http://localhost:34957/slowAndChunkyWebService?chunkCount=5&random=" & Rnd(1), varAsync:=bAsync
    
    Dim oSink As XHRSink
    Set oSink = VBA.IIf(bAsync, New XHRSink, Nothing)
    
    goXHR.OnReadyStateChange = oSink
    
    
    goXHR.send
    
    Debug.Print "send called with bAsync=" & bAsync
    If bAsync = False Then
        Debug.Print "main code handling result with bAsync=" & bAsync
        If goXHR.readyState = 4 Then Debug.Print goXHR.responseText
    End If
    
SingleExit:
    Exit Sub
ErrHandler:
    Debug.Print "Error (" & Err.Number & ") " & Err.Description
    Stop
    Resume
    
End Sub

To experiment swap the commenting on bAsync = True and bAsync = False. The code reports to the Immediate window what it is doing. Sadly no option to chunk the response is available (not that I know of). Here is some sample reported output when bAsync = True.

send called with bAsync=True
 2 
 3 
 4 
sink code handling result
foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar...

Ok, so that works but how extendable is it? What happens with multiple requests? So for multiple requests one would need to upgrade the XHRSink class with an identifier to tie it back to the source; this is a bit poor but not impossible.

Final thoughts. The limited event handling for XMLHTTP60 needs to be compared and contrasted with (a) WinHttpRequest (b) that found in modern browsers facilitated by jQuery and (c) that found on the web servers such as Node.js I would be very tempted to write web service client code in Node.js and then allow VBA to call in to the finalised and processed results.

Thursday, 25 January 2018

VBA - Node.js - Simple Javascript Webservice

So after having established the limits of ScriptControl and cscript.exe I feel the need to find a Javascript interoperability platform for VBA programmers. The ScriptControl can still parse JSON thanks to Douglas Crockford's scripts still being runnable on Ecmascript v.3 but other javascript libraries are already on Ecmascript v.6.

Javascript Web Service

So we need a new solution and Node.js is the answer. Here we give a simple webservice that takes a Javascript document, extracts some information and returns it. First, the javascript file, open Visual Studio 2017 with Node.js workload installed and open new Node.js console project and paste in the code below.

extractTitleAndUrl()

The extraction logic takes place in extractTitleAndUrl() and is expecting a document of a certain format (actually its a Google Sheets API format) and will extract two facts, title and url from each entry in an array. It adds these two facts to a new smaller object and places them in array. The array is stringified before returning. An error handler traps any problem but foes not give much information. You can see some test data for extractTitleAndUrl() commented out.

The Web Server

Everything that is not extractTitleAndUrl() is web server logic. I'll not explain too much of the plumbing here because other documentation does it better. In the requestHandler() we inspect the url to see if it has suffix '/extractTitleAndUrl' and if so run our logic otherwise print a hello world message. The body of the request is accumulated in chunks because Node.js splits these tasks into very small pieces so that code interleaves, this is the asynchronous model. Once the body is fully received then our logic extractTitleAndUrl() can be executed.

VBA client code is given below

'use strict';

const http = require('http');
const port = 80;

console.log('\nversion Juno\n');

const requestHandler = (request, response) => {

    if (request.url == '/extractTitleAndUrl') {
        //https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/

        let body = [];
        request.on('data', (chunk) => {
            body.push(chunk);
        }).on('end', () => {
            body = Buffer.concat(body).toString();
            // at this point, `body` has the entire request body stored in it as a string
            console.log('\nbody received:\n\n'+body);

            var titleAndUrl = extractTitleAndUrl(body)
            console.log('\nextracted title and url:\n\n' + titleAndUrl );

            response.end(titleAndUrl);
        });

    } else {
        console.log(request.url);
        response.end('Hello Node.js Server!');
    }
}

const server = http.createServer(requestHandler);

server.listen(port, (err) => {
    if (err) {
        return console.log('something bad happened', err);
    }

    console.log(`server is listening on ${port}`);
})

//var doc = {
//    "feed": {
//        "entry":
//        [{ "title": { "$t": "1 Med" }, "link": [{ "href": "https//removed.1.Med.." }] },
//        { "title": { "$t": "2 Dent" }, "link": [{ "href": "https//removed.2.Dent.." }] },
//        { "title": { "$t": "3 Vet" }, "link": [{ "href": "https//removed.3.Vet.." }] }]
//    }
//};

//console.log(JSON.stringify(extractTitleAndUrl(doc)));

function extractTitleAndUrl(text) {

    try {
        var doc = JSON.parse(text);
        var newArray = new Array();

        for (var i = 0; i < doc.feed.entry.length; i++) {

            var newObj = new Object();
            newObj['title'] = doc.feed.entry[i].title.$t;
            if (doc.feed.entry[i].link.length = 1) {
                newObj['url'] = doc.feed.entry[i].link[0].href;
            } else {
                newObj['url'] = doc.feed.entry[i].link[2].href;
            }

            newArray.push(newObj);
        }
        return  JSON.stringify(newArray);
    }
    catch (ex) {
        return ('#error in extractTitleAndUrl!'); 
    }
}

Run the code with the Visual Studio start button, the following should be outputted

Debugger listening on ws://127.0.0.1:15347/1c2b4b25-fa9e-4690-b6a4-524b506491cb
For help see https://nodejs.org/en/docs/inspector
Debugger attached.
(node:5212) [DEP0062] DeprecationWarning: `node --inspect --debug-brk` is deprecated. Please use `node --inspect-brk` instead.

version Juno

server is listening on 80
...

VBA client code

The VBA code is given below. One point of note is that to stop cacheing it is necessary to use ServerXMLHTTP60 and not XMLHTTP60 re this StackOverflow response. The place to start execution is TestWebService(), press F5 there. This should return with the correct results but the console for Node.js should also output some messages...


...
server is listening on 80

body received:

{ "feed": {"entry": [   {     "title": { "$t": "1 Med" },     "link": [ { "href": "https//removed...." } ]   },  
 {     "title": { "$t": "2 Dent" },     "link": [ { "href": "https//removed...." } ]   },  
 {     "title": { "$t": "3 Vet" },     "link": [  { "href": "https//removed...." }]   }] } }

extracted title and url:

[{"title":"1 Med","url":"https//removed...."},{"title":"2 Dent","url":"https//removed...."},{"title":"3 Vet","url":"https//removed...."}]


Option Explicit

'* Tools->References
'MSScriptControl        Microsoft Script Control 1.0        C:WindowsSysWOW64msscript.ocx
'MSXML2                 Microsoft XML, v6.0                 C:WindowsSysWOW64msxml6.dll

Private Function SC() As ScriptControl
    Static soSC As ScriptControl
    If soSC Is Nothing Then


        Set soSC = New ScriptControl
        soSC.Language = "JScript"

        soSC.AddCode "function deleteValueByKey(obj,keyName) { delete obj[keyName]; } "
        soSC.AddCode "function setValueByKey(obj,keyName, newValue) { obj[keyName]=newValue; } "
        soSC.AddCode "function enumKeysToMsDict(jsonObj,msDict) { for (var i in jsonObj) { msDict.Add(i,0); }  } "
        soSC.AddCode GetJavaScriptLibrary("https://raw.githubusercontent.com/douglascrockford/JSON-js/master/json2.js")
        soSC.AddCode "function JSON_stringify(value, replacer,spacer) { return JSON.stringify(value, replacer,spacer); } "
        soSC.AddCode "function JSON_parse(sJson) { return JSON.parse(sJson); } "

    End If
    Set SC = soSC
End Function

Private Function GetJavaScriptLibrary(ByVal sURL As String) As String

    Dim xHTTPRequest As MSXML2.XMLHTTP60
    Set xHTTPRequest = New MSXML2.XMLHTTP60
    xHTTPRequest.Open "GET", sURL, False
    xHTTPRequest.send
    GetJavaScriptLibrary = xHTTPRequest.responseText

End Function

Function SimpleMasterPage() As String

    SimpleMasterPage = "{ ""feed"": {" & _
    """entry"": [ " & _
    "  { " & _
    "    ""title"": { ""$t"": ""1 Med"" }, " & _
    "    ""link"": [ { ""href"": ""https//removed...."" } ] " & _
    "  }, " & _
    "  { " & _
    "    ""title"": { ""$t"": ""2 Dent"" }, " & _
    "    ""link"": [ { ""href"": ""https//removed...."" } ] " & _
    "  }, " & _
    "  { " & _
    "    ""title"": { ""$t"": ""3 Vet"" }, " & _
    "    ""link"": [  { ""href"": ""https//removed...."" }] " & _
    "  }" & _
    "] } }"

    Dim objGutted2 As Object
    Set objGutted2 = SC.Run("JSON_parse", SimpleMasterPage)

End Function

Sub TestWebService()

    '* Do not use XMLHTTP60 because it caches!
    '* https://stackoverflow.com/questions/5235464/how-to-make-microsoft-xmlhttprequest-honor-cache-control-directive#5386957
    
    Dim vBody As Variant
    vBody = SimpleMasterPage


    Dim oXHR As MSXML2.ServerXMLHTTP60
    Set oXHR = New MSXML2.ServerXMLHTTP60
    oXHR.Open "POST", "http://localhost/extractTitleAndUrl"
    oXHR.setRequestHeader "Cache-Control", "no-cache, no-store"
    oXHR.send vBody
    
    Debug.Print oXHR.responseText
    Debug.Assert oXHR.responseText = "[{""title"":""1 Med"",""url"":""https//removed....""},{""title"":""2 Dent"",""url"":""https//removed....""},{""title"":""3 Vet"",""url"":""https//removed....""}]"

    Stop

End Sub


Friday, 19 January 2018

VBA - XHR - ADODB.Stream - Save a file from the Internet

Another code sample this time to download a file from the Internet using XmlHttp request (XHR) in combination with ADODB.Stream from binary writing to disk.


Option Explicit

'* Tools->References
'MSXML2             Microsoft XML, v6.0             C:\Windows\SysWOW64\msxml6.dll

Private Sub TestSaveFileFromInternet()
    Dim sListOfBanks As String
    sListOfBanks = "https://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/authorisations/" & _
                    "which-firms-does-the-pra-regulate/list-of-banks-november-2017-excel.xls/"

    SaveFileFromInternet sListOfBanks, "n:\list-of-banks-november-2017-excel.xls"

End Sub

Private Function SaveFileFromInternet(ByVal sUrl As String, ByVal sSaveToPath As String)
    
    Dim oXHR As MSXML2.XMLHTTP60
    Set oXHR = New MSXML2.XMLHTTP60

    oXHR.Open "GET", sUrl, False
    oXHR.send

    With CreateObject("ADODB.Stream")
        .Open
        .Type = 1
        .write oXHR.responseBody
        .SaveToFile sSaveToPath
        .Close
    End With

End Function