Tuesday 27 August 2019

VBA - Export a worksheet to CSV

I just had a need to export a worksheet to CSV (comma separated value) file and was surprised that one must create a new workbook and copy over the sheet. I would have thought Microsoft would have written a nice 'ExportToCSV' method by now. Anyway the code samples of StackOVerflow did not satisfy me so I wrote my own, here is my code.

Option Explicit

Sub TestExportSheetToCsv()
    Dim sDir As String
    sDir = "C:\Users\Simon\source\repos\PythonBaseHttpPlusQueryParams\PythonBaseHttpPlusQueryParams\"
    
    ExportSheetToCsv ThisWorkbook, sDir, "DomainToBrokerMap"
    ExportSheetToCsv ThisWorkbook, sDir, "BrokerQuerySelectors"
    ExportSheetToCsv ThisWorkbook, sDir, "MarketDataItems"
    ExportSheetToCsv ThisWorkbook, sDir, "BrokerDrilldown"
    
End Sub

Sub ExportSheetToCsv(ByVal wbSrc As Excel.Workbook, ByVal sDir As String, ByVal sSheetName As String)
    Application.ScreenUpdating = False
    Dim wbExport As Excel.Workbook
    Set wbExport = Workbooks.Add

    Dim wsExport As Excel.Worksheet
    Set wsExport = ThisWorkbook.Worksheets.Item(sSheetName)
    wsExport.Copy , wbExport.Worksheets.Item(1)

    Application.DisplayAlerts = False
    wbExport.Worksheets.Item(1).Delete
    Application.DisplayAlerts = True
    
    Dim sExportFileName As String
    sExportFileName = sDir & sSheetName & ".csv"
    
    If Len(Dir(sExportFileName)) > 0 Then
        Kill sExportFileName
        Debug.Assert Len(Dir(sExportFileName)) = 0
    End If
    
    wbExport.SaveAs sExportFileName, FileFormat:=xlCSV, CreateBackup:=False
    
    Debug.Print "Exported " & sExportFileName
    
    wbExport.Close False
    Application.ScreenUpdating = True
End Sub


Friday 9 August 2019

VBA - New Python COM classes !

So by default the Python Com Gateway class does not ship an intrinsic type library; this is a shame because Python has its own reflection capabilities and could do so easily IMHO. The official sample gives and compiles an Interface Definition Language (IDL) file but one has to maintain the IDL in sync with one's class. In this post I give code that gets Python to reflect on a class and automate the writing and compiling of the IDL into a type library.

Once you have a type library then you can create objects in VBA by adding a Tools->Reference and using New instead of using CreateObject(), you will get Intellisense . You will also get the object propertles in the VBA Locals and Watch windows.

DesignatedWrapPolicy

In Python COM behaviour is driven by policies, we want a type library so we'll need to use a DesignatedWrapPolicy policy which fortunately is the default. The doc string is worth quoting as it gives a round up of the attributes we need to give a class to make a type library appear,_typelib_guid_ and _typelib_version. This is taken from doc string of the DesignatedWrapPolicy class in win32com/server/policy.py . The opening remarks of that file also detail what a policy is.

class DesignatedWrapPolicy(MappedWrapPolicy):
  """A policy which uses a mapping to link functions and dispid
     
     A MappedWrappedPolicy which allows the wrapped object to specify, via certain
     special named attributes, exactly which methods and properties are exposed.

     All a wrapped object need do is provide the special attributes, and the policy
     will handle everything else.

     Attributes:

     _public_methods_ -- Required, unless a typelib GUID is given -- A list
                  of strings, which must be the names of methods the object
                  provides.  These methods will be exposed and callable
                  from other COM hosts.
     _public_attrs_ A list of strings, which must be the names of attributes on the object.
                  These attributes will be exposed and readable and possibly writeable from other COM hosts.
     _readonly_attrs_ -- A list of strings, which must also appear in _public_attrs.  These
                  attributes will be readable, but not writable, by other COM hosts.
     _value_ -- A method that will be called if the COM host requests the "default" method
                  (ie, calls Invoke with dispid==DISPID_VALUE)
     _NewEnum -- A method that will be called if the COM host requests an enumerator on the
                  object (ie, calls Invoke with dispid==DISPID_NEWENUM.)
                  It is the responsibility of the method to ensure the returned
                  object conforms to the required Enum interface.

    _typelib_guid_ -- The GUID of the typelibrary with interface definitions we use.
    _typelib_version_ -- A tuple of (major, minor) with a default of 1,1
    _typelib_lcid_ -- The LCID of the typelib, default = LOCALE_USER_DEFAULT

     _Evaluate -- Dunno what this means, except the host has called Invoke with dispid==DISPID_EVALUATE!
                  See the COM documentation for details.
  """

Up until now all the Python COM gateway classes on this blog have used late binding with the public methods listed in _public_methods_ . The doc string says that for early binding this will not be required but I will still use _public_methods_ to tell which methods to place in the type library. So I retain _public_methods_ (contrary to the documentation).

Official sample, pippo.py

The official code sample given to us by the great Mark Hammond (eternal thanks). The code sample consists of an IDL file, pippo.idl and a Python script implementing the COM server, pippo_server.py. Here is the pippo class

class CPippo:
    #
    # COM declarations    
    #
    _reg_clsid_ = "{05AC1CCE-3F9B-4d9a-B0B5-DFE8BE45AFA8}"
    _reg_desc_ = "Pippo Python test object"
    _reg_progid_ = "Python.Test.Pippo"
    #_reg_clsctx_ = pythoncom.CLSCTX_LOCAL_SERVER    
    ###
    ### Link to typelib
    _typelib_guid_ = '{41059C57-975F-4B36-8FF3-C5117426647A}'
    _typelib_version_ = 1, 0
    _com_interfaces_ = ['IPippo']

    def __init__(self):
        self.MyProp1 = 10

    def Method1(self):
        return wrap(CPippo())

    def Method2(self, in1, inout1):
        return in1, inout1 * 2

And the given idl is thus

 [
  object,
  uuid(F1A3CC2E-4B2A-4A81-992D-67862076949B),
  dual,
  helpstring("IPippo Interface"),
  pointer_default(unique)
 ]
 interface IPippo : IDispatch
 {  
  [id(1), helpstring("method Method1")] HRESULT Method1([out, retval] IPippo **val);
  [propget, id(2), helpstring("property MyProp1")] HRESULT MyProp1([out, retval] long *pVal);
  [id(3), helpstring("method Method2")] HRESULT Method2([in] long in1, [in, out] long *inout1,
                                                        [out, retval] long *val);
 };

But as I said above, as given one would have to maintain the class and the IDL file in synchronization which is a little painful. So now I can add a little value here and give some code which will read a Python class and then write and compile an IDL file into a type library that is in sync which the original Python class.

My test class, FooBar (housed in AComGatewayClass.py)

So here is my test class call FooBar. I have added some type annotations to demonstrate these being defined in the type library. I have placed this into a script file called AComGatewayClass.py. To import one would write from AComGatewayClass import FooBar.

We still have a _public_methods_ attribute even though that is more for late-binding; I use it to determine which methods to write to the type library.

Also, I have invented a new attribute called _reg_itfid_ which is use to snap (fix) the guid of the interface, so don't expect official documentation for that!

class FooBar(object):
    
    _typelib_guid_ = "{92F288D0-4863-4030-A4EE-36DE63DB7664}"
    _typelib_version_ = 1,0
    _typelib_lcid_ = 0

    _reg_clsid_ = "{8B994B6B-0865-4D48-8A62-2EB97C291BDA}"
    _reg_itfid_ = "{B8FFDEFA-3EFB-4725-8CDD-1F6A9E35DD7C}"  ### I have invented this to snap the interface's guid the type library
    
    _reg_progid_ = 'MyPythonProject2.FooBar'
    _com_interfaces_ = ["_FooBar"]

    _reg_policy_spec_ = "DesignatedWrapPolicy" ### not strictly required as is already the default so key driver of functionality

    _public_attrs_ = ['MyProp1']
    _public_methods_ = ['Sum','NoArgs','Baz','Benjy']

    def __init__(self):
        self.MyProp1 = 10

    def Sum(self,a:float, b:float)->float:
        return a+b

    def NoArgs(self)  :
        pass

    def Baz(self,someString) -> str :
        someBoolean:bool=True
        if someBoolean:
            return "Hello " + someString 
        else:
            return "Goodbye " + someString 

    def Benjy(self,someInt:int, someDouble:float, untyped) -> int:
        pass

def RegisterThis():
    print("Registering COM servers...")
    import win32com.server.register
    win32com.server.register.UseCommandLine(FooBar)

if __name__ == '__main__':
    RegisterThis()

IdlWriter.py

So here is the code that will take a Python class and create a type library for it. It requires the MIDL launcher I gave in the previous post and I saved to a script file MidlLauncherHelper.py . The code has a little extra logic to examine an argument type and give the correct type in the IDL which carries through to the type library. There isn't a huge amount to see, it is all string concatenation to be honest.

import pythoncom
from MidlLauncherHelper import MidlLauncher


class IdlWriter(object):
    def PythonTypeToIDLType(self,annotations,argName:str) -> str:
        try:
            if argName in annotations:
                pythonargtype = annotations[argName]
                key2 = pythonargtype.__name__
                return {
                    'bool': "VARIANT_BOOL",
                    'str':"BSTR*",
                    'float':"double",
                    'int':"long",
                    '':"VARIANT*",
                    }[key2]
            else:
                return "VARIANT*"

        except Exception as e:
            print("Error: " + str(e) + "\n")


    def IdlFullFilename(self,idlFilename:str)->str:
        import os
        try:
            this_dir = os.path.dirname(__file__)
            return os.path.abspath(os.path.join(this_dir, idlFilename))
        except Exception as e:
            print("Error: " + str(e) + "\n")

    def Main(self,library:str,typelib_guid,typelib_version,coclasses,idlFilename:str):
        try:
            IdlFullFilename = self.IdlFullFilename(idlFilename)
            idlSrc = self.InspectMyClass(library,typelib_guid,typelib_version,coclasses)
            with open(IdlFullFilename, "w+") as f:
                f.write(idlSrc)
            MidlLauncher.CompileTypelib(IdlFullFilename)
        except Exception as e:
            print("Error: " + str(e) + "\n")

    def InspectMyClass(self,library:str,typelib_guid,typelib_version,coclasses):
        import inspect
        import uuid
        try:

            idl = "// Generated .IDL file (by Python code)\n//\n// typelib filename: FooBar.tlb\n"
            idl = idl + "import \"oaidl.idl\";\nimport \"ocidl.idl\";\n"
            idl = idl + "import \"unknwn.idl\";\n"
            idl = idl + "[\n  uuid(" + typelib_guid[1:-1] + "),\n  version(" + str(typelib_version[0]) + "." + str(typelib_version[1]) + ")\n]\n"
            idl = idl + "library " + library + "\n{\n"
            idl = idl + "    // TLib :     // TLib : OLE Automation : {00020430-0000-0000-C000-000000000046}\n"
            idl = idl + "    importlib(\"stdole32.tlb\");\n"
            idl = idl + "    importlib(\"stdole2.tlb\");\n\n"
            idl = idl + "    importlib(\"stdole2.tlb\");\n\n"
            
            idl = idl + "    // Forward declare all types defined in this typelib\n"

            for coclass in coclasses:
                idl = idl + "    interface " + "_" + coclass.__name__ + ";\n"

            idl = idl + "\n"

            for coclass in coclasses:
                ### rewritten to mimic pippo.idl in the win32com\test directory
                idl = idl + "    [\n"
                idl = idl + "      object,\n"
                idl = idl + "      uuid(" + coclass._reg_itfid_[1:-1] + "),\n" 
                idl = idl + "      dual,\n"
                idl = idl + "      helpstring(\"test\"),\n"
                idl = idl + "      pointer_default(unique)\n"
                idl = idl + "    ]\n"
                
                
                idl = idl + "    interface " + "_" + coclass.__name__ + " : IDispatch {\n"

                method_list2 = inspect.getmembers(coclass, inspect.isfunction)
                dispid = 1  #start from 1 as zero equates to default member
                for meth in method_list2 :
                    if meth[0] in coclass._public_methods_:
                        idl = idl + "        [id(" + str(dispid) + ")]\n"
                        idl = idl + "        HRESULT " + meth[0] + "(\n"
                        fullArgSpec = inspect.getfullargspec(meth[1])
                        argc = len(fullArgSpec.args)
                        for argIdx in range(1, argc):
                            arg = fullArgSpec.args[argIdx]
                            argType = self.PythonTypeToIDLType(fullArgSpec.annotations,arg)
                            idl = idl + "        \t\t[in] " + argType + " " + arg + ",\n"

                        argType = self.PythonTypeToIDLType(fullArgSpec.annotations,"return")

                        idl = idl + "        \t\t[out, retval] " + argType + "* retval );\n"

                    dispid = dispid + 1

                if hasattr(coclass,'_public_attrs_'):
                    for attr in coclass._public_attrs_:
                        idl = idl + "        [id(" + str(dispid) + "), propget]\n"
                        idl = idl + "        HRESULT " + attr + "([out, retval] VARIANT *pVal);\n"

                        idl = idl + "        [id(" + str(dispid) + "), propput]\n"
                        idl = idl + "        HRESULT " + attr + "([in] VARIANT rhs);\n"

                idl = idl + "\n    };\n\n"
                idl = idl + "    [\n      uuid(" + coclass._reg_clsid_[1:-1] + "),\n      version(1.0)\n    ]\n"
                idl = idl + "    coclass " + coclass.__name__ + "{\n"
                idl = idl + "        [default] interface " + "_" + coclass.__name__ + ";\n"
                idl = idl + "    };\n"
            idl = idl + "};\n"

            print(idl)
            return idl

        except Exception as e:
            print("Error: " + str(e) + "\n")

if __name__ == '__main__':
    from AComGatewayClass import FooBar
    idl = IdlWriter()
    idl.Main("MyPythonProject2",FooBar._typelib_guid_,FooBar._typelib_version_,[FooBar],"MyPythonProject2.idl")
    
    print("End of execution")

By the way, you can pass in more than one class, it is written to take a list of classes.

VBA Calling Code

So now we can New a Python class in VBA thus ...

Sub TestEarlyBound()
    On Error GoTo ErrHandler
    
    Dim obj As MyPythonProject2.FooBar
    Set obj = New MyPythonProject2.FooBar
    
    Debug.Print obj.Sum(1, -2)
    obj.MyProp1 = 256
    
    Stop  ' take a moment to admire the property MyProp1 in the Locals window, this would NOT appear without a type library
    Exit Sub
ErrHandler:
    Stop
    
End Sub

Enjoy!

Other Links

Python MIDL Launcher

If you ever need worked with COM type libraries then you will have no doubt bumped in MIDL which is the Microsoft Interface Definition Language compiler. This is a command line tool but if you want to incorporate it into part of your build process you are going to need to shell or launch the process. I have given a C# launcher Midl here. Also I have given a windows batch file launcher here. Below I give a Python equivalent Midl launcher.

I find this code necessary because I need to alter the environment variables PATH and INCLUDE in order to get MIDL to work. The code below also goes on to load the newly create type library and then register it as this is the majority use case (feel free to omit if not appropriate).

import pythoncom

class MidlLauncher(object):
    @staticmethod
    def CompileTypelib(idlFullFileName):
        from distutils.dep_util import newer
        import os
        import subprocess
        try:
            tlb = os.path.splitext(idlFullFileName)[0] + '.tlb'
            if os.path.isfile(idlFullFileName): 

                if newer(idlFullFileName, tlb):

                    import subprocess, os
                    midl_env = os.environ.copy()
                    
                    midl_env["PATH"] = 'C:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\VC\\bin\\;' + midl_env["PATH"]
                    mustInclude = ("C:\\Program Files (x86)\\Windows Kits\\10\\Include\\10.0.16299.0\\um\\;" + 
                                          "C:\\Program Files (x86)\\Windows Kits\\10\\Include\\10.0.16299.0\\shared\\;" +
                                          "C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319;" )
                    if "INCLUDE" in midl_env:
                        midl_env["INCLUDE"] = mustInclude + midl_env["INCLUDE"]
                    else:
                        midl_env["INCLUDE"] = mustInclude 

                    midl = subprocess.Popen(["C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.16299.0\\x86\\midl.exe", idlFullFileName, "/tlb" , tlb],env=midl_env)
                    midl.wait()

                    print("Registering %s" % (tlb,))
                    tli = pythoncom.LoadTypeLib(tlb)
                    pythoncom.RegisterTypeLib(tli,tlb)
        except Exception as e:
            print("Error: " + str(e) + "\n")

C++ code to debug/investigate COM class creation problems

With technology sometimes debugging and diagnostics are required, COM is no exception. Presently, I have been debugging a Python COM class that was not instantiating in VBA using the New keyword. That code works now, so watch out for a post on that soon. The following code was to be an appendix to that post but is useful in its own right so I am depositing it here.

Low level C++ Code to instantiate a COM Component

This code is to probe error messages of a COM component instantiation. VBA gives some error codes which are not necessarily useful. In VBA, when an object is created with the New keyword, 'under-the-hood' a call to CoCreateInstance is made, which in turn is made up of calls to CoGetClassObject to get a class factory and then a call to CreateInstance is called on the class factory. These steps which are implicit to VBA are given explicitly below in C++ so one can step through and better diagnose any errors.

In you want to use this code you will no doubt have to change CLSID_FooBar and IID_FooBar for your specific case.

Problems that can be examined with this technique include but are not limited to (a) registration issues, (b) path problems, (c) 32bit/64 bit mismatch problems.

The code is for a C++ console application.


#include <iostream>
#include "objbase.h"
#include <combaseapi.h>
#include <assert.h>

int main()
{
 ::CoInitialize(0);
 HRESULT hr = S_OK;

 GUID CLSID_FooBar;
 CLSIDFromString(L"{25F9C67B-8DBB-4787-AA84-D3D667ED0457}", &CLSID_FooBar);

 GUID IID_FooBar;
 CLSIDFromString(L"{B8FFDEFA-3EFB-4725-8CDD-1F6A9E35DD7C}", &IID_FooBar);

 GUID IID_IUnknown;
 CLSIDFromString(L"{00000000-0000-0000-c000-000000000046", &IID_IUnknown);

 GUID IID_IDispatch;
 CLSIDFromString(L"{00020400-0000-0000-c000-000000000046", &IID_IDispatch);

 GUID IID_IClassFactory;
 CLSIDFromString(L"{00000001-0000-0000-c000-000000000046", &IID_IClassFactory);

 IClassFactory *pFactoryFooBar;
 // CLSCTX_INPROC_SERVER |  CLSCTX_LOCAL_SERVER

 { // Test 1 Create the FooBar class via class factory requesting IUnknown 
  hr = ::CoGetClassObject(CLSID_FooBar, CLSCTX_INPROC_SERVER, NULL, IID_IClassFactory, reinterpret_cast<void**>(&pFactoryFooBar));
  assert(S_OK == hr);

  IUnknown *pUnkFooBar;
  hr = pFactoryFooBar->CreateInstance(NULL, IID_IUnknown, reinterpret_cast<void**>(&pUnkFooBar));
  assert(S_OK == hr);

  IUnknown *pFooBar;
  hr = pUnkFooBar->QueryInterface(IID_FooBar, reinterpret_cast<void**>(&pUnkFooBar));
  assert(S_OK == hr);

 }

 IDispatch *pDispFooBar;
 hr = CoCreateInstance(CLSID_FooBar, NULL, CLSCTX_INPROC_SERVER, IID_IDispatch,
  reinterpret_cast<void**>(&pDispFooBar));
 assert(S_OK == hr);

 // get disp id
 DISPID id = -1; //default
 LPOLESTR string = const_cast <LPOLESTR>(L"Sum");
 hr = pDispFooBar->GetIDsOfNames(IID_NULL, &string, DISPATCH_METHOD, LOCALE_USER_DEFAULT, &id);
 assert(S_OK == hr);

 UINT ctinfo = -1;
 hr = pDispFooBar->GetTypeInfoCount(&ctinfo);
 assert(S_OK == hr);

 ::CoUninitialize();

 
}

P.S. It is possible to drill down even further because many COM servers are implemented as DLLs, we could write code to load the DLL into memory and then manually get the entry point DllGetClassFactory and call into it to get the class factory manually. One for another day perhaps because at the moment I am working with Python which does not use Dlls in that sense (at least I don't think so)

Wednesday 7 August 2019

VBA - Python - Type Library Finder - use reflection to find a late bound object's containing library

I love finding rare stuff that works in Excel VBA, this blog is full of it! I love finding new useful type libraries of classes which drive functionality I didn't know was possible for a VBA developer. They turn up infrequently on Stack Overflow and other forum websites. Often the sample code creates an object using late binding and not with the type library. When I discover one of these I instantly want to find the containing type library to see what else is in that library. Such an investigation requires

  1. Looking up the ProgID (the text string passed to CreateObject()) to get a CLSID
  2. Looking up the CLSID to get the LIBID, the type library's guid
  3. Looking up the LIBID to get the type library's full name

So for example an interesting component is created with CreateObject("WIA.ImageFile") that has a clsid of {A2E6DDA0-06EF-4df3-B7BD-5AA224BB06E8}, and type library guid of {94A0E92D-43C0-494E-AC29-FD45948A5221} which has the full name of Microsoft Windows Image Acquisition Library v2.0.

Combing though the registry is a pain so fortunately we can write some Python code and use the reflection interfaces, ITypeInfo and ITypeLib which are usually restricted to C++ developers but which Python developers have access to and in turn we can expose to Excel VBA code.

Below is a Python COM Gateway class, I have supplied many examples on this blog so the procedure should be familiar (to regular readers at least). But to recap you need to run the following script with adminstrator privileges from a command console. Once registered, the class becomes creatable from VBA using CreateObject.

class WhichTypeLibrary(object):
    _reg_clsid_ = "{521B0A5A-4359-4874-AA9E-8F99DB35F4A6}"
    _reg_progid_ = 'PythonInVBA.WhichTypeLibrary'
    _public_methods_ = ['ReportTypeLibrary'] 

    def ReportTypeLibrary(self, o):
        try:
            pt = str(type(o))
            if pt == "<class 'win32com.client.CDispatch'>":
                ti = o._lazydata_[0]
            elif pt == "<class 'PyIDispatch'>" :
                ti = o.GetTypeInfo()
            else:
                return "Error, cannot get ITypeInfo interface."

            typelib = ti.GetContainingTypeLib()[0]

            return (typelib.GetDocumentation(-1)[0] + ":\t\t" +  typelib.GetDocumentation(-1)[1] )

        except Exception as e:
            return  "Error: " + str(e) + "\n"

def RegisterThis():
    print("Registering COM servers...")
    import win32com.server.register
    win32com.server.register.UseCommandLine(WhichTypeLibrary)


if __name__ == '__main__':
    RegisterThis()
    print("End of execution")

VBA Client Code

So the calling VBA code is shown below. It shows passing a Workbook object to the library finder which correctly reports it as belonging the Excel type library (that's obvious but it proves the logic is working). A second object is created which is far rarer (certainly I'd not heard of it) and the Python code correctly reports the library prefix, WIA, and the library description, 'Microsoft Windows Image Acquisition Library v2.0', which can then be found in the Tools References dialog box.

Option Explicit

Sub Test()
    On Error GoTo ErrHandler
    Dim oLibFinder As Object
    Set oLibFinder = CreateObject("PythonInVBA.WhichTypeLibrary")
    
    '* a simple example, find the containing type library of a Workbook, should print "Excel:      Microsoft Excel 15.0 Object Library"
    Debug.Print oLibFinder.ReportTypeLibrary(ThisWorkbook)
    
    '* a rarer example, should print "WIA:        Microsoft Windows Image Acquisition Library v2.0"
    Dim obj As Object
    Set obj = CreateObject("WIA.ImageFile")
    
    Debug.Print oLibFinder.ReportTypeLibrary(obj)
    
SingleExit:
    Exit Sub
ErrHandler:
    Debug.Print Err.Description
    Stop
    'Resume
    
End Sub

So the above program prints...

Excel:      Microsoft Excel 15.0 Object Library
WIA:        Microsoft Windows Image Acquisition Library v2.0

Factoid - ProgIDs have limit of 39 characters

What's Next, can the above Python class report on itself? No (or at least not yet).

If you are a little cheeky and wondered what would happen if you pass the library finder class itself to see what it said you would get an error.

Error: (-2147352567, 'Exception occurred.', None, None)

This is because the standard Python COM Gateway class carries no type information by default. It is capable of shipping type information is you associate it with a type library but then the type library needs to agree with the Python source. We can write some code to read a Python class and generate a type library and that will be the subject of the next post.