Python - SVG - Extract and Parse Path Data from d attribute

Introduction

SVG draw shapes using a path language with commands such as moveto x1,y1; lineto x2,y2; lineto x3,y3; lineto x4,y4 then closepath. All of this is packed into a SVG Path element's d attribute. Python has a library, svg.path to parse these commands.

Background

Ok, so previous post I gave VBA code to extract some shapes from a SVG file converted from a PDF file (in the name of extracting the underlying data point) but whilst VBA has an Xml library it does not have a library to parse the d attribute. So we'll switch into Python. Besides, its Python month on this blog and so I'm meant to be reviewing and introducing useful and interesting Python libraries.

Demonstration of parsing d attribute with svg.path

So install the code (from and admin rights command console) with...

pip install svg.path

Run Python.exe to get into Python environment and enter the following statements (responses are also shown, and indented)

C:\Users\Simon>python
Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from svg.path import Path,Line,Arc
>>> from svg.path import parse_path
>>> parse_path('m 241.666,133.557 h 2.364 v -25.886 h -2.364 z')
Path(Line(start=(241.666+133.557j), end=(244.03+133.557j)), 
     Line(start=(244.03+133.557j), end=(244.03+107.671j)), 
     Line(start=(244.03+107.671j), end=(241.666+107.671j)), 
     Line(start=(241.666+107.671j), end=(241.666+133.557j)), closed=True)
>>>

So we can see the path being parsed into a sequence of Line objects each with their own start and end co-ordinate pairs.

Python program to process paths

So now we can write some code to extract the height of the rectangle (which represents the underlying data point).

from lxml import etree
from svg.path import Path,Line
from svg.path import parse_path

sFileName = 'C:/Users/Simon/Downloads/pdf_skunkworks/inflation-report-may-2018-page6.svg'

tree=etree.parse(sFileName)

xpath = r"//svg:path[@style='fill:#19518b;fill-opacity:1;fill-rule:nonzero;stroke:none']"

#print (xpath)
bluePaths = tree.xpath(xpath,namespaces={   'svg': "http://www.w3.org/2000/svg"  })

for bluePath in bluePaths:
    parsed=parse_path (bluePath.attrib['d'])
    secondLine = parsed[1]
    print (secondLine.end.imag - secondLine.start.imag) #outputs the height

Python - Split PDF into single page SVG files by shelling Inkscape

So if you have been the last few posts you know I want to get data out of a PDF file (in this case a Bank of England Inflation Report). The python library PyPDF2 is good and will split a pdf into separate pages. No doubt, PyPDF2 will do a ton of other stuff to manipulate pdf files. However, I find the pdf file format is difficult to understand which limits how much I want to use a pdf python library. Instead, I have found that Inkscape converts PDFs to SVGs and that we can automate this on the command line.

This means I am in a position to give the latest version of a program which breaks up a PDF file into separate SVG files for each page. (You'll need Inkscape installed). Here it is...

# with thanks to user26294 at Stack Overflow
# https://stackoverflow.com/questions/490195/split-a-multi-page-pdf-file-into-multiple-pdf-files-with-python#answer-490203

from PyPDF2 import PdfFileWriter, PdfFileReader

def DecryptPdf(pdfFileReader,password):
    if pdfFileReader.isEncrypted:
        try:
            pdfFileReader.decrypt(password)
            print ('File decrypted')
        except Exception as e:
            print ('File decryption failed:' + str(e))
    else:
        print ('File not enrypted')

def SuffixFilename(fileName, suffix):
    import os.path
    filePath = os.path.split(fileName)
    
    filePath2 = filePath[1].split('.')
    return  filePath[0] + '\' +filePath2[0] + suffix + '.' + filePath2[1]


def OutputPage(pdfFileNameSrc,pdfFileNamePage, pageNum):
    pdfFileSrc = open(pdfFileNameSrc, "rb")
    pdfFileReaderSrc = PdfFileReader(pdfFileSrc)
    DecryptPdf(pdfFileReaderSrc,'')
    
    pageOutput = PdfFileWriter()
    pageOutput.addPage(pdfFileReaderSrc.getPage(pageNum))

    with open(pdfFileNamePage, "wb") as outputStream:
        pageOutput.write(outputStream)
        print('written page%s' % pageNum)
        outputStream.close 
    pdfFileSrc.close #tidy up
    

def InkscapePdfToSvg(pdfFileName):
    import subprocess 
    svgFileName=pdfFileName.replace(".pdf",".svg")
    
    completed = subprocess.run(['c:/Progra~1/Inkscape/Inkscape.exe',
            '-z', 
            '-f', pdfFileName , 
            '-l', svgFileName])

    return svgFileName


if __name__ == "__main__": 

    pdfFileNameInflation = "C:\Users\Simon\Downloads\pdf_skunkworks\inflation-report-may-2018.pdf"
    pdfFileInflation = open(pdfFileNameInflation, "rb")

    pdfFileReaderInflation = PdfFileReader(pdfFileInflation)

    DecryptPdf(pdfFileReaderInflation,'')
    pageCount = pdfFileReaderInflation.numPages

    for i in range(pageCount):
        pdfFileNamePage=SuffixFilename(pdfFileNameInflation,"-page%s" % i)
        
        OutputPage(pdfFileNameInflation,pdfFileNamePage,i)
        print (InkscapePdfToSvg(pdfFileNamePage))

Python - Use Process Monitor to diagnose Subprocess.run

Python can shell out and run other executables but sometimes it fails; and it is not always apparent why. In this post I show how Process Monitor (and some help from StackOverflowers) helped me to get the right syntax.

Inkscape converts PDFs to SVG

As part as my ongoing struggles with pdf files I was looking for a better way to get content from PDF files. I had discovered that Inkscape allows the conversion of a single page PDF into an SVG file, (there is a set of instructions here).

I'd prefer SVG files as they are XML and I know how to traverse and navigate them. Indeed, I have blogged about using VBA XML to create an SVG file.

Following the instructions given I get a single dialog box (shown below) and then success. This is great but I wanted to automate the process.

Inkscape's Command Line Options

So I wanted a command line way of getting Inkscape to do the pdf to svg conversion. I found Inkscape installed at C:\PROGRA~1\Inkscape> and queried it for command line options

C:PROGRA~1Inkscape>inkscape --help
Usage: inkscape [OPTIONS...] [FILE...]

Available options:

  -z, --without-gui                          Do not use X server (only process
                                             files from console)
  -f, --file=FILENAME                        Open specified document(s)
                                             (option string may be excluded)
...
  -l, --export-plain-svg=FILENAME            Export document to plain SVG file
                                             (no sodipodi or inkscape
                                             namespaces)
...
Help options:
  -?, --help                                 Show this help message
      --usage                                Display brief usage message

So all the required options are there and we can construct the right command line; the following worked and exported a pdf to svg...

c:\progra~1\Inkscape\inkscape -z -f "N:\pdf_skunkworks\inflation-report-may-2018-page0.pdf" -l "N:\pdf_skunkworks\inflation-report-may-2018-page0.svg

Excel VBA code to shell Inkscape to Convert PDF to SVG

As part of investigations I also wrote some VBA code to execute the above command line...

Sub TestShellToInkscape()
    '* Tools->References->Windows Script Host Object Model (IWshRuntimeLibrary)
    Dim sCmd As String
    sCmd = "c:\progra~1\Inkscape\inkscape -z -f ""N:\pdf_skunkworks\inflation-report-may-2018-page0.pdf"" -l ""N:\pdf_skunkworks\inflation-report-may-2018-page0.svg"""
    Debug.Print sCmd
    
    Dim oWshShell As IWshRuntimeLibrary.WshShell
    Set oWshShell = New IWshRuntimeLibrary.WshShell
    
    Dim lProc As Long
    lProc = oWshShell.Run(sCmd, 0, True)
    
End Sub

Python code to shell Inkscape to Convert PDF to SVG

And here is the final Python code which also shells out to Inkscape and converts pdf to svg.

import subprocess 
completed = subprocess.run(['c:/Progra~1/Inkscape/Inkscape.exe',
        '-z', 
        '-f', r'N:/pdf_skunkworks/inflation-report-may-2018-page0.pdf' , 
        '-l', r'N:/pdf_skunkworks/inflation-report-may-2018-page0.svg'])
print ("stderr:" + str(completed.stderr))
print ("stdout:" + str(completed.stdout))

So this turned out to be the right answer, specifically passing each argument separately (whereas VBA passes whole string).

Diagnosing Subprocess.run

The correct Python code is given above but this blog post is more about the journey to get there.

My early attempts did not work and I resorted to StackOverflow to get help. JacobIRR put me on the right track saying that I could use forward slashes and Python could work out when to swap them for backslashes. I took on board this suggestion but it still didn't quite work.

Another StackOverflower asked if I knew that Inkscape was actually running. I thought this unlikely but sought to provide a screenshot that it was indeed running. Task manager was insufficient for this. So instead I turned to Process Monitor to grab the screenshot.

Using Process Monitor to diagnose Subprocess.run

Taking a process shell tree was quite tricky; it required running the python script and then quick as a flash switching (ALT+TAB) to Process Monitor and then pressing Ctrl+T. Here is the first snap which shows a malfunctioning Python program with its arguments being passed to Inkscape with overzealous slashes!

This second snap is one of correctly working code (see above). You can see how the triple slashes have gone, thankfully. Also not using double quotes helped.

Final Thoughts

So, if you are having difficulty with Subprocess.run do please consider using Process Monitor to help diagnose what actually gets passed as arguments.

PDF - Gripes with PDF file format

So I wanted to extract graphics from a Bank of England report but it turned out to be very involved. I began to get drawn into the PDF file format. Here are some notes on its difficulty.

Firstly, a pdf is massive and as a first step I recommend breaking into single page pdf files. I have written a blog post here which shows how.

Secondly, we need to say that pdf files can be encrypted, the Bank of England report is but with a password of an empty string "" which is a little tedious. Luckily the Python library PyPDF2 can decrypt a file with the following code

def DecryptPdf(pdfFileReader,password):
    if pdfFileReader.isEncrypted:
        try:
            pdfFileReader.decrypt(password)
            print ('File decrypted')
        except Exception as e:
            print ('File decryption failed:' + str(e))
    else:
        print ('File not encrypted')

Thirdly, we have to deal with compression. So even after decrypting the next problem is compression, certain portions of a pdf document will be compressed and so read as gibberish in a text editor. Because of this I had great difficulty scratching the surface of the pdf file format.

What is needed is a good program that will help you explore the structure and thankfully I found PDFXplorer. Here is a screenshot showing a single page of the report being explored, it shows a compressed stream in decompressed view. Also it has a Save stream to disk button which allows the stream to be exported and then viewable in a text editor.

Fourthly, the pdf file format is unlike any xml, json or other standard file. So after using PDFXplorer to save a stream to disk and examining it in a text editor I found a key section...

/Figure <</MCID 88 >>BDC 
/PlacedGraphic /MC0 BDC 
EMC 
q
39.686 83.091 223.603 129.731 re
W n
0 0 0 1 K
0.5 w 4 M 
/GS0 gs
252.534 204.865 -212.599 -113.386 re
S
Q
0.96 0.53 0.05 0.27 k
/GS0 gs
241.666 133.557 2.364 -25.886 re
f
234.472 126.822 2.416 -19.152 re
f
227.335 128.493 2.416 -20.823 re

So to interpret this language one needs to reference Appendix A of this 756 page document . Here is a table of some of the operators signified by the letters

BDC=Begin marked-content	EMC=End marked-content	q=Save graphics state	re=Append rectangle to path	W=Set clipping...
n=End path without filling...	K=Set CMYK color for stroking ops	w=Set line width	M=Set miter limit	gs=Set ... graphics state...
S=Stroke path	Q=Restore graphics state	k=Set CMYK color for nonstroking ops	f=Fill path using nonzero winding	/=start of a name

So the line highlighted in blue 0.96 0.53 0.05 0.27 k caught my eye as I was looking for the path data of some blue rectangles in the following graph. The k operator sets the colour using a CMYK (Cyan Magenta Yellow Key) color code, to convert to RGB see this web site. The lines that follow on from the CMYK line draw rectangles, they are part of this graphic taken from page 6 of a Bank of England report. The first blue rectangle is shown selected with double arrow handles...

So, in my opinion the pdf file format is difficult to work with. I cannot imagine how to begin parsing this document. it is true that there will probably be Python libraries to help but one still needs to browse the document and figure out what are the right questions to ask any such Python library.

In a future post, I'll show how converting the page to an SVG file faciliates navigation, as a preview taster I can show you that the selected blue rectangle gets converted into the following SVG/XML which whilst it maybe verbose is clearly selectable with some XPath...

<path
    id="path5759"
    style="fill:#19518b;fill-opacity:1;fill-rule:nonzero;stroke:none"
    d="m 241.666,133.557 h 2.364 v -25.886 h -2.364 z" />

Final Thoughts

I didn't much like my dive into PDF file formats and I'd like not to revisit them again any time soon. But whether they can be dispensed with depends on one goals and the alternative technologies available.

Python - PDF - Split large file into single pages

So last post I showed how to extract text from a PDF using Python and the PyPDF2 library. My example was a Bank of England report. I wanted next to extract the graphics from the report. This turned out to be non-trivial and requires a number of steps. The first step I'd recommend is to break a large document into single page documents.

Python PDF Splitter

So for reference the test document is at BofE Inflation Report May 2018.

Luckily some code existed on Stack Overflow to break up the pages...

# with thanks to user26294 at Stack Overflow
# https://stackoverflow.com/questions/490195/split-a-multi-page-pdf-file-into-multiple-pdf-files-with-python#answer-490203

from PyPDF2 import PdfFileWriter, PdfFileReader

def DecryptPdf(pdfFileReader,password):
    if pdfFileReader.isEncrypted:
        try:
            pdfFileReader.decrypt(password)
            print ('File decrypted')
        except Exception as e:
            print ('File decryption failed:' + str(e))
    else:
        print ('File not enrypted')

def SuffixFilename(fileName, suffix):
    import os.path
    filePath = os.path.split(fileName)
    
    filePath2 = filePath[1].split('.')
    return  filePath[0] + '\' +filePath2[0] + suffix + '.' + filePath2[1]


def OutputPage(pdfFileNameSrc,pdfFileNamePage, pageNum):
    #print (pdfFileNameSrc)
    #print (pdfFileNamePage)
    pdfFileSrc = open(pdfFileNameSrc, "rb")
    pdfFileReaderSrc = PdfFileReader(pdfFileSrc)
    DecryptPdf(pdfFileReaderSrc,'')
    
    pageOutput = PdfFileWriter()
    pageOutput.addPage(pdfFileReaderSrc.getPage(pageNum))

    with open(pdfFileNamePage, "wb") as outputStream:
        pageOutput.write(outputStream)
        print('written page%s' % pageNum)
    pdfFileSrc.close #tidy up


if __name__ == "__main__": 

    pdfFileNameInflation = "n:\\pdf_skunkworks\\inflation-report-may-2018.pdf"
    pdfFileInflation = open(pdfFileNameInflation, "rb")

    pdfFileReaderInflation = PdfFileReader(pdfFileInflation)

    DecryptPdf(pdfFileReaderInflation,'')

    for i in range(pdfFileReaderInflation.numPages):
        pdfFileNamePage=SuffixFilename(pdfFileNameInflation,"-page%s" % i)
        
        OutputPage(pdfFileNameInflation,pdfFileNamePage,i)

I have triaged the original code given in a StackOverflow answer because it would not behave in a loop. this must have been some file handle tidfyup issue. My (heavy-handed) solution was to re-initialise the PdfFileReader class in OutputPage() for each iteration. I'm sure a better solution exists and if you know better then feel free to comment below.

Now I have single page pdfs, I can move on ...