A Tried and Tested True Believer

I was first introduced to the concept of testing by Killer Web Development. I sort of understood it, but at first glance it sounded ridiculous. How can I possibly write an app to test an app that doesn’t exist yet? I thought computer programmers were logical people! And doesn’t this double my work, having to write two apps instead of one? I honestly didn’t fully grasp it. Who has time to write tests? All I have to do is run my app and look at it myself and see if it does what I want..should be simple, right? I want to crank out my code and see results NOW! So, what changed my mind? My work on CsvChamp, that’s what. I found myself tediously opening CSV files, running them through my module and looking at the output. I got weird results I didn’t understand, and my brain seemed to freeze up trying to figure them out. I spent what seemed like hours staring at my screen, trying to untangle the convoluted flight path my data took as it crashed and burned. Somewhere I read something about testing that spoke to me right where I was: wouldn’t it be great if I could have all this work done by….Python? That got my attention. Back to Google…

Thanks to this chapter of Dive Into Python, I now have my very first snippet of “tested first” code. It’s my Python 3 revision of my column alignment function in CsvChamp. Following along with the example, I started off with a blank code file. As I tried to figure out how to write my test, lo and behold — I found myself thinking much more clearly about what I wanted the function to do and what would constitute valid input. I also liked the idea of Python doing my work for me! Unlike before, when I’d run the function again and again on live data and view the results myself to see if they looked OK, I only had to run it through a unit test, with data I already knew was right, and have the test results display a happy green “ok” symbol in my console. After a few go-arounds, here’s the result:

import csv

# Define exceptions
class CsvChampError(Exception): pass
class WidthError(CsvChampError): pass
class AlignmentSpecError(CsvChampError): pass

def StrAlign(text, width, align):
    """Returns text aligned within column according to alignment specified"""

    # Width must be at least 1 more tha len(text) so there will be at least
    # one space between columns.
    if width <= len(text):
        raise WidthError("Invalid width. Must be > length of text.")

    # Padding = total number of blank spaces
    padding = width - len(text)

    if align == "L":
        aligned = text + (padding * " ")

    elif align == "R":
        aligned = (padding * " ") + text

    elif align == "C":
        # Divide padding by 2 and round down to nearest integer
        spaces_before = int(padding/2)

        # Put the rest of the spaces after the text
        spaces_after = padding-spaces_before

        aligned = (spaces_before * " ") + text + (spaces_after * " ")

        raise AlignmentSpecError("invalid alignment spec (must be L, R or C).")

    return aligned

For comparison’s sake, here’s the old version of the same function:

def FormatCsvCol(width, align, cont):
    Returns a column of data with content aligned within an allotted width'
    column = ""
    # The total number of empty spaces in the column.
    spaces = width - len(cont)

    if align == "L":
        # Place all the spaces after the content
        column = cont + (spaces * " ")

    elif align == "R":
        # Place all the spaces before the content.
        column = (spaces * " ") + cont

    elif align == "C":
        # Divide the number of spaces by 2 and round down to the nearest
        # integer.
        # Put the content between this number of spaces, and the rest.
        sp_before = int(spaces / 2) * " "
        sp_after = (spaces - len(sp_before)) * " "
        column = sp_before + cont + sp_after

    return column

Obviously, the big difference is the addition validity checking and informative error messages when invalid input is received. This was a direct result of testing using Dive Into Python’s example, forcing me to look at the big picture. This function could end up anywhere: It could be used to display neat columns of data in a console, in a GUI widget or in a text file. The deeper it gets buried in a complicated app, the more crucial it is to not just know invalid inputs somehow crept into it, but WHY those inputs were invalid. Now I know the function works, without having had to keep feeding it actual CSV data and repeatedly viewing text files myself. Here’s the unit test I came up with:

"""Unit test for CsvChamp3"""

from CsvChamp3 import StrAlign
import unittest

Function StrAlign to align a string within a column of given width, according
to an alignment specification of left, center or right.
class StrValues(unittest.TestCase):
    StrValues = (("example",10,"L","example   "),
                 ("example",10,"C"," example  "),
                 ("example",10,"R","   example"),
                 ("platypus",12,"L","platypus    "),
                 ("boomerang",15,"R","      boomerang"),
                 ("bubba",13,"C","    bubba    "))

    def testValidValues(self):
        """StrAlign should give known results with known values"""
        for text, width, align, text_out in self.StrValues:
            result = StrAlign(text,width,align)
            self.assertEqual(text_out, result)

if __name__ == "__main__":

Of course, I still need to progress to testing invalid input, and functional testing. But this is certainly an encouraging start.

Thank you. Danke. dziękuję. спасибо. धन्यवाद. Merci. Dank u. Gracias.

2014-02-17 MPA countriesAs you all can see, the title of this blog is “Thank you” in the native languages of the top ten countries in which this blog’s viewers live. That’s because I am thankful for every one of you. Really.

This blog has only existed for a month, yet the response I have received from it has been far beyond what I expected in such a short time. I am truly amazed. I’m sure part of this is the fact that I’m an old-timer who still remembers a world without the Internet, so perhaps there’s a sense of childlike wonder in me that still hasn’t died. Actually, I hope it doesn’t.

No thank you would be complete without a huge shout out to the Google+ Python community, to which I owe most, if not all of my viewership. My appreciation of Google+ has grown alongside my Python learning, to the point where the days when I scoffed at it as just another Facebook wannabe are long over. It has been a joy to find instant, international communities of people who share my interests and whose courtesy and sociability seem to be above average when it comes to internet conversations.

Finally, I want to close off this one-month anniversary post by again thanking those who are listed to the right in the ever-growing “Friends, Helpers and Inspiration” section. I wouldn’t have come this far without you.

Thank you for sharing in the adventure.

Leaps and Bounds

Some realizations and decisions I have come to in the last week or so…

It appears most Python programmers work in Linux. Many Python learning resources seem to assume a Linux environment. One of my reasons for adopting Python was the fact that it’s open source, unlike MS Access wherein is all my previous coding experience. On the other hand, I need Windows 7 running natively on my laptop so I can take full advantage of all the functions of my two USB scanners. To save my life, I couldn’t get these to work on Linux, either running natively on my laptop, or with Windows 7 in a Virtual Box VM. I have therefore switched my Python coding to Linux Mint, running in a Virtual Box virtual machine under Windows 7. I have all my code projects in Dropbox, which works great in Linux and Windows 7, and a selected few also on GitHub.

I have come to the conclusion that, going forward, my personal coding projects will be in Python 3. If I run across a learning opportunity that requires Python 2.x (for example, something requiring wxPython), then I will revert back. On that topic, this article by Jeff Knupp alerted me to the necessity of virtual environments.

Oldřich Vetešník became my first collaborator on GitHub and brought me back to the necessity of unit testing and had some helpful suggestions on basic app design, i.e., making the app itself a class. This poor soul heard loudly from Snarkoverflow (oops I mean Stackoverflow) about the undesirability of writing an app first and THEN testing it. Ouch. That was me with my CsvChamp module. Luckily Mr. Vetešník was much kinder. But I do need to get with the program regarding testing, so in keeping with that goal and my need to learn Python 3, I’ve decided to re-do CsvChamp from scratch, in Python 3, starting out with unit testing. That is my current project, and updates will follow!

Next challenge: How do I pull together the need to work in both Python 3 and Python 2, version control with GitHub, coding in Linux. Virtual environments, testing and debugging? I was using Notepad++ and IDLE alternately, and the GitHub Windows app, plus a whole lot of directory-switching in both the Windows and Linux command lines. All of these had their quirks and annoyances. IDLE is decent for debugging, but its code editor doesn’t display line numbers. How annoying is an IDE that tells me what line in my code has an error, but can’t show me the line number? IDLE also has the annoying habit of giving me blue screens of death on Windows XP (yes, there is a situation in which I have to use XP. Don’t ask). Notepad++ is a good editor but doesn’t do debugging. Regarding GitHub, the Windows GitHub app is slow, bloated and feature-sparse. I read an article that suggested many Python developers use Vim, but from the little I know about Vim it appears there would be a steep learning curve with its multitude of keyboard-based commands and all the add-ons I’d need to accomplish what I want.

Enter PyCharm. I did some research on IDE’s and decided to give this one a spin, and so far, I love it. It connects seamlessly to GitHub and incorporates virtualenv in a way that makes it easy to switch between Python 2 and 3 and maintain separate sets of libraries for each, and even lists what’s in each library set in case I forget what I installed and where. I was able to configure its text editor in the same nice colors I enjoyed in Notepad++. It displays line numbers, debugs, and even admonishes me when my coding style strays from PEP 8, among other things. It has both Windows and Linux versions. Perhaps by doing so much for me, it’s short-changing some more learning experiences, but that’s a trade off I’m willing to make to be able to focus more on getting some actual coding done, and done right.

A Pythonic Valentines Day

One of my biggest challenges in growing my Python skills is time.  I have a full-time office job, a wife, and am active in mentoring high school and college students in our church’s youth and college group.  I also try to exercise and stay in shape, as I am no longer, by any stretch of the imagination, a young man. These activities are all vital and important to me, but I often find many days can slip by without me doing a single line of code, and all the cool things I learned a couple of weeks ago start to get fuzzy in my mind.  My tendency is to want to hammer away at a single, huge, complicated goal, such as my Tkinter/database interface, while my everyday Python skills languish, such as finding values in a dictionary or reading from a file. Therefore, I’ve decided that in between progress on gallant quests such as The Silvery Mystery Table or the GUI for the Application of Coolness, I’m going to continue working through the Real Python course, chapter by chapter.  After all, I did spend money on it, the least I can do is use it!  I have already found that it’s a great way to learn and reinforce those everyday skills that can make Python useful for me in a more immediate way.   So, without further ado, here’s my latest silly, yet fun and educational, effort.

Now that February is upon us, the Valentines Day displays at stores are out in full force.  But, for an aspiring Pythonian, what better way to impress his or her significant other than code that generates romantic poetry? I recently completed this assignment for the Real Python course, to test my understanding of lists, just in time to shore up the skills of shaky Shakespearians. There was a word list assigned which I originally hard-coded in as a dictionary, but I took it a step higher and wrote in some code to read word lists from a *.csv file.  That way, I was able to ask my wife Donna to supply the romantic words of her choice in a simple Notepad file, and voilà! – with the press of F5, out came words to woo her by!  Well, not quite…but at least it was a learning and skill-strengthening experience, and I was able to get her interested in my coding progress.  And as of now, we’re still married, so the poems couldn’t have been that bad.

Here’s the code, followed by a sample word list that can be copied and pasted into a *.csv text file.  Enjoy, and Happy Valentines Day!

"""Poetry 3.0
My project from "Real Python" 6.1.
In my humble opinion my code is more efficient than the "official"
This program reads lists of words from a *.csv file and uses them
to create random poems."""

from random import choice
import csv
import os

# A dictionary of word types. Key = word type, values = word list.
# We are creating empty lists in preparation for appending from
# CSV file

word_lists = {"nouns":[],"verbs":[],"adjectives":[],

# The path to the folder where this code and the *.csv file reside.
# Change this as needed.
myPath = "D:\Dropbox\RealPython"

"""Assume the *.csv file is "words.csv." Each row is a set of words.
The first column of each row must be the word type. Example:
with open(os.path.join(myPath, "words.csv"), "rb") as myFile:
    myFileReader = csv.reader(myFile)
    for row in myFileReader:
            if row[0] == "nouns":
                for w in range(1,len(row)):
            elif row[0] == "verbs":
                for w in range(1,len(row)):
            elif row[0] == "adjectives":
                for w in range(1,len(row)):
            elif row[0] == "prepositions":
                for w in range(1,len(row)):
            elif row[0] == "adverbs":
                for w in range(1,len(row)):

        except (IndexError):

def create_list(kind, num):
# type = string = "nouns", "verbs", etc.
# num = number of words of type

    word_list = []

    # Add the first random choice word to the list
    selection = choice(word_lists[kind])

    # Add the additional required words
    for n in range(0,num-1):

        # If the selected word is already in the list,
        # choose another random word.
        while selection in word_list:
            selection = choice(word_lists[kind])

        # Now that we know the selected word isn't already in the list,
        # append it.

    return word_list

def makePoem():
    # Create the differnt types of word lists
    poem_nouns = create_list('nouns',3)
    poem_verbs = create_list('verbs',3)
    poem_adj = create_list('adjectives',3)
    poem_prep = create_list('prepositions',2)
    poem_adv = create_list('adverbs',1)

    # Check if first letter of first adjective is a vowel;
    # if it is, the starting word is "An", else it's "A".
    if "aeiou".find(poem_adj[0][0]) != -1:
        article = "An"
        article = "A"

    poem = "{} {} {}".format(article, poem_adj[0], poem_nouns[0]) + "\n" + "\n"
    poem = poem + "{} {} {} {} {} the {} {}".format(article, poem_adj[0],
                                            poem_nouns[1]) + "\n"
    poem = poem + "{}, the {} {}".format(poem_adv[0],poem_nouns[0],
                                         poem_verbs[1]) + "\n"
    poem = poem + "the {} {} {} a {} {}".format(poem_nouns[1],poem_verbs[2],
                                        poem_prep[1],poem_adj[2],poem_nouns[2]) + "\n"

    return poem

print makePoem()

Sample word list:

adjectives,gracious,ruby red,fastidious,quirky,passionate,toffeebrown,deep,cherished,straight,fierce,shiny,ecstatic,monstrous,golden,elvish

Wrestling Text

One thing I have learned about learning is that it’s often a random, messy process, full of unexpected twists and turns. I set out toward a particular goal, intent on learning every single step required to get to that goal, but then some unexpected obstacle trips me up on my way and I become tempted to give up in frustration. Then, to my surprise and wonder, the obstacle itself becomes a learning experience, and I attain valuable knowledge I wouldn’t have discovered if it had been 100% up to me to find it. This new knowledge may cause me to reconsider my goal and alter it, or even scrap it altogether and set a new one. Those who have followed this blog have seen this process first hand. With that in mind, a funny thing happened on my way to completion of my latest cool app. I found my quest unexpectedly distracted by that noisy, demanding child whose name is Immediate Practical Need.

At my job, every week I have to upload a very precisely formatted text file to our third-party retirement plan administrator. The administrator tracks this data for us to ensure compliance with federal regulations and reporting requirements. The text file includes weekly payroll information, such as employee names, gross pay and retirement plan contribution amounts. Each field must be an exact number of spaces and aligned just right in order to upload correctly.

Currently, I produce this file using Microsoft Access 2010. Access has an option called “Saved Exports” that I use to export the payroll data to a text file. Back when I set this process up I didn’t know much about MS Access’s built-in programming language (Visual Basic for Applications, aka VBA), so all the precise spacing required for the text column is “hard wired” into the report object. Should the plan administrator ever change their file format, I’d have to change the report manually. Another problem is that Access’s “Saved Exports” can’t be edited without some high-end hacking, so if the directory where I need to save the file changes, or I have to move my applications and data to a new PC, I’d have to re-do the whole “Saved Export.” I could just re-do this whole procedure using VBA code, but I decided I’d prefer to use this as an opportunity to learn more Python! Result: “CsvChamp.”

“CsvChamp” is a tool set of Python functions I designed to import, format, display and export data from a comma-separated (CSV) file. At first glance, it’s obvious none of these require Python to accomplish. The same results could be produced using a spreadsheet app. Their power, like everything in Python, lies in their programmability and flexibility. The heart of this tool set is the function GetCsvData. It reads content from a CSV file into a Python dictionary. It adds a unique integer row ID to each row read from the file, and these row IDs become the keys in the dictionary and the rows of data are stored as list object values. Once the data is in a Python dictionary, the sky is the limit, as each and every item can be accessed using code. In this way, code can be written to sort, display, add, delete, summarize and so on. Not only does this have an immediate practical use to me on the job, I can also see this applying to my database management goals. Here’s a summary of what each tool does. See the comments in the code for more details.

CsvFilePicker uses good old Tkinter to prompt the user to select a CSV file.

GetCsvData reads the CSV data into a dictionary, so a line like this:
Doe,John,456 Any St.,Anywhere,TX,45678,(997) 777-7878,john@doeadeer.com

is stored like this:
{1:[‘Doe’,’John’,’456 Any St.’,’Anywhere’,’TX’,’45678′,'(997) 777-7878′,’john@doeadeer.com’]}

GetCsvCols reads the column names from the first line of the CSV file and stores them as a list, so this:
Last,First,Address,City,State,Postal Code,Phone,Email

is stored like this:
[‘Last’,’First’,’Address’,’City’,’State’,’Postal Code’,’Phone’,’Email’]

CreateColSpec examines each column in a dictionary produced by Get CsvData to determine how to align and space the columns for display or export. It adds one space to the maximum with of each column, and defaults to left-alignment of each column.

FormatCsvCol takes a column of data and adds spaces before and/or after it — depending on the specifications read from the dictionary produced by CreateColSpec — so it will align correctly when displayed or output.

Here’s some sample input and output:

Messy, ugly, mean CSV file:

Clean, pretty, friendly output file:

Of course, all of this is 100% flexible. Column specifications can be find-tuned by individual column, so some could be left-aligned and others right-aligned. We don’t have to use the column names in the CSV file, nor do we even have to have the CSV file provide column names at all; we could easily pick our own. We could change the display options so that only selected columns are displayed. The surface has barely been scratched! Feel free to try it yourself. Enjoy!

CSV champ code below. Sorry for the long lines. I haven’t been able to get wrapping to work right yet.

def CsvFilePicker():
    """Prompts user to select a file, using the GUI file dialog. Returns file path.
    # Use csv & the Tkinter graphical file dialog.
    import csv
    import Tkinter, tkFileDialog

    # Asks the user to select a file to load
    root = Tkinter.Tk()
    file_path = tkFileDialog.askopenfilename()

    return file_path

def GetCsvData(file_path):
    """Reads data from a CSV file and returns a dictionary of the file's data, assigning each row a unique integer row ID.
    import csv
    # Copy the contents of the file into a dictionary.
    with open(file_path,"rb") as csvData:

        FileReader = csv.reader(csvData)

        # FileData = the dictionary to hold the data.
        FileData = {}

        # Counts number of rows in file, to use for row IDs.
        row_count = 1
        for row in FileReader:
            row_id = row_count

            # Add the row to the dictionary, assigning the
            # row ID as its key.
            FileData[row_id] = row

            row_count = row_count+1

    return FileData

def GetCsvCols(FileData):
    """Reads column names from the first value of a dictionary created by GetCSVdata.
    col_names = []
    for item in FileData[1]:

    return col_names

def CreateColSpec(FileData):
    col_spec = {}

    #Iterate through each column
    for c in range(0, len(FileData[1])):

        #Set the maximum with to the column name
        max = len(file[1])

        #Check each row in the col
        for r in FileData:

            #If the current row in the column > the
            #current maximum, store it.
            if len(FileData[r]) > max:
                max = len(FileData[r])

        #Specify the width of the column to be 1
        #space more than the width of its longest item.
        col_spec[FileData[1]] = [max+1,"L"]

    return col_spec

def ListCsvCols(col_names):
    """List and count the columns."""
    for item in range(1, len(col_names)+1):
        print str(item)+": " + col_names[item-1]

def FormatCsvCol(width,align,cont):
    """Returns a column of data with content aligned
    according to these specifications:

    width = total number of characters allotted to the column including leading and/or trailing spaces.

    align = "L" (left), "C" {center} or "R" (right).

    cont = content (the actual data to be displayed).


    # The total number of empty spaces in the column.
    spaces = width - len(cont)

    if align == "L":
        # Place all the spaces after the content
        column = cont + (spaces * " ")

    elif align == "R":
        # Place all the spaces before the content.
        column = spaces * " " + cont

    elif align == "C":
        """Divide the number of spaces by 2 and round down to the nearest integer. Put the content between this number of spaces, and the rest.
        sp_before = int(spaces/2) * " "
        sp_after = (spaces - sp_before) * " "
        column = sp_before + cont + sp_after

    return column

def DisplayCsvCols(FileData, col_spec):
    """Uses FormatCsvCol to display the data in FileData.
    Requires col_spec.
     #Get the column names from the 1st row.
    columns = GetCsvCols(FileData)

    #Step through FileData in row ID order.
    for row_id in range(1,len(FileData)+1):
        disp_row = ""

        #Step through each column in a row
        for col_num in range(0,len(columns)):
            disp_row = disp_row + FormatCsvCol(col_spec[columns[col_num]][0],col_spec[columns[col_num]][1],FileData[row_id][col_num])

        print disp_row

def OutputCsv(FileData, output_spec):
    """Uses FormatCsvCol to output the data in FileData to a space-delimited text file. output_spec = a dictionary providing column output specifications.
    import os

    # Edit path and name as needed.
    output_path = "D:\Dropbox\projects\CsvChamp"

    file_name = "output.txt"

	#Get the column names from the 1st row.
    columns = GetCsvCols(FileData)

    with open(os.path.join(output_path,file_name),"w") as OutputFile:

        #Step through FileData in row ID order.
            for row_id in range(1,len(FileData)+1):
                    disp_row = ""

                    #Step through each column in a row
                    for col_num in range(0,len(columns)):
                            disp_row = disp_row + FormatCsvCol(col_spec[columns[col_num]][0],col_spec[columns[col_num]][1],FileData[row_id][col_num])

                    disp_row = disp_row + "\n"


"""Sample CsvChamp usage. We need to grab data from a messy, mean, ugly CSV file and export it as a nice, neat, clean, friendly text file.

#Prompt the user to select a file.
file = GetCsvData(CsvFilePicker())

#Determine column width and alignment for output.
col_spec = CreateColSpec(file)

#Create the output file.
OutputCsv(file, col_spec)

Getting warmer at getting cooler.

In my last post, I attempted to create a Tkinter GUI for my “Coolness” app,  placing the widgets (that is, the buttons, boxes and other objects that make up a GUI interface) using the pack() method only.  This seemed to do little more than make a mess.  Luckily there is another, more precise method in Tkinter for placing widgets: the grid() method.  This is how to tell the widgets where to go when you are serious about getting them to obey you. I therefore sent pack() packing and rewrote my code. Here is Apply Coolness 2.0:

# Apply Coolness 2.0
# Import my cool function first!
from cool_tools import coolness
from Tkinter import *
import ttk

# Create the root window
root = Tk()
root.title(&quot;Apply Coolness&quot;)

# Create the frame to hold the boxes &amp; button
mainframe = ttk.Frame(root)

# &quot;Sticky&quot; tells the frame to expand in all directions
# to fill up the root window.
mainframe.grid(column=0, row=0, sticky=(N, W, E, S))

# Set the variable for the name entered by the user
name = StringVar()

# Label that tells the user what to do.
lblPrompt = ttk.Label(mainframe, text=&quot;Enter the name of the person to be made cool:&quot;)

# Row 0, Column 0 = upper left corner of mainframe.
# The 'grid' option for each successive widget will use row
# and column coordinates in relation to this one.
lblPrompt.grid(column=0, row=0)

# Box in which user enters a person's name.
txtEntry = ttk.Entry(mainframe,width=7)

# column 0, row 1 = place the entry widget directly
# below the label widget, which is row 0.
txtEntry.grid(column=0, row=1, sticky=(N, W, E, S))

# The button that activates the coolness function.
btnCool = ttk.Button(mainframe,text=&quot;Make Cool&quot;)

# Row 2 = the row below the entry box.
btnCool.grid(column=0, row=2, sticky=(N, W, E, S))

# Box that displays the result of the procedure.
txtResult = Text(mainframe, width = 20, height = 10)
txtResult.grid(column=0, row=3, sticky=(E, W))

# Set the focus to the entry box so the user won't have to
# click on it first.

And here is the result:


Now I’m getting somewhere! See the comments in the code for details on how and why I did what I did. This is pretty much the GUI I wanted. We have an entry box for a person’s name, a decidedly cool button to press to apply coolness to that person and a large box to display the result of the operation, including a warning in case the user is so carelessly uncool to attempt to apply coolness to someone who is already cool.

Next, of course, is the complicated part: getting the entry box, the button and the result box to all communicate with each other and the user to accomplish their noble function. Guess what — I haven’t learned in detail how to do that yet, so stay tuned!