Several years ago I took a look at Base, the database component of OpenOffice and now, LibreOffice. It seemed extremely rudimentary compared to MS Access. For some reason, it came into my head to check it out again recently. Base seems to have come a long way since my first encounter. I noted that it has the ability to construct complicated SQL queries graphically, and build forms that include drop-down boxes and lookups. It meets my main qualifications for writing essential apps such as my current finance management project: it’s open source and cross platform. For a few days, it appeared so suited to what I’m trying to accomplish that it seemed this blog’s title would have to change to “My LibreOffice Base Adventure.” But it wasn’t meant to be. The main reason is that as I continued to experiment with Base, it soon became apparent that, in order to do what I want, I would need to go beyond using only tables, forms and queries, and learn to code in LibreOffice Basic — LibreOffice’s equivalent to Microsoft’s Visual Basic that allows more sophisticated programming of the behavior of Base’s database objects. Why, I asked myself, after coming as far as I have with Python, then PySide, would I want to set all that aside to learn yet ANOTHER language? To be honest, what I really needed to do was get back on track with what I’ve already built, and take it to the next level. Amazing how the thrill of learning something new can sometimes distract from persevering with what I already have began learning. So now, back to work. I found this PySide tutorial this past week and hope to use it to brush up on what I’ve already learned, and then continue forward.
Sometime in the last month, I ran across one of my all-time favorite PC games: SimCity 4. I was quickly pulled back into the wonderful (for me, at least) world of creating cities and watching them grow, carefully managing resources and the economy so as to attract as many “Sim citizens” as possible. Heck, the thought even crossed my mind to write a Python app to calculate the optimal procedure for growing a huge SimCity 4 metropolis with skyscrapers! One little problem: I’m absolutely clueless as to how to even begin such a task, or even think about it. Besides, I haven’t finished my little data management project yet. I like to think of myself as a firm believer in finishing one project before moving on to the next, though I adhere to this principle less than perfectly.
Now back to Python land. As recounted in my last post, about a month ago I started dabbling in the sorcery known as ‘Qt’, and in doing so discovered the QSqlRelationalTableModel, an object that lets me query multiple Sqlite tables, complete with combo boxes providing lookup values from the “foreign” tables, without writing one single line of SQL. Next goal: drop a TableView based on a QSqlRelationalTableModel into my lovingly designed, Microsoft Access mimicking main form.
..and code the button and drop down boxes to insert, filter and sort transactions. The view based on the QSqlRelationalTableModel will go in the rectangular box. I just need to plan some time away from my responsibilities as Mayor so I can re-focus on my coding. Target date: Saturday, July 12. Nothing like a public commitment to help me focus on a goal! Wish me luck. I’ll have an update no matter what happens (or does not happen).
UPDATE 2014-06-01: Added links to code examples that helped me learn how to do this.
Here it is, people…brought to you by the magic of the QSqlRelationalTableModel: my editable transactions display widget, complete with combo boxes for selecting categories and accounts from the ‘accounts’ table, coded in elegant, model-view-controller style and in less than 70 lines of code, counting the blank lines added for readability. I’m viewing data from multiple Sqlite tables, even editing, without a single line of SQL. All I have to do is tell ‘QSRTM’ (my affectionate nickname for my new friend, QSqlRelationalTableModel) the name of the table and what to display, and he obediently gets what I want the way I want it. Meanwhile Tkinter is throwing a tempter tantrum in the corner when I try to ask it nicely to please synchronize row selection across a set of list boxes.
While documentation such as Pyside’s is excellent and thorough, I find I need examples of how to use a class. I experiment with altering example code to fit what I am trying to accomplish, and when I feel like I have a grasp of the class’s basics, I go back to the documentation to get a full appreciation of all its capabilities. Two table model examples in particular were essential to me figuring out how to use the ‘QSRTM’: this one which demonstrates general use of a table model, and this one which is a ‘QSRTM’ example.
Screenshot below, and my code is here.
Once upon a time, fresh out of Code Academy, I Googled for a way to design a GUI in Python. I came across Qt Designer. Cool! It reminded me of Microsoft Access and Visual Basic. I could lay out windows, buttons, boxes and so on just the way I wanted, set their properties and add some code. I began learning about “slots” and “signals”, Qt’s system of connecting what a widget is supposed to do to what the user does to it. But before long, it hit me:
The code I was using to create this wonderful GUI was not Python; it was Qt.
Somewhere along the line I’d read that Qt could be used to make a GUI for use in Python, yet Qt and Python are completely separate languages, so how, I asked myself, could I possibly use a GUI created in Qt, in Python? It sounded about as likely as running a Mac OS application on my Android phone. For whatever reason at that time, I couldn’t find a clear answer to this, and other options such as wxPython and Tkinter drifted into my view, so Qt was quickly forgotten. wxPython looked cool but when I decided it would be best for me as a newbie to learn in Python 3, wxPython quickly dropped off my radar. I noted Tkinter is included with Python and has been around forever, so it seemed to be the best option of the three. Unfortunately, I found Tkinter’s documentation to be fragmented and in some cases, ancient. Trying to produce a simple table object gradually wore me down over a period of months, which taught me the hard way that Tkinter is apparently just fine for GUIs with simple buttons, text boxes and lists, but nearly useless for data objects any more complex than a listbox. The one or two times I’ve run across table-like objects for Tkinter, I found they were not compatible with Python 3.
Luckily, I was only one more frustrated Google search away from discovering an amazing concept that brings diverse languages together in harmony, letting them play ball joyously with one another and get along like BFFs:
I eventually discovered Pyside, which re-introduced me to its friend Qt, and now Qt and I are buddies again! The secret? Lay out, code and design the GUI of my dreams in Qt, and when done, it’s only one magical command-line incantation away form being transformed into a Python class object, ready to be called upon any time! Pyside changed the direction of my DataQ project and injected it, and my Python learning adventure, with new life. Before long I was able to finally create a real GUI widget for my app and even able to my data into it.
Next step: wrapping my brain around model-view programming.
Have I got a deal for you! How about a GUI widget with over double the functionality, with less than half the code? Here it is:
import sys from PySide.QtCore import * from PySide.QtGui import * from PySide.QtSql import * # My function for assembling SQL queries for my 'transactions' table. I'll share the code another time. from dq_sql import * app = QApplication(sys.argv) class DqTableForm(QWidget): def __init__(self, parent = None): super(DqTableForm,self).__init__(parent) self.setWindowTitle('Transactions') self.setMinimumWidth(950) self.setMinimumHeight(300) db = QSqlDatabase.addDatabase("QSQLITE") db.setDatabaseName("dataq.db") db.open() self.DqModel = QSqlQueryModel() # My TransQry function takes parameters and returns a SQL query using the parameters as criteria for filtering my 'transactions' table. self.DqModel.setQuery(TransQry("2014-03-01","2014-03-31","DESC","","","202",""), db) self.DqTable = QTableView() self.DqTable.setModel(self.DqModel) self.DqTable.hideColumn(0) self.DqTable.resizeColumnsToContents() self.DqTable.resizeRowsToContents() grid = QGridLayout() grid.addWidget(self.DqTable,0,0) self.setLayout(grid) form = DqTableForm() form.show() app.exec_()
See? Less than 45 lines. Sure beats the nearly 100-line, marginally useful alternative, which I have been sweating and struggling under for months. Lesson learned: there’s a very fuzzy boundary between challenging myself toward productive learning, and finding the most efficient solution to a need. In this instance, given the complex data filtering and sorting I’m trying to accomplish, Qt’s QTableView soundly beats anything Tkinter is even remotely capable of without astronomically more code. Shout out to Bo’s Qt GUI tutorials for introducing me to Qt, the documentation for the PySide Python Qt bindings for helping me figure out QTableView, and to my friend and constant companion Google, for helping me evolve.
In the last few weeks I realized I have a long way to go to figure out how the MultiListbox code works…it seems like even the simple function of selecting a line of data requires torturous hand-coding. Almost two months after setting out on my quest for a Python data management application, I am still tripping and stumbling over a single GUI element. And yet all it does is display columns of data, something Microsoft Access does in its sleep, and the MultiListbox isn’t even editable. I’ve become distracted and discouraged. There are functions in the MLB class that I’m not even close to understanding, and there is no detailed explanation of the code anywhere. Lately I’ve thinking, there’s got to be a better way. There’s got to be a way where I can focus my coding on what I want the GUI widgets to DO, instead of struggling just to bring them into functional existence.
So, this afternoon, I once again Googled “Python GUI builder” or something like this and found this series of videos on building Python GUI applications with the Qt framework. https://www.youtube.com/user/Deusdies2.
The first series of videos walk the user through how to download and install a Python interface to the Qt development framework, in this case PySide. The author of this video series uses PyCharm on Windows 7, so his desktop and mine are nearly identical, meaning it couldn’t be easier to follow along with him. The first few videos walk through the creation of simple Qt-based GUI applications such as a calculator. Eventually it moves on to QtDesigner, which, wonder of wonders, is a “drag and drop” GUI designer very much like Microsoft Access and Visual Basic. Where Tkinter requires 100% hand-coding of just about every step of a widget’s function, even PySide coding by itself without the graphical Qt Designer seems much simpler and less labor intensive.
Yes, I have spent many months on Tkinter. Yes, it seemed like I was on the right track when I discovered the MultiListbox. I enjoyed, for a time, the challenge of trying to figure it out. But somewhere along the line, it wasn’t fun anymore. It was a chore. This made it all too easy to become distracted with other projects. That was the signal that something was wrong. I like a challenge and I have loved learning Python, but when it’s no longer enjoyable, it’s no longer worth doing. It’s a little intimidating to think of learning a completely new GUI builder. However once I saw that drag and drop designer, that shimmering promise of instantly knowing my app will look the way I want it to without all the trial and error, and each item I drop into my app already knowing how to do what it’s supposed to and all I have to to is give it parameters, I thought, “This could make Python fun again.” So off I go on my next, unexpected Python path. I’m going spend the next phase of my Python adventure getting as much as I can out of Mr. Milanovic’s videos, and of course, sharing the results.
It would be easy to grab a chunk of someone else’s code, modify it only as necessary, and through a lot of trial and error get it to do what I want. That is what I originally set out to do when I discovered the MultiListbox custom widget in this project. Using this method I might be able to patch together something functional, without ever learning in depth about how the code actually works. This course of action is a continual temptation to my impatient self, especially now as I am grappling with the most complex GUI object in my DataQ application.
But that is not my goal. I don’t just want to learn how to use Python for specific tasks; I want to learn how Python WORKS. The MultiListbox fascinated me because it’s not just any old Tkinter widget; it’s a custom widget made out of other widgets — in this case, a series of Listboxes put together to display rows of data. In examining the MultiLisbox code, I realized I knew very little how even a single Listbox works, let alone a group of them squashed together and synchronized. I therefore decided to learn about and experiment with Listboxes, learning how they work from the ground up. I have found it hard to stay focused. Sometimes I get away from Python completely for days at time because my brain just feels fried by it.
Despite the challenges, I’m happy to report that I’m making progress. I’ve even started designing my very own MultiListbox class (mostly) without even looking at the original code! My version makes each individual column in the MultiListbox its own class, and I use the .grid() geometry manager instead of the often unpredictable .pack(). Here is the ‘rough draft’: Rob’s MultiListbox.
As of today, I am no longer allowing comments or user registrations on this blog.
The image below sums up the reasons. Out of the hundreds of “comments” (in quotes because they don’t even deserve to be called such) I have received, I can count on the fingers of one hand the number that are from actual readers of this blog who have something to contribute. The same goes for “user” registrations.
I wish I could devote all my working hours to my Python learning and this blog, and if that were the case, perhaps I would think differently. However, I have a full-time job that is not Python related, a wife, and numerous other family and social commitments that are important to me. I simply don’t wish to devote the time to moderating “canned lunch meat”, and I certainly don’t want to spend even a penny on software to do it for me.
For those real humans out there who have been following this blog, especially those who have commented, thank you for your support and please accept my apologies. Feel free to contact me through Google Plus or Twitter if you have any feedback or suggestions regarding the blog.
The first chapter of my quest was the hunt for a suitable GUI element to display data in columns. I eventually settled on the Multilistbox, a custom widget I found that was created using the standard Tkinter Listbox widget. I then enthusiastically dove into learning all I could about Tkinter. This revealed the need to thoroughly test my Multilistbox with real-world data..of which I had none. Then I focused on methods of getting data into my DataQ app, which led to experimenting with code to parse data from text and CSV files. Before I knew it I was off on multiple, simultaneous rabbit trails, including trying to figure out how to parse data from Evernote notes. Before I knew it, I was wandering aimlessly and making little progress.
Now I’ve decided to focus back on the original goal: financial management. This would be, by far, the most useful function for me, something I would use on a daily basis. It will also be instructive for other possible uses for DataQ. I therefore have now imported all 3000+ of my financial transactions from my Microsoft Access database to Sqlite. I’m laying aside the GUI for now and focusing on the heart and soul of DataQ which is data! I need to learn how to get the data I want using queries, something I’ve had passing acquaintance with through MS Access, but don’t know thoroughly because Access lets me build queries graphically. That’s great for getting results quickly, not so great for learning.
I have two tables: ‘transactions’ and ‘accounts’, with the following structure:
TransactionID * Integer primary key
Category * Income or expense category. In correct bookkeeping terminology, also an ‘account.’
Income * Amount
Expense * Amount
Account * Asset or liability account, ex: checking, savings, credit card
AccountNum * Integer primary key
AccountType * Asset, liability, income, expense
Clear * 1 = transaction has cleared the bank, 0 = not cleared.
Both the ‘Category’ and ‘Account’ fields contain an ‘AccountNum’ from ‘accounts.’ Every transaction query includes a calculated field called ‘Net’ which subtracts the ‘Expense’ amount from the ‘Income’ amount, giving the net effect on the account balance. A typical SQL query will look like the following, which displays all transactions in descending order from most recent:
SELECT transactions.Date, transactions.PayeeOrName, transactions.TransactionDescription, accounts.AccountDescription AS Category, transactions.Income, transactions.Expense, (transactions.Income - transactions.Expense) AS Net, accounts_1.AccountDescription AS Account, transactions.Clear
FROM (transactions INNER JOIN accounts AS accounts_1 ON transactions.Account = accounts_1.AccountNum)
INNER JOIN accounts ON transactions.Category = accounts.AccountNum ORDER BY transactions.Date DESC, PayeeOrName ASC
The data source for the Multilistbox will be obtained by queries such as this, so I think it’s time I get to know Sqlite better.
Wow. The learning process sure is messy, and at times embarrassing.
Here’s how it all started:
I have a folder called ‘tools’ which contains various modules I wrote that I want to be accessible for import to all my Python projects. One of these modules is called ‘DbSession’ — it contains a class for handling various Sqlite database tasks. Each Python project is in its own folder, with the ‘tools’ folder as a ‘sibling’ under the same parent directory, as in this example:
I use Dropbox to sync my projects between two computers, one running Windows XP and one running Windows 7. The path to the Dropbox folder is different on each PC. No matter which PC I’m using, I want the modules in ‘tools’ to be accessible for importing to all my projects.
As previously posted, my first solution was to do a sys.path.append() in my code, but afterward I learned this was not the best solution because it would involve changing my code every time I switched computers or re-arranged directories. I then decided I’d like to be able to set the PATH in each PC appropriately so the imports will work on both without changing my code. After much Googling and sifting through what seemed to be a multitude of possible answers, I found what I *thought* was the solution, but upon further testing, discovered I was mistaken. I thought making the imports work in my project directories was just a matter of adding my ‘tools’ directory to the PATH variable. Here’s what I did (Windows XP machine):
Added ‘tools’ directory to PATH variable and confirmed it is set correctly by typing “PATH” on a command line:
and Settings\Rob\My Documents\Dropbox\projects\tools
The ‘tools’ directory contains a test ‘Hello World’ program, ‘hello.py’, which I can run from the C:\ directory just by typing the file name. This proves the PATH variable is set correctly to ‘tools’:
..although if I try to run ‘hello.py’ from Python, Python can’t find it!
python: can’t open file ‘hello.py’: [Errno 2] No such file or directory
Here’s my attempt at importing from the module ‘DbSession.py’, which resides in ‘tools.’ First, I ran Python from the C:\ directory. It can’t find the ‘DbSession’ module in ‘tools’, even though the above test with ‘hello.py’ at least showed the ‘tools’ directory was correctly set in PATH. I tried putting a blank ‘__init__.py’ file in ‘tools’, but it still didn’t work:
Python 3.3.4 (v3.3.4:7ff62415e426, Feb 10 2014, 18:12:08) [MSC v.1600 32 bit (In
tel)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.
>>>from DbSession import DbSession
Traceback (most recent call last):
ImportError: No module named ‘DbSession’
When I ran Python from the ‘tools’ directory the import worked fine, proving there’s no problem with the module itself:
c:\Documents and Settings\Rob\My Documents\Dropbox\projects\tools\python
Python 3.3.4 (v3.3.4:7ff62415e426, Feb 10 2014, 18:12:08) [MSC v.1600 32 bit (In
tel)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.
>>>from DbSession import DbSession
So what’s up? I KNOW I set the PATH right…how come Python can’t import from the ‘tools’ directory into a project directory, when I can run the ‘hello.py’ code, which is also in ‘tools’, from any directory I want? This seemed so insane it tempted me to go back to using sys.path.append().
Until I ran across this one little nugget, in a 5-year-old article, on a site I’d never heard of:
“However, unlike PATH (which tells the operating system which directories to look for executable files in), PYTHONPATH is used by the Python interpreter to find out where to look for modules to import.”
BINGO! EUREKA! HALLELUJAH!
So, I need TWO SEPARATE PATH VARIABLES…the PATH variable to help Python find itself and its various packages:
..and the PYTHONPATH variable to help it find modules for importing!
I don’t know if it’s my misfiring brain, but somehow in all my recent travels, this simple truth eluded me. So, now my jolly good mates at Stereoplex have been blessed with a mention and a backlink. Cheers!