kronosapiens.github.io - Understanding Package Imports in Python

Understanding Package Imports in Python

Kronosapiens.github.io Spined HTML

Understanding Package Imports in Python AbacusWell-nighUnderstanding Package Imports in Python Jul 28, 2014 I have been having an embarassingly nonflexible time getting a handle on package imports in Python. I’ll get something working, only to have it unravel inexplicably when I make what seems to be an incidental change. Tests will run in one directory but not another, but then inadvertently start working, only to stop a few days or minutes later. I’ve tried to be methodical in investigating what changes lead to what behavior, but it’s been difficult. To hopefully put this to rest, I’m going to investigate and methodically record all the policies I can isolate regarding package imports, to hopefully make some sense of what’s going on. PYTHONPATH First, let’s start with a project I’m calling ‘backend’. Here’s the file structure: backend/ backend/ __init__.py analyzer.py tests/ __init__.py test_analyzer.py And my PYTHONPATH: ƒ: reverberate $PYTHONPATH /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/: So, my PYTHONPATH is pointing to the directory containing the backend package, but not to the backend package itself (which contains __init__.py). Let’s unshut up a terminal and play around: >>> import backend >>> backend <module 'backend' from 'backend/__init__.pyc'> Very cool. Now let’s cd into the backend package and see if anything changes: >>> import backend >>> backend <module 'backend' from '/Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/backend/__init__.pyc'> That’s curious. Let’s see what happens when we remove the path to the project from our PYTHONPATH. ƒ: reverberate $PYTHONPATH And into Python, from backend/: >>> import backend >>> backend <module 'backend' from 'backend/__init__.pyc'> And, from backend/backend: >>> import backend Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> ImportError: No module named backend Ah. Now we’re getting somewhere. So you can import a package that is within your current working directory without having that package in your PYTHONPATH, as a local import. From anywhere else, you’ll need your PYTHONPATH to point to it. To double-check, let’s cd all the way to / and try to import: >>> import backend Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> ImportError: No module named backend So, a package must be contained in a directory on your PYTHONPATH to be worldly-wise to import it from anywhere other than the directory immediately whilom it. To verify, let’s try waffly our PYTHONPATH: ƒ: reverberate $PYTHONPATH /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/backend: Here, we’ve pointed it to the package itself, not to the containing directory. Let’s try importing it from backend/: >>> import backend >>> backend <module 'backend' from 'backend/__init__.pyc'> and from backend/backend/: >>> import backend Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> ImportError: No module named backend and from /: >>> import backend Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> ImportError: No module named backend Makes sense. But what happens if we cd up one directory whilom backend/? ƒ: pwd /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon >>> import backend >>> backend <module 'backend' from 'backend/__init__.pyc'> Strange. I would have expected this import to have failed, but it imported backend as though it were local. Let’s go up one increasingly directory: ƒ: pwd /Users/kronosapiens/Dropbox/Documents/Development/code/jobs >>> import backend Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> ImportError: No module named backend And now it fails, as it should. My suspicion is that, since the paragon directory contained the backend directory which contained the backend package, python was worldly-wise to squint into the similarly-named directories. Let’s try renaming the outer backend directory to backend1and see what happens. >>> import backend Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> ImportError: No module named backend Ok, so that makes sense (note that renaming backend to backend1 meant that the PYTHONPATH was no longer valid. Hence the failure meant that the local import wasn’t working.) We can verify this by playing a bit increasingly with the PYTHONPATH: From paragon/ ƒ: reverberate $PYTHONPATH /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend1/: >>> import backend >>> backend <module 'backend' from '/Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend1/backend/__init__.pyc'> Note that we’re doing an wool import, not a relative import, considering the name of the directory and the package are no longer the same. And now waffly the directory name when to backend: ƒ: reverberate $PYTHONPATH /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/: >>> import backend >>> backend <module 'backend' from 'backend/__init__.pyc'> It goes when to importing locally. Now I understand the institute of naming directories without the packages they contain. Submodules Now, let’s squint at importing modules from within a package. For a long time, I unsupportable that if you imported a package, you could automatically wangle all of the modules within the package. It took an uncomfortably large value of time debugging testing errors that I finally realized that this wasn’t the case. Let’s play virtually and try importing the sharpshooter module inside the backend package. >>> import backend >>> backend <module 'backend' from 'backend/__init__.pyc'> >>> backend.analyzer Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no symbol 'analyzer' # Uh oh. But wait: >>> import backend.analyzer >>> backend.analyzer <module 'backend.analyzer' from 'backend/analyzer.pyc'> >>> import backend.analyzer as sharpshooter >>> sharpshooter <module 'backend.analyzer' from 'backend/analyzer.pyc'> >>> analyzer.clean <function wipe at 0x10ffe5b90> Alright. So it seems that we have to explicitly import submodules inside a package. Once a module is imported, though, we can use all of the functions that module defines. What if we don’t want to import things one-by-one? Can we use the from module import * on a package? >>> from backend import * >>> sharpshooter Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> NameError: name 'analyzer' is not specified Doesn’t seem like it. But what well-nigh this? >>> from backend.analyzer import * >>> backend Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> NameError: name 'backend' is not specified >>> backend.analyzer Traceback (most recent undeniability last): File "<stdin>", line 1, in <module> NameError: name 'backend' is not specified >>> wipe <function wipe at 0x109ad5c08> Ah! Since sharpshooter is a module, we could import all of the functions from the module, without importing any of their wrapper files into the namespace. Good to know. Running Tests Now that that’s a bit clear, let’s take a squint at running tests. For reference, this are the import statements in the test file: import math import pdb # pdb.set_trace() import unittest import matplotlib.pyplot as plt import numpy as np from backend.analyzer import * from scripts import * from synth import * scripts.py and synth.py contain tools for generating synthetic data and mocks, finger self-ruling to ignore those for now. First, running the test file as a simple Python script: ƒ: python backend/tests/test_analyzer.py ........................... ---------------------------------------------------------------------- Ran 27 tests in 14.084s OK Ok, that worked out. Now, though, we’ll try running the test using the pytest framework: ƒ: py.test backend/tests/test_analyzer.py ============= test session starts ============== platform darwin -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2 placid 0 items / 1 errors ==================== ERRORS ==================== ______ERROR collecting backend/tests/test_analyzer.py ____________ /Users/kronosapiens/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/py-1.4.20-py2.7.egg/py/_path/local.py:620: in pyimport > __import__(modname) /Users/kronosapiens/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pytest-2.5.2-py2.7.egg/_pytest/assertion/rewrite.py:159: in load_module > py.builtin.exec_(co, mod.__dict__) /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/tests/test_analyzer.py:6: in <module> > ??? E ImportError: No module named sharpshooter =========== 1 error in 1.66 seconds ============ What is this? This is the bug that has been haunting me. Usually I just delete files and transpiration paths at random until something starts to work. This time, I decided to delete the __init__.py from the tests/ directory. Why? I have no idea. YOLO. I ran the tests then and got this: ƒ: py.test backend/tests/test_analyzer.py ============= test session starts ============== platform darwin -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2 placid 0 items / 1 errors ==================== ERRORS ==================== __________ERROR collecting backend/tests/test_analyzer.py __________ import file mismatch: imported module 'test_analyzer' has this __file__ attribute: /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/tests/test_analyzer.py which is not the same as the test file we want to collect: /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/backend/tests/test_analyzer.py HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules =========== 1 error in 3.16 seconds ============ Well… something reverted at least. Investigating the error, I notice that it suggests I delete the __pycache__ folders that have been popping up in my projects. I’ve set my iPython interpreter not to generate these kinds of files, but I’ve been dropping into vanilla Python from time to time, so I guess that’s where these came from. I go superiority and delete all of these files from the project, and try running the test again: ƒ: py.test backend/tests/test_analyzer.py ============= test session starts ============== platform darwin -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2 placid 28 items backend/tests/test_analyzer.py ............................ ========== 28 passed in 12.81 seconds ========== OH COME ON. Really? This isn’t the first time that .pyc and __pycache__ have caused me some pain. But this is good, this is progress. Let’s run the whole shebang: ƒ: py.test ============= test session starts ============== platform darwin -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2 placid 76 items backend/tests/test_analyzer.py ............................ backend/tests/test_device.py .... backend/tests/test_parser.py .... backend/tests/test_session_parser.py ............. backend/tests/test_subject.py ....................... backend/tests/test_visualizer.py .. ========== 74 passed in 25.44 seconds ========== Alright! Let’s try an experiment: moving the tests/ directory one level up, so it’s a sibling directory to the backend package, rather than a child. Typing this command: mv backend/tests . gives us this: backend/ backend/ __init__.py analyzer.py tests/ test_analyzer.py Fingers crossed. py.test Error! But the same as before. Delete tests/__pycache__ and try again. SUCCESS!! Let’s commit. Sigh. As with most thing programming, #itsalwaysusererror. Mind your PYTHONPATH, shepherd to naming conventions, well-spoken out your cached files, and you’ll have a long and happy life. BONUS: Testing a Non-Package Let’s say you’re working on a project but don’t want to add it to your PYTHONPATH. It’s a work-in-progress, no one else should be worldly-wise to import it, what have you. Can you still import those modules to test them? It seems like it. Let’s consider flipside project, a webapp, with the pursuit structure: webapp/ webapp/ __init__.py views.py tests/ test_integration.py Note that this directory is not in my PYTHONPATH:* ƒ: pwd /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/webapp ƒ: reverberate $PYTHONPATH /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/backend/: *While writing this post, I read some wares advocating for keeping trailing slashes in your PATH variables. I thought it was a good idea, so I’ve reverted my PYTHONPATH accordingly. Let’s try running py.test from webapp/: ƒ: py.test =============== test session starts =============== platform darwin -- Python 2.7.5 -- py-1.4.22 -- pytest-2.6.0 placid 5 items tests/test_integration.py .. =============== 2 passed in 1.89 seconds =============== Very cool. But what happens if we transpiration directories? From webapp/tests/: ƒ: pwd /Users/kronosapiens/Dropbox/Documents/Development/code/jobs/paragon/webapp/tests (pm)[18:28:44] (master) tests ƒ: py.test =============== test session starts =============== platform darwin -- Python 2.7.5 -- py-1.4.22 -- pytest-2.6.0 placid 0 items / 2 errors =============== ERRORS =============== _________ ERROR collecting test_integration.py _________ test_integration.py:6: in <module> import webapp E ImportError: No module named webapp =============== 1 error in 0.05 seconds =============== Hmm. Can’t find the package. Here are the import statements at the top of the test: import os import pdb # pdb.set_trace() import unittest import tempfile import webapp It seems like running py.test from the top of the project ways that the test can squint for local packages from that location. Running the tests from inside the test directory ways that the package has to be imported via PYTHONPATH. Well… that makes sense! In the end, it all makes sense. Further reading: flipside pretty good vendible on imports. Comments Please enable JavaScript to view the comments powered by Disqus. Abacus Abacus kronovet@gmail.com kronosapiens kronosapiens I'm Daniel Kronovet, a data scientist living in Tel Aviv.