Manipulating OS functionalities using Python

os module: directory and file manipulation

The os module is a portable way of using operating system dependent functionality. With os, we can basically execute the Linux commands Basic Unix/Linux Commands from Python. For a more complete list please see https://docs.python.org/2/library/os.path.html.

Current directory:

>>> import os
>>> os.getcwd()
'/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04'

List a directory:

>>> os.curdir
'.'
>>> os.listdir(os.curdir)
['ch04_python_exception_debugging.rst', 'ch04_python_oop.rst', 'ch04_python_IO.rst', 'ch04_python_libraries.rst']

Change to another directory:

>>> path = '../'
>>> os.chdir(path)
  >>> os.getcwd()
'/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters'

Make a directory:

>>> os.mkdir('codes')
>>> 'codes' in os.listdir(os.curdir)
True

Rename the directory:

>>> os.rename('codes','example_codes')
>>> 'codes' in os.listdir(os.curdir)
False
>>> 'example_codes' in os.listdir(os.curdir)
True

>>> os.rmdir('example_codes')
>>> 'example_codes' in os.listdir(os.curdir)
False

Delete a file

>>> fp = open('junk.txt','w')
>>> fp.close()
>>> 'junk.txt' in os.listdir(os.curdir)
True
>>> os.remove('junk.txt')
>>> 'junk.txt' in os.listdir(os.curdir)
False

os.path: path manipulations

os.path provides common operations on pathnames:

>>> fp = open('junk.txt','w')
>>> fp.close()

>>> a = os.path.abspath('junk.txt') # displays an absolute path from the top most directory, as opposed to showing a relative one
>>> a
'/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/junk.txt'

>>> os.path.split(a)
('/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04', 'junk.txt')

>>> os.path.dirname(a)
'/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04'

>>> os.path.basename(a)
'junk.txt'

>>> os.path.splitext(os.path.basename(a))
('junk', '.txt')

>>> os.path.exists('junk.txt')
True

>>> os.path.isfile('junk.txt')
True

>>> os.path.isdir('junk.txt')
False

Running Linux commands

You can also run Linux commands Basic Unix/Linux Commands using os.system:

>>> cmd1 = 'ls -l'
>>> os.system(cmd1)

total 16
-rw-r--r--  1 dongwook  staff  3708 Nov  5 23:30 ch04_python_IO.rst
-rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_exception_debugging.rst
-rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_oop.rst
-rw-r--r--  1 dongwook  staff  2320 Nov  6 00:07 ch04_python_libraries.rst
-rw-r--r--  1 dongwook  staff     0 Nov  6 00:02 junk.txt

With os.system you can indeed perform almost everything within a directory:

>>> cmd2 = 'touch foo'
>>> os.system(cmd2)
>>> os.system(cmd1)    # or you can use os.listdir(os.curdir)
total 16
-rw-r--r--  1 dongwook  staff  3708 Nov  5 23:30 ch04_python_IO.rst
-rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_exception_debugging.rst
-rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_oop.rst
-rw-r--r--  1 dongwook  staff  3402 Nov  6 00:20 ch04_python_libraries.rst
-rw-r--r--  1 dongwook  staff     0 Nov  6 00:20 foo
-rw-r--r--  1 dongwook  staff     0 Nov  6 00:02 junk.txt

Walking a directory

os.walk generates a list of filenames in a directory tree:

>>> for dirpath,dirnames,filenames in os.walk(os.curdir):
...     for fp in filenames:
...             print os.path.abspath(fp)
...
/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_exception_debugging.rst
/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_oop.rst
/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_IO.rst
/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_libraries.rst
/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/foo
/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/junk.txt

If you try to see what os.walk(os.curdir) is:

>>> os.walk(os.curdir)
<generator object walk at 0x1007e8050>
>>> print os.walk(os.curdir)
<generator object walk at 0x1007e80a0>

and these are not so much useful. You can use list to see more detail:

>>> list(os.walk(os.curdir))
[('.', [], ['ch04_python_exception_debugging.rst', 'ch04_python_oop.rst', 'ch04_python_IO.rst', 'ch04_python_libraries.rst', 'foo', 'junk.txt'])]

Note

Some of you may have already tried to print what enumerate look like and failed in the same way. You can use list to see what it looks like in this case as well:

>>> mylist=['a','b','c']
>>> enumerate(mylist)
<enumerate object at 0x1007c6f50>

>>> list(enumerate(mylist))
[(0, 'a'), (1, 'b'), (2, 'c')]

glob: pattern matching on files

The glob module provides convenient file pattern matching, which is same as ls *.keyword. For example, if we wish to find all files that have .rst file extensions, in Linux you would do:

$ ls *rst
ch04_python_IO.rst                  ch04_python_exception_debugging.rst ch04_python_oop.rst       ch04_python_libraries.rst

From Python, you can do the same as:

>>> import glob
>>> dir(glob)
['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_unicode', 'fnmatch', 'glob', 'glob0', 'glob1', 'has_magic', 'iglob', 'magic_check', 'os', 're', 'sys']

>>> glob.glob('*.rst')
['ch04_python_exception_debugging.rst', 'ch04_python_oop.rst', 'ch04_python_IO.rst', 'ch04_python_libraries.rst']

sys module: system-specific information

The sys module provides system-specific information related to the Python interpreter.

To see which version of Python you are running and where it is installed:

>>> import sys
>>> sys.platform
'darwin'

>>> sys.version
'2.7.10 |Anaconda 2.3.0 (x86_64)| (default, May 28 2015, 17:04:42) \n[GCC 4.2.1 (Apple Inc. build 5577)]'

>>> sys.prefix
'/Users/dongwook/anaconda'

We already saw that sys.path is a list of strings that specifies the search path for modules. Initialized from PYTHONPATH:

>>> sys.path
['', '/Users/dongwook/anaconda/lib/python27.zip', '/Users/dongwook/anaconda/lib/python2.7', '/Users/dongwook/anaconda/lib/python2.7/plat-darwin', '/Users/dongwook/anaconda/lib/python2.7/plat-mac', '/Users/dongwook/anaconda/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/dongwook/anaconda/lib/python2.7/lib-tk', '/Users/dongwook/anaconda/lib/python2.7/lib-old', '/Users/dongwook/anaconda/lib/python2.7/lib-dynload', '/Users/dongwook/anaconda/lib/python2.7/site-packages', '/Users/dongwook/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/Users/dongwook/anaconda/lib/python2.7/site-packages/aeosa', '/Users/dongwook/anaconda/lib/python2.7/site-packages/setuptools-17.1.1-py2.7.egg']

Exercise

1. Write a simple routine that removes those junk_1.txt, ..., junk_n.txt files you created in the exercise problems from the previous section Recap & Outlook.