.. _ch04-python-libraries:

===============================================================
Manipulating OS functionalities using Python 
===============================================================

-------------------------------------------------------------------------------
``os`` module: directory and file manipulation
-------------------------------------------------------------------------------

The ``os`` module is a portable way of using operating system dependent functionality.
With ``os``, we can basically execute the Linux commands :ref:`ch01-unix-commands` from Python.
For a more complete list please see https://docs.python.org/2/library/os.path.html.

Current directory::

  >>> import os
  >>> os.getcwd()
  '/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04'

List a directory::

  >>> os.curdir
  '.'
  >>> os.listdir(os.curdir)
  ['ch04_python_exception_debugging.rst', 'ch04_python_oop.rst', 'ch04_python_IO.rst', 'ch04_python_libraries.rst']

  
Change to another directory::

  >>> path = '../'
  >>> os.chdir(path)
    >>> os.getcwd()
  '/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters'
  
Make a directory::

  >>> os.mkdir('codes')
  >>> 'codes' in os.listdir(os.curdir)
  True


Rename the directory::

  >>> os.rename('codes','example_codes')
  >>> 'codes' in os.listdir(os.curdir)
  False
  >>> 'example_codes' in os.listdir(os.curdir)
  True

  >>> os.rmdir('example_codes')
  >>> 'example_codes' in os.listdir(os.curdir)
  False

Delete a file ::

  >>> fp = open('junk.txt','w')
  >>> fp.close()
  >>> 'junk.txt' in os.listdir(os.curdir)
  True
  >>> os.remove('junk.txt')
  >>> 'junk.txt' in os.listdir(os.curdir)
  False

-------------------------------------------------------------------------------
``os.path``: path manipulations
-------------------------------------------------------------------------------

``os.path`` provides common operations on pathnames::

  >>> fp = open('junk.txt','w')
  >>> fp.close()
  
  >>> a = os.path.abspath('junk.txt') # displays an absolute path from the top most directory, as opposed to showing a relative one
  >>> a
  '/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/junk.txt'

  >>> os.path.split(a)
  ('/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04', 'junk.txt')

  >>> os.path.dirname(a)
  '/Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04'

  >>> os.path.basename(a)
  'junk.txt'

  >>> os.path.splitext(os.path.basename(a))
  ('junk', '.txt')

  >>> os.path.exists('junk.txt')
  True

  >>> os.path.isfile('junk.txt')
  True

  >>> os.path.isdir('junk.txt')
  False


-------------------------------------------------------------------------------
Running Linux commands
-------------------------------------------------------------------------------
  
You can also run Linux commands :ref:`ch01-unix-commands` using ``os.system``::

  >>> cmd1 = 'ls -l'
  >>> os.system(cmd1)
  
  total 16
  -rw-r--r--  1 dongwook  staff  3708 Nov  5 23:30 ch04_python_IO.rst
  -rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_exception_debugging.rst
  -rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_oop.rst
  -rw-r--r--  1 dongwook  staff  2320 Nov  6 00:07 ch04_python_libraries.rst
  -rw-r--r--  1 dongwook  staff     0 Nov  6 00:02 junk.txt


With ``os.system`` you can indeed perform almost everything *within* a directory::

  >>> cmd2 = 'touch foo'
  >>> os.system(cmd2)
  >>> os.system(cmd1)    # or you can use os.listdir(os.curdir)
  total 16
  -rw-r--r--  1 dongwook  staff  3708 Nov  5 23:30 ch04_python_IO.rst
  -rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_exception_debugging.rst
  -rw-r--r--  1 dongwook  staff     0 Nov  5 22:04 ch04_python_oop.rst
  -rw-r--r--  1 dongwook  staff  3402 Nov  6 00:20 ch04_python_libraries.rst
  -rw-r--r--  1 dongwook  staff     0 Nov  6 00:20 foo
  -rw-r--r--  1 dongwook  staff     0 Nov  6 00:02 junk.txt


-------------------------------------------------------------------------------
Walking a directory
-------------------------------------------------------------------------------

``os.walk`` generates a list of filenames in a directory tree::

  >>> for dirpath,dirnames,filenames in os.walk(os.curdir):
  ...     for fp in filenames:
  ...             print os.path.abspath(fp)
  ... 
  /Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_exception_debugging.rst
  /Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_oop.rst
  /Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_IO.rst
  /Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/ch04_python_libraries.rst
  /Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/foo
  /Users/dongwook/Repos/ucsc/soe/teaching/2015-2016/Fall/AMS209/lectureNote/chapters/chapt04/junk.txt


If you try to see what ``os.walk(os.curdir)`` is::

  >>> os.walk(os.curdir)
  <generator object walk at 0x1007e8050>
  >>> print os.walk(os.curdir)
  <generator object walk at 0x1007e80a0>

and these are not so much useful. You can use ``list`` to see more detail::

  >>> list(os.walk(os.curdir))
  [('.', [], ['ch04_python_exception_debugging.rst', 'ch04_python_oop.rst', 'ch04_python_IO.rst', 'ch04_python_libraries.rst', 'foo', 'junk.txt'])]


.. note::
   Some of you may have already tried to print what ``enumerate`` look like and failed in the same way.
   You can use ``list`` to see what it looks like in this case as well::
     
     >>> mylist=['a','b','c']
     >>> enumerate(mylist)
     <enumerate object at 0x1007c6f50>

     >>> list(enumerate(mylist))
     [(0, 'a'), (1, 'b'), (2, 'c')]
   
-------------------------------------------------------------------------------
``glob``: pattern matching on files
-------------------------------------------------------------------------------

The ``glob`` module provides convenient file pattern matching, which is same as ``ls *.keyword``.
For example, if we wish to find all files that have ``.rst`` file extensions, in Linux you would do::

  $ ls *rst
  ch04_python_IO.rst                  ch04_python_exception_debugging.rst ch04_python_oop.rst       ch04_python_libraries.rst

From Python, you can do the same as::

  >>> import glob
  >>> dir(glob)
  ['__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_unicode', 'fnmatch', 'glob', 'glob0', 'glob1', 'has_magic', 'iglob', 'magic_check', 'os', 're', 'sys']
  
  >>> glob.glob('*.rst')
  ['ch04_python_exception_debugging.rst', 'ch04_python_oop.rst', 'ch04_python_IO.rst', 'ch04_python_libraries.rst']


-------------------------------------------------------------------------------
``sys`` module: system-specific information
-------------------------------------------------------------------------------
The ``sys`` module provides system-specific information related to the Python interpreter.

To see which version of Python you are running and where it is installed::

  >>> import sys
  >>> sys.platform
  'darwin'

  >>> sys.version
  '2.7.10 |Anaconda 2.3.0 (x86_64)| (default, May 28 2015, 17:04:42) \n[GCC 4.2.1 (Apple Inc. build 5577)]'
  
  >>> sys.prefix
  '/Users/dongwook/anaconda'

We already saw that ``sys.path`` is a list of strings that specifies the search path for modules. Initialized from PYTHONPATH::

  >>> sys.path
  ['', '/Users/dongwook/anaconda/lib/python27.zip', '/Users/dongwook/anaconda/lib/python2.7', '/Users/dongwook/anaconda/lib/python2.7/plat-darwin', '/Users/dongwook/anaconda/lib/python2.7/plat-mac', '/Users/dongwook/anaconda/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/dongwook/anaconda/lib/python2.7/lib-tk', '/Users/dongwook/anaconda/lib/python2.7/lib-old', '/Users/dongwook/anaconda/lib/python2.7/lib-dynload', '/Users/dongwook/anaconda/lib/python2.7/site-packages', '/Users/dongwook/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/Users/dongwook/anaconda/lib/python2.7/site-packages/aeosa', '/Users/dongwook/anaconda/lib/python2.7/site-packages/setuptools-17.1.1-py2.7.egg']


-------------------------------------------------------------------------------
Exercise
-------------------------------------------------------------------------------
1. Write a simple routine that removes those
   ``junk_1.txt``, ..., ``junk_n.txt`` files you created in the
   exercise problems from the 
   previous section :ref:`ch04-python-IO`.