.. _ch01-programming-platform-os: Programming Languages, Platforms, Operating Systems ===================================================== .. _ch01-scientific-languages: Scientific languages --------------------------------------- One of the crucial components of the entire course work is to write computer programming codes for homework sets, programming assignments, and a final coding project. A choice for a default reference language is going to be Fortran 90 in this course. Fortran 90 (or higher) has been one of the most widely used programming languages in high performance computing communities for over half a centry. It is easy, compatible, and has been chosen for various benchmark tests that run on large scale computing architectures, or supercomputers. The main goal in this course is to learn practical experiences in different programming languages as well, such as C and python, and learn how to use them for of scientific computing. You should be able to submit your course assignments by successfully implementing required numerical algorithms in those languages. .. _ch01-platforms: Computing platforms ---------------------------------- In order to conduct scientific programming, it is required that you have an access to a Linux/Unix computing platform, where you can excersise such series of programming studies. Basically, there are several options to bring a Linux/Unix computing system to your daily scientific adventures. Please see also :ref:`ch01-installingLinuxOnPC`. * if your machine is either a Linux or Mac machine, use your own machine to run your code locally. * if your machine is a Windows PC, you can remotely access to a Linux computer using an X-forwarding terminal such as `PuTTy `_ over the network. Putty is one of the SSH clients on Windows allowing you to work on a remote Linux computers. * if you prefer to run programming locally rather remotely (e.g., limited internet access at your place), you can either install `Cygwin `_ which brings you functionality similar to a Linux distribution on Windows. Note that Cygwin is *not* a way to run native Linux apps on Windows. * if you rather wish to have a native Linux setup on your Windows PC, you can run a pure linux environment using a free virtualization software called `virtualbox `_ which is quite excellent. This also allows file sharing between your host operating system (e.g., Windows) and the virtual operating system (e.g., Linux). A good Youtube video to learn how to share folders between the host (e.g., Windows) and the guest (e.g., Ubuntu) is found `here `_. **Remark** You can also learn useful tips, not only from google searches but also from youtube these days. So please use those visual resources as well as readable resources. **Remark** And, don't forget one thing. One very important thing. If you need help, please don't be shy and never hesitate to ask around good people. And I am one of them, hopefully. .. _ch01-ams-resources: Computing Resources in AMS """""""""""""""""""""""""""""""""""""""""""" In addition to your own computing resources (your own Linux or Mac), you can use other computers available on campus. Two choices are (i) three non-parallel servers: jerez, muscat, and mencia, and (ii) a parallel linux cluster called 'Grape'. These machines are owned by the AMS department. You can see `the description of the AMS computers `_. If you are not an SOE student, please come to see the instructor to get an account on these machines. Basic Fortran and C/C++ compilers (e.g., GNU gfortran), some other necessary libraries, and softwares (gnuplot, matlab, idl, etc) are available on them. If you find any specific software needs to be installed, please let the instructor know your request. Since it uses a Linux operating system (e.g., Grape runs on a Linux operating system called *Rocks*), and it is a remote cluster, in order to remotely access to Grape for instance, you will need to * learn how to use Unix/Linux command lines (or simply commands) (see :ref:`ch01-unix-commands`) * make sure you have remote access to it and to its text editors (emacs, xemacs, vi, vim, etc.). As mentioned, this should be trivial if you are using either a Linux or Mac machines. If you are a Windows PC user, you can install PuTTy (enough for accessing Grape remotely) or Cygwin (allowing remote access capability as well as Linux functionality), or linux on virtualBox (ultimately better than the previous two options). If you haven't had any chance to work on Linux/Unix type operating systems, please make sure you first familiarize yourself with basic Linux/Unix commands (This is very crucial. See :ref:`ch01-unix-commands`). It is your prime responsibility to learn about working on Linux/Unix environment as quickly as possible -- you should trust that you can do it! The instructor is happy to help you, however, is bound to be limited to provide you with detailed technical supports at all levels. **Remark** If you prefer to use your own laptop/desktop to program, it is your responsibility to install Fortran and all the libraries we will be using. You should be able to find a free Fortran 90 compiler (e.g., GNU gfortran) for most platforms. The most basic one will be sufficient for this class, but please be aware that the quality of a compiler has a lot to do with the speed of execution of the program. **Remark** In order to obtain an account on the Grape Linux cluster machine in AMS, please go visit the link and request an account for youself for the course: * `Get a BSOE account `_ You need to specify me as your sponsoring faculty member. .. _ch01-unix: Scientific computing on Linux/Unix, Mac OS X -------------------------------------------------------------------------------------- Most projects of scientific computing heavily rely on developers' interactive programming and software handlings across various components of softwares. Engineers, scientists and researchers in scientific computing are often assumed to carry out programming practices using different combinations of command lines on a Unix/Linux-based operating system (OS). .. _ch01-shell: Shell """"""""""""""""""""""" Most common examples of such OS are `Unix `_, `Linux `_ and `Mac OS X `_, where you perform computing jobs by typing in your command lines on a terminal, called a `shell `_. The shell provides users with a user interface by which users can access the computer's operating system and/or interact with target software packages in order to carry out computational jobs. There are a couple of choices for Unix/Linux shells: `article 1 `_; `article 2 `_; `article 3 `_; `article 4 `_. We are going to use `bash `_ shell as our default choice. If your experience with computers has only been in using Windows PC, with which you interact by clicking some icon apps to open and conduct jobs, this will be an absolutely new opportunity to use computers in a very different way. It is true that Windows system has its kernel OS, called `DOS `_, where users can possibly run command-line jobs, we will *not* use DOS (like every wise soul wouldn't) for scientific computing. .. _ch01-installingLinuxOnPC: Installing Linux on Your PC """""""""""""""""""""""""""""""""""""""""""" There are a couple of different ways to run Unix/Linux-system on your Windows PC. Here are my recommendations (you will need to do one of them). If you are considering an updating your laptop anyway, please do: * get a Mac, and download and install `Xcode `_ on it. This can be easily done in a terminal window (go to Applications –> Untilities –> Terminal to open a terminal) by typing in (also see `how-to-1 `_): :: $ xcode-select --install * get a PC, erase the Windows OS, and install a Linux OS (e.g., `Ubuntu `_) as a sole operating system. Or, you can buy one of these `machines `_ that come with Ubuntu out of the box. Otherwise, if you want to keep your Windows PC but want to do some cool scientific computings, please do: * install `virtualbox `_ (easy and best), * install `Cygwin `_ (altenatively good and efficient), * configure dual-boot to run both Windows and Linux, * access an external Linux machine over the network using `PuTTy `_ (least recommended) * More to read `here `_ If you are one of the computer grus, you might as well come up with a crazy combination, such as (I saw few people who did this): * get a Mac, erase Mac OS X, and install Linux on it (you wonder why you would want to do this?). **Remark** Further details on the Linux installing instructions on PCs can be easily found online, e.g., `how-to-2 `_. You will also find online tutorials on how-to instructions on Youtube as well. Please don't forget to use these great resources, and of course, you're always welcome to ask me or TA. .. _ch01-unix-commands: Basic Unix/Linux Commands """"""""""""""""""""""""""""""""""""""""""""""""""""""""""" In this section we give a quick overview on some of the basic Linux commands. There are several rules in using command lines in Linux. Some of the important rules are * Commands are case-sensitive. * Make sure you always logout yourself by typing ``exit`` when you're done. * The Linux command lines enables you to create complex functions by combining built-in command lines together. This capability gives you countless ways to make your commands work in various different ways. Here you're introduced to learn very basic Linux command lines. For more comprehensive studies on various options associated with Linux commands, you can use to display a manual page using the ``man`` command followed by your search commands. For instance, if you want to learn more about ``cp`` command :: $ man cp #. Here are some basic commands for managing files: * list directory contents:: $ ls * ``ls`` in long format:: $ ls -l * ``ls`` all entries of contents:: $ ls -a * rename (or move) ``filename1`` to ``filemane2``:: $ mv filename1 filename2 * copy ``filename1`` to ``filename2``:: $ cp filename1 filename2 * remove a file named ``filename``:: $ rm filename * display the contents of a file named ``filename`` as much as will fit on your screen:: $ more filename * similar to ``more`` with the extended navigation capability allowing both forward and backward navigations:: $ less filename * print the entire ``filename`` rather than a page at a time:: $ cat filename * view multiple files' contents and direct the output to save into a new file ``output`` using `standard output `_ ``>`` (``STDOUT``):: $ cat filename1 filename2 > output * you can also use another standard output ``>>`` (``STDOUT``). In this case, the contents from filename 3 will be appended at the end of the already existing contents in output:: $ cat filename 3 >> output * read more on stdout, redirections, etc.: `article 1 `_, `article 2 `_. Some of example problems are:: $ touch foo.txt $ echo I like AMS 209! > foo.txt $ echo I like scientific computing >> foo.txt $ echo I like scientific computing > foo.txt $ cat foo.txt > output.txt $ cat foo.txt 1> output.txt $ cat foo.txt 2> output.txt $ cat nop.txt 1> output.txt $ cat nop.txt 2> output.txt $ cat nop.txt > output.txt 1>&2 $ cat nop.txt > output.txt 2>&1 * tells you number of lines, words and characters in ``filename``:: $ wc filename * creating an empty file named ``filename`` (multiple filenames after ``touch`` command will create multiple empty files):: $ touch filename * find a file named ``filename`` in the current directory ``.``:: $ find . -name filename * find a file named ``filename`` in the entire file system ``/``:: $ find / -name filename * find a file named ``filename`` only under your personal directory ``~/``:: $ find ~/ -name filename * find a search keyword ``ams209`` among all files with extension ``.tex``:: $ grep "ams209" *.tex * `sorting `_:: $ sort filename $ sort -r filename * `piping `_, ``&``, ``&&``, and ``||`` operators: `article 1 `_, `article 2 `_:: $ cat filename | head -3 | sort $ cat filename | head -3 | sort -r > sorted_output_list $ cat filename | head -3 | tail -1 >> sorted_output_list $ grep apple < sorted_output_list > foo $ grep apple < sorted_output_list | sort $ false && echo ams209 $ true && echo ams209 $ true || echo ams209 $ false || echo ams209 Let's try the following example to see how ``;`` and ``&&`` work differently:: $ mkdir /backup $ mkdir /backup/old $ touch /backup/old/{foo,foo1,foo2} and see how the following two commands work:: $ cd /backup/old; rm -rf * $ cd /backup/old && rm -rf * * quick summary on redirection, piping, `&` and `&&`:: $ >: save output to a file $ >>: append output to a file $ <: read input from a file $ |: send the output from one program as input to another program $ A; B: Run A and then B, regardless of success of A $ A && B: Run B if A succeeded $ A || B: Run B if A failed $ A &: Run A in background #. Here are some basic commands for managing directories: * create a new directory called ``dirname``:: $ mkdir dirname * change directory, meaning you go to a directory called ``dirname``:: $ cd dirname * tells you where you currently are in the directory tree:: $ pwd * remove empty directory: it will fail if the directory is not empty:: $ rmdir dirname * remove non-empty directory with all the contents therein:: $ rmdir -f dirname * remove non-empty directory as well as all of the sub contents therein:: $ rmdir -rf dirname #. Here are some commands for killing jobs: Sometimes you need to kill a job that's running, perhaps because you realize it's going to run for too long, or you gave it or the wrong input data. Or you may be running a program like the IPython shell and it freezes up on you with no way to get control back. Many programs can be killed with -c. For this to work, the job must be running in the foreground, so you might need to first give the ``fg`` command. Sometimes this doesn't work, like when IPython freezes. Then try stopping it with -z (which should work), find out its PID, and use the ``kill`` command:: $ ps 18917 ttys000 0:00.19 -bash 21181 ttys000 0:00.00 /bin/bash /Users/dongwook/anaconda/bin/python.app /Users/dongwook/anaconda/bin/ipython 21182 ttys000 0:00.19 /Users/dongwook/anaconda/python.app/Contents/MacOS/python /Users/dongwook/anaconda/bin/ipython 18921 ttys001 0:00.01 -bash 18925 ttys002 0:00.02 -bash 20647 ttys003 0:00.01 -bash 20656 ttys004 0:00.01 -bash 21171 ttys005 0:00.01 -bash $ kill 21181 Hit return again you will see:: $ In [1]: Terminated: 15 $ If not, more drastic action is needed with the ``-9`` flag:: $ kill -9 21181 This almost always kills a process. Be careful what you kill. Also try to see more options in using ``kill`` command by typing ``man kill``. #. Setting up environment varialbes (case sensitive!): * `Environment variables `_: There are a couple of choices for Unix/Linux shells: `article 1 `_; `article 2 `_; `article 3 `_; `article 4 `_. One can categorize them into two, *Bourn shell* and *C shell*, where in each category there are a number of variant shells. In this class we use `bash `_ shell as our default choice. Under any circumstances where your default shell might not the ``bash`` shell, you can initiate ``bash`` by typing:: $ bash in a terminal. This will start the bash prompt. In Unix/Linux there are variables called *environment variables* which define various properties that are important in the system. They include various variables, paths and shortcuts which can be set by the system, users including you, or by the shells, or even by some of the programs that are installed or used interactively with some other programs. The following list includes several important environment variables that users often encounter (note that they are all capitalized):: +------------------------+-------------------------------------------------------------------------------------+ | Variables | Description | +------------------------+-------------------------------------------------------------------------------------+ | DISPLAY | Contains the identifier for the display that X11 programs should use by default. | +------------------------+-------------------------------------------------------------------------------------+ | HOME (or ~) | Indicates the home directory of the current user. the default argument for the cd | | | built-in command, that is to say, typing 'cd' will jump to HOME from anywhere. | +------------------------+-------------------------------------------------------------------------------------+ | LD_LIBRARY_PATH | On many Unix systems with a dynamic linker, contains a colon-separated list of | | | directories that the dynamic linker should search for shared objects when | | | building a process image after exec, before searching in any other directories. | +------------------------+-------------------------------------------------------------------------------------+ | PATH | Indicates search path for commands. It is a colon-separated list of | | | directories in which the shell looks for commands. | +------------------------+-------------------------------------------------------------------------------------+ | PWD | Indicates the current working directory as set by the cd command. | +------------------------+-------------------------------------------------------------------------------------+ | USER | Current user name(s) | +------------------------+-------------------------------------------------------------------------------------+ Please see more in `article-bash `_. Environment variables can be used to store information about the environment, and to provide a shorthand for long but useful strings such as absolute paths. They often become essential for defining the computer's behavior when the user compiles programs and builds libraries from the command lines. To see all of the environment variables that are active in your terminal session, use the ``env`` command:: $ env * Customizing enviromental variables in ``.bashrc`` or ``.bash_profile``: Such variables are exported to the system everytime you start a new bash shell, e.g., opening a new terminal (in case bash is your default shell), or logining in to the system, or typing in ``bash``. In these cases, a file named ``.bashrc`` under your home directory is automatically executed. This means that you can always customize your own enviromental variables settings by modifying ``.bashrc`` file. What you can do by modifying the file includes: * any custom execution of program on startup * exporting environmental variables * setting paths * defining aliaes * customizing your prompt, etc. Here is an example of ``.bashrc``:: #----------------------------------- # Print current user names #----------------------------------- u="$USER" echo "user name $u" #----------------------------------- # Export some paths #----------------------------------- export ams209svn="$HOME/Repos/ucsc/soe/teaching/2016-2017/Fall/AMS209" export ams209git="$HOME/Repos/ucsc/soe/teaching/2016-2017/Fall/AMS209_git" export SVN_EDITOR="emacs -nw" export IDL_DIR="/usr/local/itt/idl" export PAPER_DIR="/Users/dongwook/Repos/ucsc/mongchi/DOCS/ucsc/research/papers" export PATH="/usr/local/Cellar/colordiff/1.0.13/bin:/usr/local/bin:$PATH" export PATH="/usr/local/Cellar/valgrind/3.8.1/bin:$PATH" . . . . . . #----------------------------------- # Aliases #----------------------------------- # Bash export LSCOLORS=gxBxhxDxfxhxhxhxhxcxcx # dark background alias lls='ls -laghFG' alias clean='rm *~' # Commom Mac programs alias reload='source ~/.bash_profile' alias sublime='/Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl' alias text='open -a TextEdit' alias pre='open -a Preview' alias grepp='grep -in' alias sshy='ssh -Y' There is another way to achieve the same using a different file called ``.bash_profile``. You can put all of the above in ``.bash_profile`` instead of ``.bashrc``, and customize your settings as you wish. One may ask then what is the difference between the two files. The answer for the Unix/Linux system is that ``.bashrc`` is executed for interactive non-login shells (e.g., opening a new terminal), whereas ``.bash_profile`` is executed for login shells. In the above ``.bashrc`` example, you as a system admin are going to see a list of the current user names logged in to the machine you just logged in, everytime when you open a new shell terminal. Usually you want this information only once when you login to the machine and to keep prompting this information on every new terminal would be unnecessary. To avoid this you can instead add such monitoring/diagnostic tools in ``.bash_profile`` which will only be executed when logins. This difference doesn't exist in Mac OS X as an exception though and ``.bash_profile`` is invoked for each new terminal window instead of ``.bashrc`` on Mac OS X operating systems. In general, you don't want to maintain the two separate files differently for login and non-login shells, especially, you want to set ``PATH`` properly in both shells. A good way of consolidating the two files into one can be done by sourcing ``.bashrc`` from ``.bash_profile``. A good example of ``.bash_profile`` then begins with:: #----------------------------------- # Source global definitions (if any) #----------------------------------- if [ -f ~/.bashrc ]; then source ~/.bashrc fi #----------------------------------- # Print current user names #----------------------------------- u="$USER" echo "user name $u" With this you can put all the paths, custom aliases and common settings only in ``.bashrc``. For more information on the bash shells and enviroment variables please read the two articles and a youtube tutorial: * `article: bash 1 `_ * `article: bash 2 `_ * `youtube: variables `_ * The ``which`` command is useful for finding out the full path to the code that is actually being executed when you type a command. For example, if you have successfully installed *gfortran* and *python* on your computer, you should be able to see:: $ which gfortran /usr/local/bin/gfortran $ whereis gfortran $ $ which python /Users/dongwook/anaconda/bin/python $whereis python /usr/bin/python $ which f77 $ In the latter case it found no program called f77 in the search path, either because it is not installed or because the proper diretory is not on the PATH. Also, we see there is a different outcome in using ``whereis`` as opposed to ``which``. Can you guess what the difference is?