Introduction to MDS software and Bash

Learning outcomes

  1. Recognize the directory hierarchy as it is commonly represented in diagrams, paths, and file explorer software.
  2. Distinguish common operators and representations of the different filesystem elements typically used in Bash.
  3. Explore the filesystem using Bash commands as ls, pwd and cd.
  4. Translate an absolute path into a relative path and vice versa.
  5. Use command-line arguments to produce alternative outputs of commands.
  6. Create, edit, move, and delete files and folders using the command line and Positron.

Platform in focus BASH

Introduction

Welcome to Computer Platforms in Data Science!

This week we are going to learn about exploring the filesystem, using the command line, and some other useful tools.

ActivityActivity

What is a Data Scientist?

A. Someone whose job is to extract value from data through reproducible and auditable processes.

B. Someone whose job is to apply machine learning algorithms.

C. Someone whose job is to analyze large volumes of often unstructured data to produce meaningful insights. https://www.springboard.com/blog/data-science/data-science-definition/

D. All of the above.

E. Someone whose job is to get well paid.

Most of the tools we are going to learn during DSCI 521 are tools that programmers used. Programming involves interacting with the system using commands

The MDS software stack

The following are the core programs that you will be using throughout the MDS program.

  • Bash: Helps you navigate filesystem, run programs, and automate tasks on your computer.
  • Git and GitHub: Used to track changes in your code and collaborate with others on projects.
  • Python and R: Popular programming languages for analyzing data, creating visualizations, performing statistical analysis, and building machine learning models.
  • Visual Studio Code, JupyterLab, RStudio, and Positron: Tools that provide an integraded development environemnt (IDE) that make writing and running code easier. They also help create interactive notebooks and reports.

These programs also have active development communities and are open source, which means that anyone can read the source code and contribute to these projects. Open source programs have many benefits, such as

  • Being able to use the software after leaving school without paying exorbitant amounts of money.
  • Having access to the source code can help you learn and fix bugs (even closed-source commercial software has bugs, but they are just harder to see and fix).
  • Open-source software is customizable and flexible.
  • It is becoming the standard in many tech companies.
  • Being a part of an awesome and active community!

Introduction to computing

Before we get into the different applications, let’s talk about computing. Essentially, computing is about humans communicating with machines to modulate flows of current in hardware. Early examples of human computer communication were quite primitive and included actually disconnecting a wire and connecting it again in a different spot.

Luckily, we are not doing this anymore; instead we have graphical user interface with menus and buttons, which is what you are commonly using on your laptop.

These graphical interface can be thought of as a layer (or shell) around the internal components of your operating system. Shells exist as an intermediate that both makes it easy for us to express our thoughts, and for computers to interpret them.

Text-based communication

We will learn how to communicate to a computer via a text-based shell, rather than a graphical one.

Using a text-based shell might at first seems counter-intuitive, since the reason for creating a shell in the first place was to facilitate user interaction with the computer. So now that we have these easy to use graphical user interfaces (GUIs), why would anyone in their right mind go back to using a text based interface?

Well, it’s partly a misconception, GUIs are nice when you are new to something, but text based interfaces can be faster and easier to use when you know what you are doing, and provide a lot more flexibility.

We can compare it to learning a language, in the beginning it’s nice to look things up in a dictionary (or a menu on the computer), but once I know what I want to say, it is just easer to say or type it directly, instead of looking in submenues.

And by extension, it would be even faster to speak or even just think of what you want to do and have it executed by the computer, this is what speech- and brain-computer interfaces are concerned with.

--- shuffle_answers: true --- ## Which is the better fit for each job? Some categories apply to more than one item. - Try a plot, tweak one parameter, look again :: JupyterLab - Run the same cleaning script on 200 files without opening each one :: Command line - Write a report that mixes prose, code, and figures :: JupyterLab - Move and rename a batch of files :: Command line - Start JupyterLab in the first place :: Command line - Walk through a new dataset one step at a time, keeping every output :: JupyterLab

Bash

bash is the most commonly used text shell. You have it installed on your computer by default if you are using Mac or Linux machine, and if you are on a Windows machine you downloaded bash as part of the setup instructions. For Macs the default shell is zsh, but we had you change it to bash in our setup instructions. Sometimes we might use “prompt”, “command line”, or “terminal”, which for the purposes of this lecture, refers to the same thing.

The abbreviation BASH stands for Bourne Again SHell. Other shells existed before Bash, and one of the most successful early shells was invented by Stephen Bourne at Bell Labs in 1977, which he called the Bourne Shell. In 1989, the Free Software Foundation improved the Bourne Shell and as a pun, named it the Bourne Again Shell to symbolize it was now “reborn” with new features.

Text-based shells are also called command-line interfaces (CLI). The heart of every CLI is a read-evaluate-print loop (REPL). When we type a command and press Return (also called Enter) the CLI reads the command, evaluates it (i.e., executes it), prints the command’s output, and loops around to wait for another command. Let’s see how to do that next!

## Put the steps of one trip around the shell's loop in order You type `ls -F` after the `$` prompt and press Enter. Drag the steps into the order they happen. 1. The shell takes in the line you typed 2. The shell finds the `ls` program and runs it with the `-F` option 3. The program's output is written to the terminal 4. A fresh `$` prompt appears, waiting for the next command

Your first words

The default prompt character:

Terminal
$

Typing in the whoami command for showing your username and pressing Enter to run it:

Terminal
whoami
Output
username

The pwd command stands for Print Working Directory.

Terminal
pwd
Output
# Linux
/home/username

# MacOS
/Users/username

# Windows
/c/Users/username

Our first shell commands will let us explore our folders and files, and will also introduce us to several conventions that most command line tools follow. To start, when bash runs it presents us with a prompt to indicate that it is waiting for us to type something. This prompt is a simple dollar sign by default ($). However, different shells may use a different symbol: for example, the default zsh shell on Macs uses %.

If we run the command whoami, the computer will tell us who we are (our login name), and if we run pwd (Print Working Directory) the shell tells us where we are (the home directory by default when we launch the shell).

Each user has a home directory; the function and location of this directory differs a little bit between operating systems. On Linux it is usually /home/username, on MacOS it is /Users/username, and on Windows it will show up as /c/Users/username (if you have Git Bash installed). Our examples in this module shows the Linux directory structure, but you will see that some of the other modules show what we would see on MacOS or Windows.

Exploring the files system

Now that we know where we are, let’s see what we have using the command ls (short for “listing”), which prints the names of the files and directories in the current directory.

Terminal
ls
Output
Downloads     Music
Documents     todo.txt
Pictures      my_program

Again, our results may be different depending on our operating system and what files or directories we have.

Using commands with options

Terminal
ls -F
Output
Downloads/    Music/
Documents/    todo.txt
Pictures/     my_program*

We can modify output of ls by providing optional “flags”. In this example, the -F option tells ls to decorate the output to show what type of file each entry is. A trailing / indicates a directory, while a trailing * tells us the file is a runnable program. Depending on our setup, the shell might also use colors to indicate whether each entry is a file or directory.

Another common option is -l (“long”), which shows one entry per line together with its size and the date it was last changed:

Terminal
ls -l
Output
drwxr-xr-x  5 username  staff   160 Aug 28 09:12 Documents
drwxr-xr-x  3 username  staff    96 Aug 12 17:40 Downloads
drwxr-xr-x  2 username  staff    64 Jul  3 11:05 Music
drwxr-xr-x  4 username  staff   128 Aug 20 14:31 Pictures
-rwxr-xr-x  1 username  staff  2048 Aug 27 16:02 my_program
-rw-r--r--  1 username  staff   412 Aug 29 08:45 todo.txt

Options can be combined, so ls -l -a (or ls -la) gives the long listing including hidden files.

ActivityActivity

Can you recognize the following folders in the picture?

  1. Home directory
  2. Parent directory
  3. Working directory
  4. Root directory

Working with Files and Directories

We now know how to explore files and directories, but how do we create them? That’s what we will find out in this chapter.

Terminal
cd ~/Documents

# MaKe Directory
mkdir notes

Naming guidelines

Let’s go back to the ~/Documents directory and create a subfolder called notes. For this we first use the ~ home directory shortcut, and then the mkdir command.

We will talk more about useful naming conventions in module 6. For now it is enough if you remember these three guidelines.

  1. Don’t use spaces
  2. Don’t begin the name with a - (hyphen)
  3. Stick to digits and letters (preferably lower case)

Creating a file

Since we just created the notes directory, ls doesn’t display anything when we ask for a listing of its contents.

Terminal
ls -F notes

Let’s change our working directory to notes, then use the VS Code editor to create a file called my-first-note.md. The command to invoke VS Code from the command line is code, and if we pass it a filename as an argument, the file will be created in the current directory. Now try launching it yourself!

Terminal
cd notes
code my-first-note.md

The .md syntax means that we want to create a markdown file. File extensions like .md and .py don’t change anything about the content of the file, but they are an indicator to other program such as VS Code what type of content we are going to put inside the file. These program can then activate special functions for certain files, such as using the appropriate colors to highlight headings and functions.

Let’s move on to the next slide to see how it looks when we are editing file in VS Code.

Editing a file in VS Code

This is what it looks like to edit a text file in VS Code. You will see that there is a small white circle next to the file name. This is an indication that the file is not saved yet.

When we are done adding our edits, we can hit Ctrl + s (Windows/Linux) ⌘ + s (Mac) to save the file, which will make the white circle disappear. After saving the file, you can optionally close VS Code before returning to the terminal.

We are only using VS Code as a simple text editor here, but it is worth knowing that you can use use it as a powerful editor for code as well.

Viewing the content of a file in the terminal

If we now type ls we will see the name of the file we just created in VS Code.

Terminal
ls
Output
my-first-note.md

To see the content of the file, we can use the head command. head show the first few lines of a file (just as when we use df.head() in pandas), and optionally takes an argument for how many lines to show: head -n 10 (the default is five).

Terminal
head my-first-note.md
Output
These are the first
few lines of content
in the file.

head works with all plain text files, such as code and markdown files, your results will vary with binary files such as word documents, spreadsheets, or images. Generally, binary file output will be incomprehensible.

To see the last few lines, you can use tail instead of head.

Moving files and directories

To move a file around, we can use the mv command. For example, to move the text file we just created to the parent directory, we would type:

Terminal
mv my-first-note.md ..
ls ..
Output
Downloads/    Music/        my-first-note.md
Documents/    todo.txt
Pictures/     my_program*

This moves it to the parent directory, and keeps the name the same as we can see if we do ls ..

If we want to move the file back to the current directory, we can use the . shortcut:

Terminal
mv ../my-fist-note.md .

Renaming a file

Renaming a file can be thought of as moving it to a new filename either in the same or a different directory. This might initially seem a bit different from a graphical file browser, where renaming and moving are two distinct functions, but the underlying operation is actually the same.

To give our file a more descriptive name, we can use mv to include the date we created this file,

Terminal
mv my-fist-note.md notte_2022-06-20.md # intensional typo

We intentionally made a typo here, let’s fix it while showing off the functionality of the -v (verbose) option, which prints what was moved/renamed so that it is easier to keep track of.

Terminal
mv -v notte_2022-06-20.md note_2022-06-20.md
Output
renamed 'notte_2022-06-20.md' -> 'note_2022-06-20.md'

We must be careful when specifying the destination because mv will overwrite existing files without warning. The option -i (“interactive”) makes mv ask us for confirmation before overwriting,. mv also works on directories where it renames the directory without changing its contents.

Copying files and directories

The cp command create a copy of a file. Just as with mv, we optionally rename the file as we are copying it.

Here, we first copy the note we created to the parent directory and then show how we could use cp to create a backup of our file in the same directory. Remember that the file extension does not change the content of the file, so this is only a visual indicator for the file type, and helps the computer pick a default program when you open it.

Terminal
# Copy the file to the parent directory
cp note_2022-06-20.md ..
Terminal
# Create a backup copy in the same directory
cp note_2022-06-20.md note_2022-06-20.md.bkp
ls
Output
note_2022-06-20.md
note_2022-06-20.md.bkp

If we ever wanted to copy a directory, we would have to specify the -r flag, which indicates that we want to copy the directory recursively, i.e. also making a copy of all the directory contents.

Removing files and directories

To delete a file we use rm (remove). Let’s remove the backup copy we just made:

Terminal
rm note_2022-06-20.md.bkp
ls
Output
note_2022-06-20.md

There is no trash can on the command line. rm does not ask for confirmation, and there is no undo. The -i (“interactive”) option we saw with mv works here too, and makes rm ask before each deletion:

Terminal
rm -i note_2022-06-20.md
Output
rm: remove regular file 'note_2022-06-20.md'? n

Answering n keeps the file.

Just like cp, rm refuses to act on a directory unless we pass -r to make it recursive:

Terminal
rm notes
Output
rm: cannot remove 'notes': Is a directory

If we really wanted to remove the whole notes directory we would go up one level and run rm -r notes, which deletes the directory and everything inside it, with no confirmation and no way back. Treat rm -r with respect, and double check the path before you press Enter.

--- shuffle_answers: true --- ## What happens when you run each command? (Not all outcomes will be used.) Your working directory contains the file `note.md` and the directory `notes/`, which has files in it. - `rm note.md` :: Deletes the file right away, no confirmation - `rm -i note.md` :: Asks before deleting the file - `rm notes` :: Refuses, saying it is a directory - `rm -r notes` :: Deletes the directory and everything inside it - :: Moves the file to the Trash - :: Deletes the directory, but only if it is empty
## Put the commands in the order that makes them work You open a terminal, which starts in your home directory. Here is what you want to do:
— change into your `Documents` folder
— make a new folder there called `notes`
— step into `notes`
— create a file `my-first-note.md` in it, using VS Code
— move that file up one level, into `Documents`
— show the first lines of the moved file, from where you are standing

Every command depends on the one before it. 1. `cd ~/Documents` 2. `mkdir notes` 3. `cd notes` 4. `code my-first-note.md` 5. `mv my-first-note.md ..` 6. `head ../my-first-note.md`

Wildcards

Wildcards (also called “globbing”) simplifies targeting multiple files with similar names in the same command. The most commonly used wildcard is * (a single asterisk). It matches zero or more characters, so typing ls *.md list all of the markdown files in the current directory.

If we wanted to list all the files created in July (month 7), we would need to type ls ls *07*, which means that the filename can include anything before and after 07. If we would have left out the second * and type ls *07 we would not have seen any matches because there is not file that end in 07, they all have some characters after.

Using wildcards is helpful when we want to delete, move, or copy files with a predictable naming pattern.

Terminal
touch note_2022-06-20.md.bkp note_2022-07-02.md note_2022-07-02.md.bkp
ls *.bkp
Output
note_2022-06-20.md.bkp
note_2022-07-02.md.bkp
Terminal
ls *07*
Output
note_2022-07-02.md
note_2022-07-02.md.bkp

Searching for text patterns in files

We can use the grep command to search for text in files. To search for the word “the” in our notes file, we can type grep "the" note_2022-06-22.md. This will return every line in the file that contains the word “the”.

Terminal
grep "the" note_2022-06-22.md
Output
note_2022-06-22.md:These are the first
note_2022-06-22.md:in the file.

Seeing the history of commands

Occasionally, we might want to re-use a command that we used in the past, but don’t remember exactly what it was. The history command can help us with this as it lists all the previous commands we have run and in which order we ran them.

Terminal
history
Output
1  pwd
2  ls
3  cd Documents/
4  cd
5  ls -F -a
6  history

Reading the manual

How can we find out what options like -t and -r do if we don’t know of them already? By reading the built-in help manual! Typing man ls brings up the manual help page for the ls command. For some systems, you may need to run --help instead of man, e.g., ls --help.

Terminal
man ls
Output
LS(1)                                                    User Commands

NAME
       ls - list directory contents

SYNOPSIS
       ls [OPTION]... [FILE]...

DESCRIPTION
       List  information  about the FILEs (the current directory by default).  Sort entries alphabetically if none of -cftuvSUX
       nor --sort is specified.

       Mandatory arguments to long options are mandatory for short options too.

       -a, --all
              do not ignore entries starting with .

       -A, --almost-all
              do not list implied . and ..

       --author
              with -l, print the author of each file

       -b, --escape
              print C-style escapes for nongraphic characters

       --block-size=SIZE
              with -l, scale sizes by SIZE when printing them; e.g., '--block-size=M'; see SIZE format below

       -B, --ignore-backups
              do not list implied entries ending with ~
...

Some helpful commands while navigating the manual pages in less

  • q: Quits the manual and takes you back to the shell prompt
  • Space: Go down the page
  • b: Go Back up the page
  • Up / Down: you can also use the arrow keys to go up and down the page
  • /: starts a search
    • Type in the search term and press Enter to go to the first hit
    • n: To continue to the next search hit
    • N (Shift + n): Go to the previous hit
Note

If you notice a : in the bottom left of your terminal, you are browsing the file in a program called less. To quit, you can press the letter q on your keyboard.

You can enter less in multiple ways when working in the terminal, it is not exclusive to only looking at the man pages.

--- shuffle_answers: true --- ## Pick the option that does the job. (Not all options will be used.) Each situation below needs one of the options introduced in this chapter. - Copy a directory *together with everything inside it* :: `-r` - Make `mv` ask before it overwrites a file that already exists :: `-i` - See the files whose names start with a dot :: `-a` - Tell directories and runnable programs apart in a listing :: `-F` - Print exactly what `mv` renamed :: `-v` - See each entry's size and when it was last changed :: `-l` - :: `--help` - :: `-n`

Relative vs absolute path

We’ve seen a bit of usage around . and ... Let’s talk a bit more on how to reference files and directories.

Let’s say you are here:

Terminal
pwd
Output
/Users/timberst/Documents

These are the files in your current working directory:

Terminal
ls
Output
DSCI-100  MDS-2018-19   textbooks   research

You can navigate to the research directory using one of two options, a relative path:

Terminal
cd research

or, using an absolute path:

Terminal
cd /Users/timberst/Documents/research

What is the difference?

When you use a relative path with a command like ls or cd, it tries to find that location from where we are, rather than from the root of the file system. However, when you specify the absolute path to a directory by including its entire path from the root directory, which is indicated by a leading slash, the leading / tells the computer to follow the path from the root of the file system, so it always refers to exactly one directory, no matter where we are when we run the command.

--- shuffle_answers: true --- ## Absolute or relative? The same idea shows up outside the filesystem. An **absolute** reference works no matter where you are. A **relative** reference only makes sense from a particular starting point. Sort each one. Some categories apply to more than one item. - `822 2211` :: Relative - `+1 604 822 2211` :: Absolute - Walk past the Nest, turn right at the bus loop, second building on your left :: Relative - `49.2606° N, 123.2460° W` :: Absolute - `2366 Main Mall, Vancouver, BC V6T 1Z4` :: Absolute - In twenty minutes :: Relative - 14:30 PT on 2026-09-01 :: Absolute - Two floors up from here :: Relative - :: A nickname
--- shuffle_answers: true --- ## Where does each command leave you? Your working directory is `/home/username/Documents/notes`. Drag each command to the directory it lands you in. Some directories are used more than once. - `cd ..` :: `/home/username/Documents` - `cd ~` :: `/home/username` - `cd .` :: `/home/username/Documents/notes` - `cd /` :: `/` - `cd ~/Documents` :: `/home/username/Documents` - `cd ../..` :: `/home/username` - `cd ../../../..` :: `/` - `cd /home/username/Documents/notes` :: `/home/username/Documents/notes` - :: `/home`

References

  • https://merely-useful.tech/py-rse/bash-tools.html
  • https://missing.csail.mit.edu/
  • https://datascienceatthecommandline.com/2e/chapter-1-introduction.html
  • https://datascienceatthecommandline.com/resources/50-reasons.pdf