A Data Carpentry Workshop
CSC - IT Center for Science
Espoo, Finland
Mar 18-19, 2015
9:00 am - 5:00 pm
Instructors: François Michonneau, Aleksandra Pawlik
Helpers: Mateusz Kuzak, Philip Lijnzaad, Diana Marek, Damiano Piovesan, Andrea Manconi, Ari-Matti Saren, Gonçalo Antunes
Workshop Information
This Data Carpentry workshop is aimed at researchers in the life sciences at all career stages. Data Carpentry is designed to teach basic concepts, skills and tools for working more effectively with data.
The workshop will cover the following topics:
- How to use spreadsheet programs (such as Excel) more effectively, and the limitations of such programs.
- Getting data out of spreadsheets and into more powerful tools (R).
- Using databases, including managing and querying data in SQL.
- Workflows and automating repetitive tasks, in particular using the command line shell.
We especially encourage to register those who may be less familiar with the above topics. There is no prerequisite as to what computing skills and knowledge is required.
Acknowledgements & Support
The workshop is supported by the ELIXIR programme.
Data Carpentry is a partnership of several NSF-funded BIO Centers (NESCent, iPlant, iDigBio, BEACON and SESYNC) and Software Carpentry, and is sponsored by the Data Observation Network for Earth (DataONE). The structure and objectives of the curriculum as well as the teaching style are informed by Software Carpentry.
Schedule
| Wednesday | 09:00 - 9:30 | Introduction |
| 09:30 - 10:30 | Better use of spreadsheets - part 1 | |
| 10:30 - 11:00 | Break | |
| 11:00 - 12:00 | Better use of spreadsheets - part 2 | |
| 12:00 - 13:00 | Lunch break | |
| 13:00 - 14:30 | Introduction to R - part 1 | |
| 14:30 - 15:00 | Coffee break | |
| 15:00 - 16:45 | Introduction to R - part 2 | |
| 16:45 - 17:00 | Wrap-up | |
| Thursday | 09:00 - 11:00 | Working with command line - part 1 |
| 11:00 - 11:30 | Break | |
| 11:30 - 13:00 | Working with command line - part 2 | |
| 13:00 - 14:00 | Lunch | |
| 14:00 - 15:00 | Introduction to data manipulation with SQL - part 1 | |
| 15:00 - 15:30 | Break | |
| 15:30 - 16:45 | Introduction to data manipulation with SQL- part 2 | |
| 16:45 - 17:00 | Wrap-up and feedback |
Venue
CSC teaching room on the ground floor (room 1372).
Syllabus
Better use of spreadsheets
- Formatting data tables in spreadsheets.
- Common formatting mistakes by spreadsheet users.
- Dates as data.
- Basic quality control and data manipulation in spreadsheets.
- Exporting data from spreadsheets.
- Caveats of popular data and file formats.
- Lessons
The Unix Shell
- Files and directories:
pwd,cd,ls,mkdir, ... - History and tab completion
- Pipes and redirection
- Looping over files
- Creating and running shell scripts
- Finding things:
grep,find, ...
Lessons
- Lesson 1: Introduction
- Lesson 2: Files and Directories
- Lesson 3: Creating things
- Lesson 4: Pipes and Filters
- Lesson 5: Loops
- Lesson 6: Shell scripts
- Lesson 7: Finding things
Introduction and data manipulation with R
- Introduction to R
- Reading and writing .csv
- Creating Functions
- Looping over files
- Analyzing Multiple Data Sets
- Making Choices - conditionals
Introduction to working with data in SQL
- Relational databases.
- Working with SQLite
- Basic queries
- Filtering
- Building more complex queries
Lessons
Setup
To participate in a Data Carpentry workshop, you will need working copies of the software described below. Please make sure to install everything (or at least to download the installers) before the start of your workshop.
Overview
Editor
When you're writing scripts or text, it's nice to have a text editor that is optimized for writing code, with features like automatic color-coding of key words. The default text editor on Mac OS X and Linux is usually set to Vim, which is not famous for being intuitive. if you accidentally find yourself stuck in it, try typing the escape key, followed by ':wq!' (colon, lower-case 'q', exclamation mark), then hitting Return to return to the shell. (This will exit Vim and save the changes you made in the file.)
The Bash Shell
Bash is a commonly-used shell. Using a shell gives you more power to do more tasks more quickly with your computer.
R
R is a programming language that specializes in statistical computing. It is a powerful tool for exploratory data analysis. To interact with R, we will use RStudio, an interactive development environment (IDE).
SQL
SQL is a specialized programming language used with databases. We use a simple database manager called SQLite, either directly or through a browser plugin.
Spreadsheet
We'll be using a spreadsheet program. If you already have a spreadsheet program installed, like LibreOffice, Excel or OpenOffice, you can use whatever you already have. If you don't have a spreadsheet program, please download and install LibreOffice.Windows
Editor
Notepad++ is a popular free code editor for Windows. Be aware that you must add its installation directory to your system path in order to launch it from the command line (or have other tools like Git launch it for you). The instructions to modify your path are available online here. Please ask your instructor to help you do this.
Git Bash
Install Git (version control) and a Bash shell for Windows from the msysGit project's homepage. This will provide you with Bash in the Git Bash program.
R
Install R by downloading and running this .exe file from CRAN. Also, please install the RStudio IDE.
SQLite
Install the Firefox SQLite browser plugin described below.
Software Carpentry Installer
Other tools used in Data Carpentry have been packaged up by Software Carpentry in an installer. This installer requires an active internet connection.
- Download the installer.
- Double click on the file to run it.
- A command window will open and then close again. The installer downloads and installs a couple of command line programs that we'll need for course.
- If you saw it open and then close again, you're set!
Mac OS X
Bash
The default shell in all versions of Mac OS X is bash,
so no need to install anything. You access bash from
the Terminal (found
in /Applications/Utilities). You may want
to keep Terminal in your dock for this workshop.
Editor
We recommend
Text Wrangler or
Sublime Text.
Alternatively, you can use nano,
which should be pre-installed.
R
Install R by downloading and running this .pkg file from CRAN. Also, please install the RStudio IDE.
SQLite
sqlite3 comes pre-installed on Mac OS X.
Also install the Firefox SQLite browser plugin described below.
Linux
Bash
The default shell is usually bash,
but if your machine is set up differently
you can run it by opening a terminal and typing bash.
There is no need to install anything.
Editor
Kate is one option for Linux users.
Alternatively, you can use nano,
which should be pre-installed.
R
You can download the binary files for your distribution
from CRAN. Or
you can use your package manager, e.g. for Debian/Ubuntu
run apt-get install r-base. Also, please install
the
RStudio IDE.
SQLite
sqlite3 comes pre-installed on Linux.
Also install the Firefox SQLite browser plugin described below.
Extras - everyone needs to install these
Firefox SQLite Plugin
Instead of using sqlite3 from the command line,
you will use this plugin
for Firefox instead. If you don't already have firefox, install it first.
To install the sqlite plugin:
- Start Firefox.
- Go to the plugin homepage.
- Click the "Add Now" button.
- Click "Install Now" on the dialog that appears after the download completes.
- Restart Firefox when prompted.
- To check that it was installed correctly, select "SQLite Manager" from the "Tools" menu.