Working with SOEP Data in Open Data Format

The SOEP offers data in the new open data format (opendf). This format includes metadata accessible in Stata and R, making it particularly useful for R-users who want to load SOEP-core datasets with metadata directly in R.

This guide demonstrates how to open and work with SOEP data in opendf format using R or Stata.

The Open Data Format

The open data format (opendf) is a metadata-enriched, non-proprietary, and platform-independent data format. It includes metadata in multiple languages stored in DDI Codebook Format. The format is specified as a zip-compressed folder containing a CSV file with the data and an XML file with the metadata. To import and export data in the opendf format, the SOEP provides packages for R and Stata.

Using the opendf in R

Installing the Opendataformat Package in R

To get started with SOEP data in the open data format you need to download the opendataformat package using the install_git()-function from the devtools-library:

#install.packages("devtools")
library(devtools)
devtools::install_git("https://git.soep.de/opendata/r-package-opendataformat.git")
library("opendataformat")

Read a Data File

Now you can use the package functions to work with opendf data files. For demonstration purposes, let’s load the sample data included in the opendataformat-package. To open the documentation/help-files for the read_opendf() function you can execute ?read_opendf.

path <- system.file("extdata", "data.zip", package="opendataformat")
?read_opendf
df <- read_opendf(file = path)

You can specify further parameters of the read_opendf()-function: The languages in which metadata should be imported, the (maximum) number of rows you want to import, the number of rows you want to skip (excluding the header), and the variables/columns you want to import (indices or column names). In the following code line, you see the default parameters:

df <- read_opendf(file = path, languages = "all", nrows = Inf, skip = 0, variables = NULL)

Display and Use Metadata

To display metadata you can use the docu_opendf()-function. With style=”print” or style=”console” the metadata is displayed in the console. Alternatively, you can display the metadata in the viewer with style=”viewer” or style=”html”. Default is style=”both”.

#To display dataset information
docu_opendf(df, style="print")

You can additionally display a list of the variables in the dataset:

#You can additionally display a list of the variables in the dataset
docu_opendf(df, variables="yes")

Or information on a specific variable:

#Variable information
docu_opendf(df$bap87)

You can also choose the language of the metadata displayed, if the information is available in a particular language, by setting the languages parameter to a particular language (languages=”de”) or display metadata in all available languages (languages=”all”). Default is the current language (which is normally English). You can set the default language of a dataframe in opendf-format using the setLanguage_opendf()-function (eg. df<-setLanguage_opendf(df, “de”)).

#You can also choose the language of the metadata (when the language is available).
docu_opendf(df$bap87, languages="de")
#To display the metadata in all available languages, set languages="all"
docu_opendf(df$bap87, languages="all")
#You can also set the default language of a dataset
df<-setLanguage_opendf(df, "de")
#Then variable information will by default be displayed in the respective language:
docu_opendf(df$bap87)

The metadata is stored in the attributes of the opendf-dataframe object and in the attributes of the columns of the opendf-dataframe object. To retrieve attributes you can use the R base functions attr() and attributes().

# display all attributes of a dataframe
attributes(df)
# display all attributes of a variable/column
attributes(df$bap87)

# You can also display a specific attribute of a dataframe:
attributes(df)$label_de
attr(df, "label_de")

# Or of a variable
attributes(df$bap87)$labels_de
attr(df$bap87, "labels_de")

Alternatively, you can use the labels_opendf()-function from the opendataformat-package to retrieve labels and other metadata.:

# display the labels of all variables in a dataframe (in the active/current language)
labels_opendf(df)
#in a specific language
labels_opendf(df, language="de")

# To see which languages are available for the dataframe, set retrieve=languages
labels_opendf(df, retrieve="languages")

#you can also display the value labels of a specific variable:
labels_opendf(df$bap87, valuelabels=T)
# alternative:
labels_opendf(df$bap87, retrieve="valuelabels")

# You can also display the descriptions, the URLs or the variable types :
# Descriptions of all variables:
labels_opendf(df, retrieve="description")
# of one variables:
labels_opendf(df$bap87, retrieve="description")

# URLs:
labels_opendf(df, retrieve="url")

# variable types:
labels_opendf(df, retrieve="type")

Write an opendf-dataframe to a Data-File

To save a dataframe as an opendf-file again, the opendataformat package provides the write_opendf()-function. All the metadata stored in the attributes, that is compatible with the opendf-specification is preserved in the opendf-file.

The compatible metadata for the dataframe includes the dataset name in the name-attribute, labels with a language tag (eg. label_en), descriptions with a language tag (eg. description_en), and a URL in the url-attribute. The compatible metadata for the variables/columns includes labels with a language tag (eg. label_en), descriptions with a language tag (eg. description_en), the variable type in the type-attribute, value labels with a language tag (eg. labels_en), and a URL in the url-attribute.

You can choose to save metadata in one or several specific languages only using the languages-argument. Default is language=”all”.

# Write the dataframe `df` to the opendf-file `my_datafile.zip` in the current working directory.
write_opendf(x=df, file="my_datafile.zip")
# Write the dataframe `df` while keeping only metadata in English.
write_opendf(x=df, file="my_datafile.zip", languages="en")

library(devtools)
devtools::install_git("https://git.soep.de/thartl/r-package-opendataformat.git")
library("opendataformat")

For further instructions on how to work with the opendf-format in R, read the vignette (PDF, HTML) for the opendataformat-package.

Using the opendf in Stata

Installing the Opendataformat Package

To work with SOEP-Data in the open data format in Stata you have to install the opendataformat package in Stata using net install to download the package files from the GitHub repository. You can find the instructions on the GitHub page.

********** Download and Install opendataformat package: *********
net install opendf, from (https://thartl-diw.github.io/opendf/) replace

Python Integration

For the opendataformat package to work, the Python integration in Stata is required. Therefore, you need Stata version 16 or higher and some Python installation on your computer. You can easily test if Python is working within Stata. If you have problems with the Python integration, you can copy a Python version to your computer using opendf installpython function from the opendataformat package:

********** opendf installpython and opendf removepython **********

* For the opendf read package to work you need a working python integration 
* in Stata. Therefore you need a python versio  on you Computer and Stata 
* must be able to find it.
* You can test if python works:
python
print("Hello World")
end

* If you have problems with your python integration in Stata, you can fix it 
* easily (for the opendataformat package). The opendataformat package offers 
* a function to copy a python version to you computer ito a location where the 
* opendataformat package will find it.
* The function by default downloads python 3.12 to the Stata ado folder where  
* the package files are stored:
opendf installpython

* Alternatively you can specify path and version
* If you choose another location, the opendataformat package will not 
* automatically find the python.exe. You have to indicate the location each 
* time you start Stata.
*opendf installpython, version("3.12") location("C:/.../")

* You can remove the python version(s) from the default folder or a folder 
* you define:
opendf removepython
*opendf removepython, version("3.12") location("C:/.../")

* For further instructions see the help files:
help opendf installpython
help opendf removepython

Read a Data File

Now you can use the package functions to work with opendf-data files. For demonstration purposes, we copy the testdataset (testdata.zip) from GitHub to the local temp-directory:

* Download testdata (testdata.zip) to Temp-Directory
copy "https://raw.githubusercontent.com/thartl-diw/opendf/main/Testdata/testdata.zip" `c(tmpdir)', replace

With the opendf read function you can load the data file to Stata:

********** opendf read **********

* To read a opendf data file you can use the opendf read function.
* Here the tastdata.zip-file is loaded from Temporary Directory.
opendf read "`c(tmpdir)'testdata.zip"

You can specify further parameters of the opendf read-function: The range of rows you want to load (excluding the header), the range of columns you want to load, and whether you want to save the dataset directly as .dta-file.

********** opendf read **********

* To read a opendf data file you can use the opendf read function.
* Here the tastdata.zip-file is loaded from Temporary Directory.
opendf read "`c(tmpdir)'testdata.zip"

* You can also specify the rows and colums to load using the rowrange and 
* colrange options. Here the first ten rows and columns 2-5 are loaded.
opendf read "`c(tmpdir)'testdata.zip",  rowr(:10) colr(2:5) save("testdata.dta") replace clear

* For more information look into the help-file
help opendf read

Display and Use Metadata

To display metadata you can use the opendf docu-function for the dataset information:

* You can display metadata/information using the opendf docu function.
* Display Metadata for Dataset:
opendf docu

Or information on a specific variable:

* Display Metadata for Variable
opendf docu bap87

You can also choose the languages for the metadata displayed, if the information is available in particular languages, by setting the languages parameter to these languages (e.g. languages(“en”)) or to display metadata for all languages (languages(“all”)). Default is the active label language. You can set the active language using the label language-command.

*To switch to another label/metadata language use the label language function
label language de

*Alternatively you can use the languages option of opendf docu
opendf docu, languages("en")
opendf docu, languages("all")

The metadata is stored in the characteristics of the dataset. To retrieve them you can use the Stata base functions:

*Display characteristics directly
char list
local _url: char _dta[url]
di "`_url'"
local _description_bap87: char bap87[description_de]
di "`_description_bap87'"

Write an opendf-dataframe to a Data-File

To save a dataframe as an opendf-file, the opendataformat package provides the opendf write-function. All the metadata stored in the attributes, that is compatible with the opendf-specification is preserved in the opendf-file.

The compatible metadata for the dataframe includes the dataset name in the dataset-attribute, labels with a language tag (eg. label_en), descriptions with a language tag (eg. description_en), and a URL in the url-attribute. The compatible metadata for the variables/columns includes labels with a language tag (eg. label_en), descriptions with a language tag (eg. description_en), the variable type in the type-attribute, value labels with a language tag (eg. labels_en), and a URL in the url-attribute.

You can also specify the columns/variables to save and the languages of the metadata. By default, the metadata is saved in all available languages (languages(“all”))

********** opendf write **********

* To save the dataset to a opendf-file you can use the opendf write function.
* To simply write it to the working directory, just enter the file-name:
* If the data file already exists and you want to replace it, 
* you need the replace option.
opendf write "testdata.zip", replace

* To save only specific variables, you can use the variables option:
opendf write "testdata.zip", variables(bap9201 bap9001 bap87) replace

* If you want to save the metadata of only one language or particular languages,
* you can use the languages option:
opendf write "testdata.zip", languages("de en") replace
opendf write "testdata.zip", languages("de") replace

* For more information look into the help-file
help opendf write

Last change: Jun 06, 2024