Export to pandas DataFrame#

NOTE: This feature is available only if using a version of pyOpenMS >= 3.0, at the time of writing this means using the nightly builds as described in the Installation Instructions.

In pyOpenMS some data structures can be converted to a tabular format as a pandas.DataFrame. This allows convenient access to data and meta values of spectra, features and identifications. The same data structures also convert to Apache Arrow tables; see Arrow and Parquet.

Required imports for the examples:

1import pyopenms as oms
2import pandas as pd
3from urllib.request import urlretrieve
4
5url = "https://raw.githubusercontent.com/OpenMS/OpenMS/develop/doc/pyopenms/src/data/"

MSExperiment#

pyopenms.MSExperiment.to_df( columns=None, ms_levels=None, long_format=False )

Generates a pandas DataFrame with all peaks in the MSExperiment

Parameters:

columns : default None

columns to export; None exports all of them

ms_levels : default None

MS levels to export, for example [1]; None exports all of them

long_format : default False

set to True if you want to have a long/expanded/melted dataframe with one row per peak. Faster but replicated RT information. If False, returns rows in the style: rt, ms_level, np.array(mz), np.array(intensity)

Returns:

pandas.DataFrame

peak map information stored in a DataFrame

Examples:

1urlretrieve(url + "BSA1.mzML", "BSA1.mzML")
2exp = oms.MSExperiment()
3oms.MzMLFile().load("BSA1.mzML", exp)
4
5df = exp.to_df()  # default: long_format = False
6df.head(2)
exp.to_df()#

rt

ms_level

mz_array

intensity_array

0

1501.41394

1

[300.0897645621494, 300.18132740129533, 300.20…

[3431.0261, 1181.809, 1516.1746, 1719.8547, 11…

1

1503.03125

1

[300.06577092599525, 300.08932376441896, 300.2…

[914.79034, 1842.2311, 2395.1025, 851.4738, 16…

 
1df = exp.to_df(long_format=True)
2df.head(2)
exp.to_df(long_format=True)#

rt

mz

intensity

ms_level

0

1501.41394

300.089765

3431.026123

1

1

1501.41394

300.181327

1181.808960

1

PeptideIdentification#

pyopenms.PeptideIdentificationList.to_df( decode_ontology=True, default_missing_values=None, export_unidentified=True, columns=None )

Generates a pandas DataFrame with the peptide identifications in the list

Parameters:

decode_ontology : default True

if meta values contain CV identifer (e.g., from PSI-MS) they will be automatically decoded into the human readable CV term name.

default_missing_values : default None

default value for missing values for each data type; None uses {bool: False, int: -9999, float: np.nan, str: ‘’}

export_unidentified : default True

export PeptideIdentifications without PeptideHit

columns : default None

columns to export; None exports all of them

Returns:

pandas.DataFrame

peptide identifications in a DataFrame

Example:

1urlretrieve(url + "small.idXML", "small.idXML")
2prot_ids = []
3pep_ids = oms.PeptideIdentificationList()
4oms.IdXMLFile().load("small.idXML", prot_ids, pep_ids)
5
6df = pep_ids.to_df()
7df.head(2)
pep_ids.to_df()#

id

rt

mz

q-value

charge

protein_accession

start

end

NuXL:z2 mass

NuXL:z3 mass

…

isotope_error

NuXL:peptide_mass_z0

NuXL:XL_U

NuXL:sequence_score

0

OpenNuXL_2019-12-04T16:39:43_1021782429466859437

900.425415

414.730865

0.368649

4

DECOY_sp|Q86UQ0|ZN589_HUMAN

255

267

828.458069

552.641113

…

0

1654.901611

0

0.173912

1

OpenNuXL_2019-12-04T16:39:43_7293634134684008928

903.565186

506.259521

0.422779

2

sp|P61313|RL15_HUMAN

179

187

0.0

0.0

…

0

1010.504639

0

0.290786

FeatureMap#

pyopenms.FeatureMap.to_df( columns = None, meta_values = None, export_peptide_identifications = True )

Generates a pandas DataFrame with information contained in the FeatureMap, indexed by feature_id.

Optionally the feature meta values and information for the assigned PeptideHit can be exported.

Parameters:

columns : default None

columns to export; None exports all of them

meta_values : default None

meta values to include (None, [custom list of meta value names] or ‘all’)

export_peptide_identifications (bool): default True

Export sequence and score for best PeptideHit assigned to a feature. Additionally the ID_filename (file name of the corresponding ProteinIdentification) and the ID_native_id (spectrum ID of the corresponding Feature) are exported. To export all assigned PeptideIdentifications and merge them with this DataFrame, use FeatureMap.to_peptide_df(), described below.

Returns:

pandas.DataFrame

feature information stored in a DataFrame

Examples:

1urlretrieve(url + "BSA1_F1_idmapped.featureXML", "BSA1_F1_idmapped.featureXML")
2feature_map = oms.FeatureMap()
3oms.FeatureXMLFile().load("BSA1_F1_idmapped.featureXML", feature_map)
4
5df = feature_map.to_df()  # default: meta_values = None
6df.head(2)
feature_map.to_df()#

feature_id

peptide_sequence

peptide_score

ID_filename

ID_native_id

charge

rt

mz

rt_start

rt_end

mz_start

mz_end

quality

intensity

9650885788371886430

LVTDLTK

0.000000

None

spectrum=1270

2

1942.600083

395.239277

1932.484009

1950.834351

395.239199

397.245758

0.808494

157572000.0

18416216708636999474

DDSPDLPK

0.034483

None

spectrum=1167

2

1749.138335

443.711224

1735.693115

1763.343506

443.711122

445.717531

0.893553

54069300.0

 
1df = feature_map.to_df(meta_values="all", export_peptide_identifications=False)
2df.head(2)
feature_map.to_df(meta_values = ‘all’, export_peptide_identifications = False)#

feature_id

charge

rt

mz

rt_start

rt_end

mz_start

mz_end

quality

intensity

FWHM

spectrum_index

spectrum_native_id

label

score_correlation

score_fit

9650885788371886430

2

1942.600083

395.239277

1932.484009

1950.834351

395.239199

397.245758

0.808494

157572000.0

10.061090

259

spectrum=1270

168

0.989969

0.660286

18416216708636999474

2

1749.138335

443.711224

1735.693115

1763.343506

443.711122

445.717531

0.893553

54069300.0

14.156094

156

spectrum=1167

169

0.999002

0.799234

 
1df = feature_map.to_df(meta_values=["FWHM", "label"])
2df.head(2)
feature_map.to_df(meta_values = [‘FWHM’, ‘label’])#

feature_id

peptide_sequence

peptide_score

ID_filename

ID_native_id

charge

rt

mz

rt_start

rt_end

mz_start

mz_end

quality

intensity

FWHM

label

9650885788371886430

LVTDLTK

0.000000

None

spectrum=1270

2

1942.600083

395.239277

1932.484009

1950.834351

395.239199

397.245758

0.808494

157572000.0

10.061090

168

18416216708636999474

DDSPDLPK

0.034483

None

spectrum=1167

2

1749.138335

443.711224

1735.693115

1763.343506

443.711122

445.717531

0.893553

54069300.0

14.156094

169

Extract assigned peptide identifications from a feature map

Peptide identifications can be mapped to their corresponding features in a FeatureMap. pyopenms.FeatureMap.get_assigned_peptide_identifications() returns them as a list of PeptideIdentification objects, and pyopenms.FeatureMap.to_peptide_df() exports them to a DataFrame that links each one to its feature.

pyopenms.FeatureMap.get_assigned_peptide_identifications()

Generates a list with the peptide identifications assigned to the features, as the features store them. The FeatureMap itself is not changed.

Returns:

PeptideIdentificationList

list of PeptideIdentification objects

pyopenms.FeatureMap.to_peptide_df( decode_ontology = True, default_missing_values = None, export_unidentified = True, columns = None )

Generates a pandas DataFrame with one row per assigned PeptideIdentification: the columns of PeptideIdentificationList.to_df(), preceded by feature_id, the unique ID of the feature the identification is assigned to. It is the same unsigned 64-bit number that indexes FeatureMap.to_df(). P_ID is the identification’s position in get_assigned_peptide_identifications().

Parameters:

as for PeptideIdentificationList.to_df(); columns always keeps feature_id

Returns:

pandas.DataFrame

peptide identifications with the feature_id of each

Feature map and peptide data frames can be merged on the feature_id column to contain the complete information for peptides and features in a single data frame. export_unidentified=False leaves out PeptideIdentifications without hits.

Example:

1merged_df = pd.merge(
2    feature_map.to_df().reset_index(),
3    feature_map.to_peptide_df(export_unidentified=False),
4    on="feature_id",
5    suffixes=("", "_psm"),
6)
7merged_df.head(2)
merged_df.head(2)#

feature_id

peptide_sequence

peptide_score

ID_filename

ID_native_id

charge

rt

mz

rt_start

rt_end

…

id

rt_psm

mz_psm

q-value

charge_psm

protein_accession

start

end

P_ID

PSM_ID

OMSSA_score

target_decoy

0

9650885788371886430

LVTDLTK

0.000000

None

spectrum=1270

2

1942.600083

395.239277

1932.484009

1950.834351

…

OMSSA_2009-11-17T11:11:11_1048455125463481685

1933.405151

395.239349

0.000000

2

P02769|ALBU_BOVIN

-1

-1

0

0

0.001084

True

1

18416216708636999474

DDSPDLPK

0.034483

None

spectrum=1167

2

1749.138335

443.711224

1735.693115

1763.343506

…

OMSSA_2009-11-17T11:11:11_1048455125463481685

1738.033447

443.711243

0.034483

2

P02769|ALBU_BOVIN

-1

-1

1

0

0.003951

True

ConsensusMap#

pyopenms.ConsensusMap.to_df( columns=None )

Generates a pandas DataFrame with both consensus feature meta data and intensities from each sample.

Returns:

pandas.DataFrame

consensus map meta data and intensity stored in pandas DataFrame

pyopenms.ConsensusMap.get_intensity_df()

Generates a pandas DataFrame with feature intensities from each sample in long format (over files).

For labelled analyses channel intensities will be in one row, therefore resulting in a semi-long/block format. Resulting DataFrame can be joined with result from get_metadata_df by their index ‘id’.

Returns:

pandas.DataFrame

intensity DataFrame

pyopenms.ConsensusMap.get_metadata_df()

Generates a pandas DataFrame with feature meta data (sequence, charge, mz, RT, quality).

Resulting DataFrame can be joined with result from get_intensity_df by their index ‘id’.

Returns:

pandas.DataFrame

DataFrame with metadata for each feature (such as: best identified sequence, charge, centroid RT/mz, fitting quality)

Examples:

1urlretrieve(
2    url + "ProteomicsLFQ_1_out.consensusXML", "ProteomicsLFQ_1_out.consensusXML"
3)
4consensus_map = oms.ConsensusMap()
5oms.ConsensusXMLFile().load("ProteomicsLFQ_1_out.consensusXML", consensus_map)
6
7df = consensus_map.to_df()
8df.head(2)
consensus_map.to_df()#

id

sequence

charge

rt

mz

quality

BSA1_F1.mzML

…

BSA1_F2.mzML

2935923263525422257

DGDIEAEISR

3

1523.370634

368.843773

0.000000

0.0

…

0.0

10409195546240342212

SHC(Carbamidomethyl)IAEVEK

3

1552.032973

358.174576

0.491247

1358151.0

…

0.0

 
1df = consensus_map.get_intensity_df()
2df.head(2)
consensus_map.get_intensity_df()#

id

BSA1_F1.mzML

…

BSA1_F2.mzML

2935923263525422257

0.0

…

0.0

10409195546240342212

1358151.0

…

0.0

 
1df = consensus_map.get_metadata_df()
2df.head(2)
consensus_map.get_metadata_df()#

id

sequence

charge

rt

mz

quality

2935923263525422257

DGDIEAEISR

3

1523.370634

368.843773

0.000000

10409195546240342212

SHC(Carbamidomethyl)IAEVEK

3

1552.032973

358.174576

0.491247