Data Storage at the Department of Geosciences

Data storage solutions at the Department of Geosciences.

Image may contain: Camera lens, Camera accessory, Lens, Cameras & optics, Circle.

As a researcher or student at Department of Geosciences (Geo for short), there are several options for Data Storage. This is a brief outline, with (some) references to more detailed articles.

See also this page, and note that there are rules and laws that you must abide to when storing sensitive data.

Local drives

Local drives include the C:-drive on Windows machines, your home directory on laptops and all Mac's, and external drives. Also, some servers and workstations have scratch space on local drives.

Data on local drives are not backed up, so these should only be used for data that you can afford to loose. Data will be lost if the drive is damaged or stolen, or if you delete files by mistake.

Home directory

Your network home directory (historically known as M:) is immediately available on most login services at UiO. This includes Windows and Linux servers and desktop machines. Windows laptops from UiO can access it by installing the app "UiO Legacy Homedrive" from Company Portal (autopilot), with Direct Access (legacy Win10).

On stationary Macs, it can be connected to manually. All other machines can access it with an sshfs client (WinSCP, Cyberduck, etc.)

Most users have 50 GB limit on M:. It will not be expanded, but keep reading if you need more space.

Data on M: are backed up regularly. Also, snapshots make for a very convenient way to retrieve recently lost data.

Note that your home directory will be deleted when your affiliation with UiO ends. The procedure is described here.

Astra

Astra is an old service that are not offered to new users. Existing users can follow this guide to connect. See below for replacement services.

Hypatia "felles"

We create storage areas here as needed, mostly for projects or other common areas.

For project storage, we will charge money. Currently, the price is NOK 500 per TB per year.

We will also need a DMP (Data Management Plan), detailing at least the following:

Name of project.
Project leader.
Other project contributors.
Project number, for accounting.
Access policy. Readable for all, or open to contributors only?
Requested size (GB/TB). This can be modified later.
Expected end of project.
Plans for archiving and reuse of data, in brief.

We can do with a text file or email detailing the above points, but we suggest you use EasyDMP for this. Please provide the link to the DMP with your storage request.

A Unix file group will be created, this will be used for access rights and also tied to the storage size (quota).

Hypatia "personlig"

Personal storage. This will function as an extension of your home directory, and a replacement for Astra (see above).

Scientific employees will get up to one TB of space here on request.

If you already have an Astra drive, this is intended to replace it. Let us know if you need help moving your data.

If more than one TB is requested, we will need a DMP similar to project space, as discussed above.

Your data here will be deleted when your affiliation with the Department ends.

Master Students will get a storage area here, upon recommendation by their supervisor. No DMP is needed, supervisor and student are responsible for data archiving. The area must be cleaned before a Master diploma can be issued.

OneDrive, Google Drive, Dropbox

This is personal cloud storage, not managed by Geo-IT. It is useful for Windows laptops and personal workstations, less so for servers and e.g. Linux machines. See the USIT pages for OneDrive, Google Workspace, Dropbox.

If you have a recent ("autopilot") Windows laptop, some folders will be associated with your OneDrive. Make sure you know which folders this is if you plan to rely on this.

Note that these services are for employees. You will loose access when your contract expires, even if you continue at UiO as a guest or emeritus.

vann

"vann" is the old storage server at Geo. It is out of official support, but it still contains useful disk space. vann is quite fast, as network storage goes. The server is located in the Geology building.

We can set up personal storage here for those that ask for it. A DMP is not required, but the following should be understood:

Data will be removed when studies are finished or employment is terminated. This is not an archival solution!
Storage is accessible from Linux (as /mn/vann/something); Windows/Mac users can access it with an sshfs client (WinSCP, Cyberduck, etc).
Data is backed up regularly, but there are no snapshots. In the event of data loss or system failure, it might take a few days to retrieve data from tape backup.

You can log in to vann. First time, you will get allocated a local home directory where you can e.g. install your ssh keys. This home directory is not for data storage.

Furthermore, vann is not for running calculations, but you can e.g. compress and sort and index your data there.

The 00-Admin directory

Some storage directories (notably on hypatia "felles") will contain a subdirectory 00‑Admin. This is created by us (Geo-IT), and will contain notes about the storage and a few other files. Usually, this will be readable to the users.

Of special interest would be these two files, usually created automatically every week or so:

grupper.txt.xz : A list of files, and their corresponding group owner.
du-m.txt.xz : The disk usage of the directory and subdirectories, numbers i Mebibytes (units of 1048576 bytes), sorted with biggest (i.e. the total) first.

The files are xz-compressed. You can decompress them in Linux with the xz* utilities, for example:

# View the list of files and groups:
$ xzless 00-Admin/grupper.txt.xz

# Find the twenty biggest subdirectories:
$ xzcat  00-Admin/du-m.txt.xz | head -21

# Search for a file by name:
$ xzgrep "filename" 00-Admin/grupper.txt.xz

Data archiving

The Department has no archiving service, but check out these offerings: