How to use SciServer

1. 1. Access the Menu in the top right-hand corner of the SciServer Dashboard:

1. 1. Select “Help” to display a small dialog with a number of links:

1. Here you can find “User Guides”, “API Documentation” and “Help Desk”

1. 1. Access the Menu in the top right-hand corner of the SciServer Dashboard:

1. 1. Select “Help” to display a small dialog with a number of links:

1. 1. Here you can find “Bug Report Form”
   2. Select this and it will open a new Web Page on the sciserver.org website:

Small question mark icons can be found on most of the Dashboard pages with links to specific section of this How to Use SciServer document. The icons look like this:

The SciServer Dashboard is the application that allows users to manage the work they are doing with SciServer and the other applications available. A user will be sent to the Dashboard Home Page after logging in from the Login Portal, and can navigate back to the Dashboard from most other SciServer Applications via shortcuts.

Home Page

The Homepage for the Dashboard looks like this:

Menu and Functions

The Dashboard has a main “Menu Bar” across the top which is always visible in all views of the Dashboard, providing access to the following core features:

• • Brings the user back to the Home Page

• • Navigates the User to a Tab to manage Files and Folders

• • Navigates the User to a Tab to manage user defined Groups and sharing

• • Displays a drop-down menu allowing the user to launch any of the supporting SciServer Applications (Home, Compute, Compute Jobs, CasJobs, SkyServer)

• • Navigates the User to a tab displaying a tabular summary of the history of their activity with SciServer functions and applications

• A Menu dropdown that displays additional options for the user (access Profile, Access Help, Change Password, Sign Out)

Function Shortcuts

A set of shortcuts are displayed to access the many functions in the Dashboard application itself:

In particular the User can access:

• Files
• Groups
• Compute Jobs
• Activity Logs

Each of these shortcuts also displays information about recent activity on, or notifications about, each function.

Application Shortcuts

A second row of shortcuts allows the user to launch any of SciServer’s core applications:

Each of these shortcuts will launch a separate web app in a separate Browser tab.

Note that “Compute Jobs” is in both shortcut lists.

The top right of the Dashboard page, next to your username, displays the bell-shaped notification icon. Notifications will appear when you upload a file – both when the file begins to upload and when the upload completes. Notifications appear when you move or copy a file as well.

If you have active notifications, the number of active notifications will appear next to the icon: .

When you click on the Notifications icon, a panel will open from the right showing your active notifications:

In the Notifications panel pictured above, green sections specify actions currently in progress, while red cards specify actions that have failed, along with their error messages. Once an action has completed, it will disappear from the list; once all actions have completed, a blue “No new notifications” message will display.

You can clear the notifications for any failed actions by clicking the ‘x’ at the bottom of the notification message. You can clear all notifications by clicking the red “Clear all” button at the top of the notifications panel. To hide the notifications panel, click on the dashboard outside the notifications panel space.

All notifications are cleared automatically when you log out of SciServer, or when you reload the dashboard page in your browser.

SciServer provides two ways to view your recent activities:

• The Dashboard displays the most recent activity related to File Management, Group Management, and Job Execution
• A separate “Activity View” accessible from the main Menu in the RHS corner provides a detailed table of all logged activities within SciServer, with filtering and sorting.

Dashboard Activity Summary

• On the main dashboard, a summary of Jobs, Groups and Files is provided as shown below
• In particular, Invites to new groups are shown, which the user can select to take them to the Groups tab in SciServer

Activity Log

• • Access the top right-hand menu bar and click the Activity Log icon:

• • This will display the Activities table:

SciServer provides a feature called ‘Groups’ for users to share their resources with their collaborators privately. The resources that users can share are file folders, databases, volume containers, and Docker images. Also SciServer provides a Group view which lists all the shared resources among group members.

SciServer Groups allow you to create lists of Users and to share resources, such as file Folders, to all of them at once. You can manage a team of people in a project by creating a group with the relevant users in, and allowing everyone to work from the same shared folder. Importantly, you can also make sure that no one who is not in the group can access the shared folder, so you can keep things private.

1. 1. Select the Groups tab in SciServer

1. 1. This will show the Groups View

1. 1. The left hand box will show you all the groups that you are a member of. As you select each one, the middle set of boxes will show you which resources have been shared with that group, and the right hand box will show you all the members in the group.
   2. To Create a New Group, click the “+” button on the left-hand “Groups” box:

1. 1. This will open the “New Group” dialog box:

1. 1. Enter a name for the Group and an optional Description, then press the “Create” button. This will show your new Group in the “Groups” box, and will show you as the only member in the box on the right-hand side:

1. This is your new group!

1. 1. You can invite other SciServer users only to groups that you own, or that someone else has given you the grant privilege on
   2. In the groups view, go to the “members” box on the right hand side:

1. 1. Press the green “+” button to show the Add Members dialog box

1. 1. Select Users from the left hand panel. You can multi-select by holding down the “Ctrl” key and clicking on users. You can also search for users by typing characters into the Search box.
   2. Press the Invite User button:

1. 1. You will need to choose “Admin”or “member”:
      • Admin: means that this user will be able to invite other users to your group
      • Member: means this user will not be able to invite other users to your group
   2. This will add the users to your group with the status “Invited”

1. 1. Press the OK button in the lower right-hand corner to dismiss the dialog box
   2. NOTE: This will invite the users. They will need to accept the invitation before being a member of the group:

1. 1. On the Dashboard tab, you will see an Invite:

1. 1. Select the Groups tab, and you will see the Group listed with an icon next to it:

1. 1. Select the group and you will see an option to accept the invitation:

1. Select “Accept Invitation”, and you will join the group and be able to access it.

1. 1. In the Groups Tab, selecting any of the groups on the left hand panel will show the resources already shared with that group. Resources that can be shared are: (1) File Folders (2) Compute Docker Images (3) Compute Volume Containers (4) Databases (in some circumstances). It will only show each box if there are any resources that you are able to share.
   2. Each of these resources is shared in a separate box, and each has a button
   3. Clicking the button will display a dialog box where you can select resources that you have access to:

1. 1. Select the resource and permission level, then press “Share” and the resource will appear in the group:

1. 1. In the Groups Tab, selecting any of the groups on the left hand panel will show the resources already shared with that group.
   2. Click on the shared resource
   3. Click “Remove”

1. An alert box will appear asking you to confirm deletion
2. Click OK to proceed

1. 1. In the Groups Tab, selecting any of the groups on the left hand panel will show the resources already shared with that group.
   2. Click on the shared resource
   3. Click “More Info”

1. 1. Pressing this will display a dialog box where you will find more information:

SciServer provides a number of features for managing files and compute notebooks, all of which will be made available in both the Sciserver dashboard application and in SciServer Compute containers from within Jupyter Notebooks. All the common capabilities of File management are provided, such as renaming, moving, copying etc, but also sharing of “User Volumes” for collaborative work.

NOTE: the File Management features provided in the SciServer UI are different to the services provided by SciDrive, which is an older storage system that is still supported for legacy applications.

User Volumes and Data Volumes

SciServer supports two different types of file volumes:

• User Volumes: These are created and owned by individual SciServer users
• Data Volumes: These are institutional or organizational data repositories that are currently owned and managed by the SciServer system. In some cases, a SciServer user who originally provided the dataset in the first place can be made an owner and given administrative rights, and may grant teams of other users read-write access to their Data Volume(s).Otherwise, Data Volumes are public (all SciServer users can see them), and read-only to all users. For those users familiar with SciServer Compute, Data Volumes are the same storage volumes as the “Volume Containers” made available in Compute.

NOTE: This section of the Help documentation (“File Management: User Volumes”) will deal only with User Volumes. Data Volumes will be covered separately in the next major section (“File Management: Data Volumes”).

Permanent or Temporary Storage?

Users can create a number of top level “User Volumes,” under which new folders and files may be added. User Volumes can be created in one of two different storage pools: a permanent pool called Storage and a short-lived pool called Temporary.

Folders and files in User Volumes under Storage will be backed up and permanent, but there is a quota limit of 10GB. Folders and files in User Volumes under Temporary are not backed up, and will be deleted after a particular time period, but there is no imposed limit or quota on how much data can be stored (because it will be deleted).

Most users should store their data and files in a Storage User Volume. The Temporary User Volumes are meant to be used as intermediate storage for SciServer Compute calculations.

1. 1. Login to the SciServer dashboard and select the “Files” tab:

1. 1. This will show the Files View at the top level of “User Volumes”:
      
   2. NEW: Volumes and files are sortable! ​By default, Volumes are sorted alphabetically by volume name. Within a volume, files are sorted alphabetically by file name. You can change the sort order by clicking on the column header (e.g. Volume, Root Volume, etc.) Click on the same column header again to sort in descending order.
   3. There are a number of User Volume operations available in this view
   4. Hover over each row to view available operations
   5. Click on the ellipsis button to view available operations:
      • Share: If the user has the appropriate permissions, they can share a User Volume with other users, or groups of users. A User Volume created by the user is always shareable by them.
      • Delete: If the user has the appropriate permissions, they can delete a User Volume. A user can always delete a User Volume they create.
      • Edit: The User Volume name can be changed, and a description provided for it.
   6. Different icons refer to different levels of sharing:
      • • This User Volume is owned by the user, and has not been shared
      • • This User Volume is not owned by the user, but has been shared with the user by another user or group.
      • • This User Volume is owned by the user, but has been shared with another user or group.
   7. Selecting a Folder in the column on the left-hand side will open that folder up, one level at a time. It does not provide a tree view.

1. 1. There are a number of file and folder operations available in this view
   2. Hover on each to view available operations
   3. Click on the ellipsis button to view available operations
      
      • Copy
      • Rename
      • Move
      • Delete
      • Download: A file can be downloaded (but a Folder cannot)
   4. Perform multiple operations on files or folders, or on User Volumes (currently only for the Delete operation):
      • Check one or multiple checkboxes
      • The menu gets displayed with available operations on top
      • Click on any menu item to perform an operation

1. When running SciServer Compute, the same filesystem is presented by the Jupyter application, but is presented in a more traditional hierarchical manner with a full path access that supports working with files in a Linux Console:
   

   
2. The operations on files and folders available in this view are provided by Jupyter.

A User Volume is a top-level folder that is created and owned by a SciServer User. It is different to a “normal” Folder in a few ways:

• There is only the one top level of User Volumes, all subsequent lower levels are normal Folders
• A User Volume can be shared with others users and groups, but a normal folder cannot. This is very important.
• A User Volume can be selectively “mounted” in a Compute Container, and made accessible to a Jupyter Notebook. Folders at lower levels under a Volume Container cannot be.
• A User Volume cannot be moved or copied.

1. 1. On the SciServer “Files” tab, click the “View Quotas” button:

1. The “View Quotas” dialog box appears

1. 1. On the SciServer Files tab, click the button to get back to the User Volume view:

1. 1. Press the “Create User Volume” button:

1. 1. In the dialog that pops up, enter a name and optional description:

1. 1. Select a “Root Volume,” which means decide whether to create the new User Volume in a permanent and backed-up Storage pool, or to create it in a Temporary pool, knowing that it will be deleted after a certain period of time (defined by the SciServer Data Storage Policy).

1. Press the Create User Volume button and the new User Volume will be created.

Files cannot be uploaded directly to the top-level User Volume level.

Files can be uploaded to any Folder inside an existing User Volume, including in that User Volume’s top-level directory.

Upload Files using the Upload Button

1. 1. Navigate into a User Volume and you will see an “Upload” button:

1. 1. Press the Upload button to display a dialog box:

1. 1. You must navigate to a File, select it then press the “Open” button

1. 1. This will then show the file you just uploaded:

Upload Files using Drag and Drop

1. 1. At the User Folder level (not Root folder level), the files view displays a “Drag and Drop” box

1. 1. You can drag and drop multiple files from, say, Windows Explorer, onto this box, and they will be uploaded to the displayed subfolder
   2. Upload progress is displayed

Like GitHub, SciServer allows you to upload a file called README.md offering a high-level introduction to the purpose, contents, and structure of your User Volume. Adding a README.md file makes it much easier for your colleagues and others to understand your data.

The name is case-sensitive; README must be in all capitals and the file extension must be lowercase: README.md

When you upload a README.md file into a User Volume, SciServer will automatically render the file and display its formatted text at the top of the page, as shown in the image below. You can click on the “Skip to file list” link to jump down the page to the list of files.

If you delete the README.md file from your user volume, this extra information will disappear.

1. 1. You can share any User Volume that you own, and you can share others’ User Volumes if they assigned you that permission when they shared it with you
   2. On the SciServer Files tab, click the button to get back to the User Volume view. You will see a list of User Volumes. All User Volumes that you are allowed to share will have a Sharing button next to them:

1. 1. Click the “Sharing” button and it will display a pop up dialog box:

1. 1. The left hand panel is where you choose, one at a time, the users, or groups of users, that you wish to share with. To find users more easily, type something in the “Search:” box and the list will be filtered:

1. 1. Click on the name you want to add to the group to highlight it, then press the “Add” button:

1. 1. This will add that name to the right hand panel, where you choose the Permissions that you want the user/group to have on your User Volume. These permissions are as follows:
      • Read: The user is able to view and read the User Volume and its contents, but not change anything
      • Write: The user is able to write new files and folders to the User Volume
      • Delete: The user is able to delete the User Volume (not recommended!)
      • Grant: The user has the ability to also further Share this User Volume with other users.

      
   2. Select the Permissions you want to assign by checking the checkboxes.
   3. You can keep adding more people/groups.
   4. Press “Save Changes” when you are done.
   5. The User Volume you have just shared will have its Icon changed to indicate that its sharing permissions have been changed:

The process for Unsharing is the same as sharing described in the section How do I share a User Volume?.

1. At the Files tab, click the button to get to the top-level User Volume view, and you will see the list of User Volumes. All User Volumes that you are allowed to share will have a “Sharing” button next to them:
2. Click the “Sharing” button and it will display a pop up dialog box:
   
3. On the right hand panel, select the user you wish to Unshare with
4. Press the “Remove Access” button OR just uncheck all the checkboxes manually
   
5. Press Save Changes, and the User Volume will be displayed with the regular blue folder icon indicating that it is no longer shared

• • Regular folder OR user volume that has not been shared

• • User volume that has been shared BY you

• User volume (owned by another user) that another SciServer user has shared WITH you

To move or copy files between folders in SciServer, use the Files tab in the SciServer Dashboard:

1. Select the Files tab
2. Navigate to the files that need to be copied – hover the mouse over the file to show the “icons” (one of which is Copy, and one of which is Move). If you wish to move or copy are multiple files, select them all using the checkboxes, and select the “Copy” or “Move” button that appears at the top:
   

   
3. When you press the Copy or Move button (in either of the above cases), this dialog will appear:
4. Navigate to where you want to copy or move the files, and then press the “Paste” button:
   

NOTE: You must have write permissions on the destination folder in order to move/copy a file or folder there

Data Volumes are hosted data sets that belong to an institution or organization, and are shared with select groups of users.

Some of the Data Volumes belonging to Johns Hopkins University are public, and all SciServer users will see them. Data Volumes are not part of any given user’s quota, though a particular user can be given the “owner” role by JHU Admin staff for them to subsequently share their data volume with particular users in their team or domain. A good example of a Data Volume is the Sloan Digital Sky Survey (SDSS) Data Archive Server (DAS) data.

Data Volumes as presented in the Files view of the dashboard are the same underlying datasets as Volume Containers presented (and accessible) in SciServer Compute.

The datasets were exposed as Volume Containers in SciServer Compute early on in the SciServer project to support the growing number of projects that needed to do analysis on the data. Now, exposing the datasets through the dashboard as data volumes makes it easier to quickly find out what datasets you are allowed to see, to view their contents, and to copy data from them to your own personal storage space. For some data volumes you may be granted write access, in which case you can add new data to a Data Volume from your SciServer dashboard.

1. Access the Files view from the SciServer dashboard:
2. You will see two options on the left-hand side: User Volumes and Data Volumes
3. Pick Data Volumes and this will change the RHS view to display those data volumes that you have been given permission to see:

1. First, note that you can only copy data and/or folders from within a Data Volume, you cannot copy an entire Data Volume
2. Inside a Data Volume of interest, navigate to the folder or file you want to copy – you will see an option on the right-hand side to “Copy” the folder or file:
3. Select the “Copy” button; this will show a pop up dialog where you can select where to copy the data to:
4. Navigate to the directory of interest, and press the “Paste” Button:
5. The folder or file will now appear in the new place:

SciServer Compute is an application that allows users to easily create and run Jupyter Notebooks containing code and instructions to analyze and process SciServer hosted data sets. SciServer provides a rich API to access all aspects of SciServers resources, including databases, Filesystems, user and group management, and even Compute Jobs.

There are a couple of steps involved, which SciServer makes easy:

1. Create a new “Container” to run the Jupyter Notebook in. A container defines the “environment” for the user, and is configurable
2. Open the container, which will start Jupyter, and create, save and execute Notebooks.
3. The full capabilities of Jupyter (which is a third party application) are available to the user and will not be covered here.

An important step in setting up a Container environment is specifying what external file systems will be accessible to the Jupyter environment.

A Container in SciServer Compute is a defined environment within which Jupyter Notebooks can be run. Its is technically a Docker Container (Docker is the technology used), and provides a way to isolate the user and their code from the rest of the SciServer system, and other users.

A Container in SciServer Compute is a “long-lived” resource, and as such there are some resource management issues you need to know about:

• Each User can create up to 3 containers at any given time. If you need another one, you need to delete one first.
• Containers have a lifecycle, and can be “running” or “stopped”. SciServer keeps Containers running for a certain period of time, even if the User is not actively working with it, to ensure that when a user comes back to it, it starts up nice and fast without delays.
• Running Containers consume system resources like memory etc., and run on an environment with inherent risk of failure. We do not make any guarantees on the lifetime of a container and they are subject to be terminated without notice. Any data stored on a container’s ephemeral filesystem (as opposed to user and data volumes) is subject to loss.
• Whereas data (files, folders) can be stored “in” the container, you should never do this for any data that you need to keep. Always store data files in the Storage or Temporary storage pools, which are external to the container. Data in these storage pools are accessible when all containers are closed or deleted, and the same data is accessible across any containers that includes those Volume Containers in their environment.

Creating a new Compute Container is easy! However there a few parameters required to define the compute environment that you need.

1. 1. Access the Compute Application by clicking on the “Compute” icon on the SciServer Dashboard:
   2. OR from the “Apps Menu”
      

1. This will take you to the “Compute dashboard”
   
2. You will see a large text box explaining how to use the Storage capabilities to ensure that you do not lose data if you accidentally store it in the Container itself.
3. Press the “Create Container” button, and this will pop a dialog box:
   
4. The following needs to be entered:
   • Container Name: you choose this
   • Domain: This is a drop down from which you should most always leave at the default value “Interactive Docker Compute Domain”.
   • Image: This define a “software environment” for the Jupyter notebooks that you want to run. The images contain libraries tailored to different needs. For the most part you will choose an image that supports the language you are interested in (python, R, Matlab etc), but there are “specialty” science domain specific images that you may have access to if the creator of those images has shared it with you. Additional information on “Images” can be found here at the Compute Images help page.
   • User Volumes: This is a list of all User Volumes that you have access to, either which you own, or which have been shared with you. When you select some of these, on container creation these folders will be “mounted” and will be accessible as if they were local files. This makes file access and management much simpler. NOTE: they will be mounted with the same access controls as you would have in the SciServer File UI i.e. “readonly” or “readwrite”.
   • Data Volumes: These are a series of special Data Volumes that are either shared publicly with all users, or for which you have been given special access privileges to see. Again, selecting these Volumes will mount them, and make them appear “local” in the Container. These will always be mounted “readonly”. Additional information on “Data Volumes” can be found at the Available Datasets page.
5. Once created, it is not possible to add additional User Volumes or Data Volumes, so you should be sure to get this right.
6. Pressing the “Create” button will create a new Container and show it in the table with the name you provided.
7. Clicking on the link in the column titled “name” will launch Jupyter

1. 1. Start one of your Compute Containers to Launch Jupyter:

1. A new window will open showing your filesystem, which is accessible through Compute.
2. Navigate to the directory in which you want to create your notebook.
3. Select New from the top right corner of the interface to open the dropdown menu
4. Select the language that you would like the new notebook to use
5. The notebook will open in a new window

1. 1. When you open a Container, Jupyter will start showing the files and folders, and you will see an option at the top to “switch To JupyterLab”:

1. 1. Selecting this will put your Jupyter session:

1. 1. To get back to “Classic Jupyter” view, select the menu item Help -> Launch Classic Notebook:

Compute Jobs allows a user to run a Jupyter Notebook or a standard script in offline batch mode. The same exact capabilities are provided as for Interactive Compute:

• Compute Images and software environment
• Mounting external volume folders

Executing a job will put it in a queue, and it will be run when there are resources available on the server cluster.

You might create a Job for the following reasons:

• Executing your notebook may take a long time and you want to set it running and do something else without worrying about browser sessions timing out etc.
• You may develop your code interactively to make sure the algorithm works, using a small amount of data to test it out. But you really want to run your code against a full dataset which will require massive resources for memory and CPU, as well as execution time.
• You are provided with far more resources (CPU and memory) to execute a Job than you are in an Interactive Session.

SciServer allows you to define two “types” of job:

1. Specify a script to execute, or a command line command
2. Specify an existing Jupyter Notebook that you have previously developed

The second of these is the most useful in that you can develop your Jupyter Notebook interactively then “submit” the exact same notebook as a Job.

Creating a new Job is easy! We explain how to create a notebook based job, but creating a script based job is very similar.

1. 1. Go to the Compute Jobs Page:
      
   2. Click “Run Existing Notebook”
      
   3. On the ‘Compute Domain’ Tab:
      • Choose the Compute Domain, for which in most cases currently there will only be one option
      • Optionally enter a “Job Alias’ to easily identify your Job later
   4. On the ‘Compute Image’ Tab:
      
      • Pick the ‘Image’ you need to use.
      • Each image contains different tools and programming language support.
      • (Compute Images are described in more detail at the Compute Images help page)

1. 1. On the ‘Data Volumes’ tab:
      
      • Select all the data volumes with appropriate permissions needed for this job.

1. 1. On the ‘User Volumes’ Tab:
      
      • Select all the Folder systems that you would like to be made accessible to your Compute notebook
      • For Folders that you own, or that have been shared with you and you were given the appropriate permissions, you can select whether a given folder is read only or writable. Folders that you do not own will be readonly by default.

1. 1. On the ‘Notebook’ tab:
      
      • Navigate to the Notebook you wish to use as the basis for your Job, and select it
      • Enter any additional parameters that the Notebook can read in to affect how the code is executed
      • Choose a directory where the output results will go
        • By default these will go to jobs within which subdirectories will be created and your results written to.
        • Alternatively you can choose a specific directory to output results. The directory you choose will be a ‘root’ within which subdirectories will be created and your results written to.

1. When everything has been entered you can press ‘Create Job’, and the Job will be submitted, and displayed in a Jobs Table view:
   
2. The Table will be refreshed every several seconds, telling you the status of the Job.
3. While the Job is still running there will be a red “X” button, and pressing this will Cancel the job.
4. Pressing the down triangle on the right-hand side will expand the view and show more information about the Job. This is what you see for a completed Job:
   
5. This gives status information about the Job, the path to the location of the results, and links to the results output

1. 1. After submitting a Job, and before it has completed execution, it can be cancelled.
   2. Display the Jobs table, and your running Jobs will be identified by having a red “X” next to them:

1. Press the “X” button to cancel the job.

1. 1. When your job is complete the Jobs Table will display the status:

1. The results output will go to the location specified in the Job Definition
2. In the Jobs Table, expand the job of interest:
   
3. This give status information about the Job, the path to the location of the results, and three hyperlinks:
   • Browse Working Directory will take you to the Dashboard Files tab and show you the output files as well as the original Python Notebooks.
   • Download Standard Output and Download Standard Error will allow you to download these two text files as appropriate. They will be downloaded according to your Browser settings.