Home → Student Access Initiative → Student Access → Uploading Data Feeds Directly
1.4. Uploading Data Feeds Directly
Last Updated 11/04/2024
Uploading Data Feeds Directly by Secure File Transport (SFTP)
Available as of Fall 2020, library systems may generate and upload their data files themselves on whatever basis they so choose. Each system who does their loads in this way should designate one person to be the one responsible for uploading the data. This helps to keep information secure (both the student information and the FTP information) and to make the process faster, more efficient, and automatic for the system. It will also let the Cardinal team know to whom to reach out in case of issues, as the Evergreen result email notifications will still be sent to the Cardinal team.
Public Key
To begin the SFTP set up process, the interested library will need to generate and share a public encryption key which will be added to the Evergreen server to allow staff members to authenticate and upload feeds.
Directions for Windows users can be found here (but do not specify a password/passphrase).
Alternatively, MacOS/Linux users can type "ssh-keygen -t rsa".
In both cases, an RSA key should be generated, and a password/passphrase is not needed. The library system (or designated staff member) will then need to share the RSA public key with NC Cardinal, either copied and pasted out of PuTTYgen for Windows users, or the ~/.ssh/id_rsa.pub file on MacOS/Linux. Your private key will remain on your computer.
SFTP Software
Any SFTP software will work, and WinSCP and FileZilla are popular choices.
https://winscp.net/eng/download.php
https://filezilla-project.org/
Here are FileZilla directions for adding to FileZilla the private key that was created earlier.
https://filezillapro.com/docs/v3/advanced/ssh-private-keys-for-sftp/
Connecting to the Server and Uploading a File
Once your public key is registered on the server, you can connect set up the connection using the following settings: (This screenshot is the configuration using WinSCP)
Rather than using a password, you point the SFTP software to your local private key (it should essentially be a text file with a .ppk file extension)
The SFTP connection should have the following details:
host: 52.23.227.84
port: 22
user: externaldata
path: /home/externaldata/uploads/student_csv_import
Upload your .csv file(s) to the /uploads/ folder, and avoid uploading directories. Only .csv files are processed. Uploaded files are moved out of /uploads/ every 5 minutes and processed every 10 minutes, so the process can take 10-11 minutes to be handled.
Example screenshot of Filezilla file transfer window:
Example screenshot of Default Remote Directory setting in FileZilla’s Site Manager:
A member of the NC Cardinal team and the library system's designated staff member can schedule a phone call or online video conference to walk through adding a file from the library system's set up together.
File set up and naming conventions
There are two columns to add to the data set from the template file shared here.
They are columns for “Home Library” and “County” – county is self-explanatory, but for the home library you will need to use the capitalized "shortname" branch code for the branch or main library that will be the Home Library of the students. For regional library systems, Home Library may be decided in each county. For example, Appalachian Regional Library (ARL) uses ASHE as Home Library for Ashe County, WATAUGA for Watauga, etc. HENDERSON_MAIN is home library for Henderson County Public Library system.
If you do not have data for each column, that is ok – still include it with the header, the rest can be blank. The software needs to import columns A-P in the template order regardless. The most important columns that need to be filled in are ID, first and last name, and DOB.
It is important to note that the software uses a date prefix on each file to assign expiration dates for accounts. Each year these are updated, but the format is yearmonthday, no dashes or spaces, like this: 20241001.
We use October 1st as the expiration date to allow for time in the beginning of fall semesters for account updates by way of file uploads. It is very helpful to have the file name reflect the system in question so if there are any errors the Cardinal team can notify the right person. With this information in mind, a file name would ideally look something like: 20241001_Davidson_students.csv