Using Marmot SFTP Server for Sideloaded MARC

Owned by Ashley Sneesby, created
Last updated: Nov 16, 2023
9 min read

This document details how to configure and use the FileZilla FTP client in order to load MARC record sets for sideloaded collections in Pika.

Table of Contents

Top of page

Configuring FileZilla

Marmot recommends using FileZilla for your FTP client.

First time server setup using FileZilla

Open Filezilla.  You will notice your desktop folders and files on the left.

Click on File and Site Manager.

The Site Manager box will appear.  Click on New Site.

This will allow you to access the General tab to enter the Host, Port, Protocol, Logon Type, User and Password.

The Host is sftp.marmot.org.  Do not enter a value in the ‘Port’ field.  Use the dropdown menu to change the Protocol to SFTP – SSH File Transfer Protocol.  Use the dropdown menu to change the Logon Type to Normal.  Enter the User and Password provided by the Pika team. Once you have the User and Password enter both, and click Connect.

The User and Password fields are both case sensitive. These need to be entered exactly as provided by Marmot.

Marmot may need to update our firewall to enable connection. If you cannot connect, please submit a service request in the Marmot Help Desk with your IP address, and we can give you access.

You will return the Enter password pop up box.  Reenter the password.  You can also check the box to remember the password until FileZilla is closed. Click OK.

You return the Unknown host key pop up box the first time you login.  Click the box to Always trust this host, add this key to the cache.  Click OK.

After you complete the site manager setup and login, your local site will be listed on the left.  You library’s FTP site will be on the right.  Under filename will be any files you downloaded, as well as all the processed files.

Top of page

Renaming ‘New Site’ Folder

It is best to go back to the File >> Site Manager, and Rename the ‘New Site’ to something relevant. We recommend your library’s name.

 

 

Top of page

Save the password

To save the password go to Edit and Settings.

Go to Interface >> Passwords. Toggle on the Save Passwords setting. Click OK.

Go back to File and Site Manager.

Go back into your newly renamed site. Reenter the password and click OK.  The next time you login, you will not need to enter the password.

Top of page

Adding MARC records to an FTP Site

All MARC sets used for sideloads must be in .mrc format for use in Pika. Pika cannot handle MARC sets in .mrk format. This may mean you need to compile your MARC sets using a tool like Marcedit.

  • The first step will be getting a file of MARC records from the eContent vendor of all items your patrons have access to. The filename should end with a .mrc extension.  If your library needs to use proxy links for your patrons to access the content, please inform us how the proxy links should be set.  You may not need to modify the MARC records to include the proxy links; there are options in Pika we can usually use to account for proxy wrappers as part of the sideload set up.

  • After you request the MARC file from the vendor, please contact Marmot via the Marmot Help Desk for an FTP account and credentials, including the Public IP of the machine you will upload the files from.  We will create an account on our FTP server for your library to send us updated MARC files. The Public IP is needed to allow you to connect to the FTP server through Marmot’s firewall.

  • If the URLs your library will use differ from the URLs other libraries would use, it is typically easier to have a separate set of MARC records even if the collection from the vendor is similar to one from another library.

    • Pika can now modify eContent links if there is a reliable pattern to follow, so there are exceptions to this.

    • With the first file, we will do an initial configuration on the test server for you to review.

  • The next major step will be setting up for regular updates of the MARC data after the initial setup.

    • Using FileZilla, sign in to the FTP server using File and Site Manager.

    • Find the appropriate side loading ftp account.  If you follow the instructions on saving your password, the user and password information will populate automatically.

    • Click on the Connect button.

    • After clicking Connect you will return a pop-up window in the client. Toggle on the option to Always trust this host, add this key to cache and click okay.

    • On the right side are the vendor files.  Click on the vendor name to see the Marmot files associated with that vendor for your library. Drag and drop the MARC files you’re working with in the local left hand ‘Filename’ pane (your local files) to the appropriate folder(s) on the remote right hand ‘Filename’ pane (the FTP server) to transfer.

       

    • There are three options in the vendor file directory :

      • Complete Collection is a full export (a MARC file of the entire collection)

      • Adds/Updates Only (MARC file that only has updates and additions for the collection)

      • Deletes Only (MARC file of records to be removed from the collection)

  • The configuration entails that once a MARC file has been uploaded to the FTP server, that data will be copied to the Pika servers and Pika will display the sideloaded collection with the next full index (usually overnight).

  • Once the sideload has been reviewed in your library’s test site, we ask that you confirm the setup looks correct. After your confirmation, the settings for the sideload will be copied over to your library’s production site.

  • Moving a sideload into production doesn’t always correspond with the code deployment schedule, though we might wait until a code deployment or patch deployment if the timing allows – usually when the setup work requires additional development. We prefer to have an explicit approval message from you move the sideload settings into production.

Please find an example of a sideloaded record here. In this example, Mesa County Libraries have a unique sideload for their Learning Express collection – the sideload is only included in Mesa County’s ‘Records to Include.'

Sideloaded collections are easily found using the eContent Collection facet, where sideloaded content displays along with other eContent, separated by source.

Top of page

Which folder should I upload MARC updates to?

Complete Collection 

The Complete Collection folder should have a single .mrc file which is the best representation of the full collection, as provided by your vendor.  If your vendor provides you with several files, these should be compiled (usually via MARCedit) into a single .mrc file, since our process only recognizes a single file in this folder.  Adds and updates and deletion files both check against the complete collection file.  If the complete collection file is replaced/reloaded, the most recent complete collection file will replace existing results in Pika for the side-loaded collection. 

Adds/Updates Only

The Adds/Updates Only folder should be used to add new records to your vendor collection or to update existing records in the complete collection. There can be several .mrc files in the Add/Updates Only file. 

Additions check against the existing complete collection file by record ID number (usually in the 001 field) - if the record is not in the complete collection, then it is added to the collection. Updates check against the complete collection file by record ID number - when the record is found in the complete collection file, then the updated record replaces the existing record in the complete collection with the newer updated record. This process can be used to make cataloging fixes such as updated URLs. 

Deletes Only 

The Deletes Only folder should be used to remove records from the complete collection, typically when purchases have expired and are no longer available or titles have been weeded. There are several methods by which we can delete holdings from sideloaded collections.

MARC file

Some vendors provide MARC sets of deleted or removed holdings. This must be a .mrc file transferred to the FTP server, representing titles to be removed.

Record number file

Some vendors provide a .csv file of records to be removed. Records to delete in this format must include a record ID number in the corresponding MARC field (usually the 001 field) to check against the existing complete collection. 

Deletions check against the complete collection file by record ID number (usually the 001 field) in this process. When the record is found in the complete collection file, it is removed from the collection.

ISBN file

Some vendors provide a .csv file of records to be removed with ISBN information. Records to delete in this format must have a heading of ISBN in the source .csv file so Pika can reference the correct field.

Deletions check against the complete collection file by ISBN in this process. When the record is found in the complete collection file, it is removed from the collection.

Top of page


Important Information About Records from the Integrated Library System (ILS) and EBSCO

  • If vendor records to be sideloaded are currently stored in your ILS, they will need to be suppressed or deleted in your ILS when Marmot puts the side load on the production Pika server.  

  • Please let us know at pika@marmot.org, if this collection is currently in your ILS.

  • Another issue to consider: if ILS eContent records were previously part of the export of records from your ILS that gets sent to EBSCO Discovery Services (EDS).  If the potential sideloaded collection is stored in the ILS and exported to EDS, they would not be part of the export to EDS once suppressed in your ILS.  The records can be included again in the EDS export, but it also requires EDS to create a custom URL for the records by filling out a new custom catalog questionnaire. In the past, there have been fees associated with this process, so it’s preferable that you contact your EBSCO/EDS rep to confirm any changes.