I recently have to migrate Rutgers Plus from Firebase to my self-hosted mongodb (Atlas) because of the Chinese great firewall.

First install gsutil, firebase-tools and mongo shell.

To migrate data collections one at a time, create and run a bash file:

echo "Exporting from firebase.."
firebase database:get /$1 --project <PROJECT_NAME> --pretty > $1.json
echo "Switching id..."
awk '$0 ~ /^  "/ {split($0, a, "\""); print "\{ " "\"_id\": " " \"" a[2] "\","; next}{print}' $1.json > $1_tmp.json
echo "Converting to array..."
awk 'NR==1{print "[";next}/^}/{print "]";next}{print}' $1_tmp.json > $1_array.json
echo "Uploading to mongodb..."
$HOME/.../mongodb-osx-x86_64-4.0.0/bin/mongoimport --host <MONGO_HOSTNAME> --ssl --username <USERNAME> --password <PASSWORD> --db rutgersplus --collection $1 --jsonArray --file $1_array.json

where $1 will be the name of the collection. In the file, I converted the id from firebase’s “id as the key” format to mongodb export’s “id as the value” format. Also mongoexport doesn’t strictly follow the JSON standard as it requires an array without comma separators instead of an object with id-as-keys. I did the tranformation as well.

To download static files from Firebase for re-uploads:

gsutil cp -r gs://<PROJECT_NAME>.appspot.com/users .

To migrate users, find password hash parameters in the Firebase authentication tab, you get {key} {salt separater} {rounds} {memcost} there. Download the users info csv:

firebase auth:export --project <PROJECT_NAME> auth.csv

and mark the column right before the first name and last names of the users. Those are {salt base} for each user.

We need the same algorithm firebase used for hashing passwords to keep using the existing password hashes, or we need to ask the users to reset their passwords or authenticate with firebase in the future to migrate users one at a time. These two methods are also the official methods from AWS sources.

Luckily after some decent researching, I learnt that Firebase kindly provides their modified SCRYPT hashing algorithm on Github. Therefore, on the server:

apt-get install autoconf
apt-get install build-essential
apt-get install openssl
apt-get install libssl-dev
git clone https://github.com/firebase/scrypt.git
cd scrypt
autoreconf -i
./configure
make
echo `./scrypt {key} {salt base} {salt separator} {rounds} {memcost} -P <<< "password"`

You can use this tool for password hashing in the future.

At last we uploads the authentication info up to the new database. I cut out the thrid party provider fields because I’m not using them.

cut -d ',' -f 1-7,25-26 auth.csv > auth_cut.csv

echo "_id,email,isVerified,passwordHash,passwordSalt,name,photoUrl,createdAt,signedInAt" | cat - auth_cut.csv > auth_header.csv

$HOME/.../mongodb-osx-x86_64-4.0.0/bin/mongoimport --host <MONGO_HOSTNAME> --ssl --username <USERNAME> --password <PASSWORD> --db rutgersplus --collection auth --type csv --headerline --file auth_header.csv