Welcome to Anbo's Blog.

Migrating Firebase to Self-hosted Mongodb

I recently have to migrate Rutgers Plus from Firebase to my self-hosted mongodb (Atlas) because of the Chinese great firewall.

First install gsutil, firebase-tools and mongo shell.

To migrate data collections one at a time, create and run a bash file:

echo "Exporting from firebase.."
firebase database:get /$1 --project <PROJECT_NAME> --pretty > $1.json
echo "Switching id..."
awk '$0 ~ /^  "/ {split($0, a, "\""); print "\{ " "\"_id\": " " \"" a[2] "\","; next}{print}' $1.json > $1_tmp.json
echo "Converting to array..."
awk 'NR==1{print "[";next}/^}/{print "]";next}{print}' $1_tmp.json > $1_array.json
echo "Uploading to mongodb..."
$HOME/.../mongodb-osx-x86_64-4.0.0/bin/mongoimport --host <MONGO_HOSTNAME> --ssl --username <USERNAME> --password <PASSWORD> --db rutgersplus --collection $1 --jsonArray --file $1_array.json

where $1 will be the name of the collection. In the file, I converted the id from firebase’s “id as the key” format to mongodb export’s “id as the value” format. Also mongoexport doesn’t strictly follow the JSON standard as it requires an array without comma separators instead of an object with id-as-keys. I did the tranformation as well.

To download static files from Firebase for re-uploads:

gsutil cp -r gs://<PROJECT_NAME>.appspot.com/users .

To migrate users, find password hash parameters in the Firebase authentication tab, you get {key} {salt separater} {rounds} {memcost} there. Download the users info csv:

firebase auth:export --project <PROJECT_NAME> auth.csv

and mark the column right before the first name and last names of the users. Those are {salt base} for each user.

We need the same algorithm firebase used for hashing passwords to keep using the existing password hashes, or we need to ask the users to reset their passwords or authenticate with firebase in the future to migrate users one at a time. These two methods are also the official methods from AWS sources.

Luckily after some decent researching, I learnt that Firebase kindly provides their modified SCRYPT hashing algorithm on Github. Therefore, on the server:

apt-get install autoconf
apt-get install build-essential
apt-get install openssl
apt-get install libssl-dev
git clone https://github.com/firebase/scrypt.git
cd scrypt
autoreconf -i
./configure
make
echo `./scrypt {key} {salt base} {salt separator} {rounds} {memcost} -P <<< "password"`

You can use this tool for password hashing in the future.

At last we uploads the authentication info up to the new database. I cut out the thrid party provider fields because I’m not using them.

cut -d ',' -f 1-7,25-26 auth.csv > auth_cut.csv

echo "_id,email,isVerified,passwordHash,passwordSalt,name,photoUrl,createdAt,signedInAt" | cat - auth_cut.csv > auth_header.csv

$HOME/.../mongodb-osx-x86_64-4.0.0/bin/mongoimport --host <MONGO_HOSTNAME> --ssl --username <USERNAME> --password <PASSWORD> --db rutgersplus --collection auth --type csv --headerline --file auth_header.csv

MORE...

Using Ace Editor With React

React is a modern front end framework, and Ace editor is a popular javascript based code editor. However, things can get complicated in the process of integrating the two since Ace editor is not designed to work with react. “brace”, the browserify compatible version of Ace editor, and “react-ace”, the “brace” editor embedded in a react component, can be used to smooth the process. I noted down the procedure as follows:

In the project folder, install the node version of Ace editor “brace”, and the “react-ace” package that wrap the “brace” editor as a react component:

npm install brace --save
npm install react-ace --save

require the “brace” and “react-act” packages, together with a “sh” syntax highlighting feature and “chrome” theme from “brace” package:

var brace = require('brace');
var AceEditor = require('react-ace');
require('brace/mode/sh');
require('brace/theme/chrome');

inside the component, return the editor in for rendering:

<AceEditor
  mode="sh"
  theme="chrome"
  name="code"
  width="100%"
  maxLines={50}
  ref="ace"
  fontSize={18}
  value="#type your code here"
  editorProps={{$blockScrolling: Infinity}}
  onLoad={(editor) => {
    editor.focus();
    editor.getSession().setUseWrapMode(true);
  }}
/>

There are two ways to address auto focusing and line wrapping: either by receiving a parameter from onLoad event and call the ace editor native methods (as shown), or by setting a ref="ace" attribute within the AceEditor tag, and then call method like this.refs.ace.editor.focus at “componentDidMount” lifecycle.

Also, setting width: 100% will allow the editor scale with the container, however, setting

height: 100%

will just disable the editor. The work around would be setting a large value of maxLines (for example: 500) and add

editorProps={{$blockScrolling: Infinity}}

attribute to the editor.

Ace editor will allow performing undo actions until the editor is in it’s original empty state, even though the content of the editor is loaded immediately when the component initialize. To fix this, remove the undo history right after the component mounts:

var undo_manager = this.refs.ace.editor.getSession().getUndoManager();
undo_manager.reset();
this.refs.ace.editor.getSession().setUndoManager(undo_manager);

MORE...

Multiple Meteor instances on one server with nginx

As a good full stack platform as it is, meteor doesn’t have explicitly documentation for customization of the environment underneath.

I’m about to start coding a web application with an idea of starting with a core user API first, and slowly work on developing a series of extensions around the user API. This means I’ll have a single database queried and maintained by multiple meteor apps. In addition, all of the meteor apps will be set up in a single VPS to save money for my body’s digestive system.

I disregarded the meteor self-contained databases and install a mongodb of my own.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

In the shell:

mongo
use [database]
db.createUser({ user: "<name>",
  pwd: "<cleartext password>",
  roles: [
    { role: "<role>", db: "<database>" } | "<role>",
    ...
  ]
})

to create a new database and a new user.
I use mup for deploying multiple Meteor app on the server. Mup is installed on the local machine through npm. In the same Meteor project folder on the local machine:
mup init
then add the environment variable in mup.json:

"PORT": 300X
"MONGO_URL": "mongodb://[username]:[password]@localhost:27017/[database]"

and change “setupMongo” to false.
This way, meteor will run in separate ports connect to the database set by the environment instead of its own.

After that, create A records for subdomains in the domain zone file, and point them all to the server’s IP address.

Install nginx to take meteor’s place of dealing with http requests. In /etc/nginx/site-avaliable/, make new sites and put proxy_pass configurations in:

server {
    listen  80;
    server_name subdomain.doamin.com;
    root  /home/user/meteorapp;
    location / {
        proxy_pass http://domain.com:300X;
    }
}

Finally,

nginx -s reload

and starting all the Meteor instances to see multiple Meteors running in harmony.

EDIT: specify “setupMongo: true” will automatically have multiple Meteor instances share the same mongodb.

MORE...

Reloading Code by Restarting Mod_wsgi’s Daemon Process

In order to refresh pages without restarting apache, hosting the web application inside the a Mod_wsgi daemon process and restarting the process once the code is changed.
I’m quoting the official documentation from official Mod_wsgi site to here:

…if you are using Django in daemon mode and needed to change your ‘settings.py’ file, once you have made the required change, also touch the script file containing the WSGI application entry point. Having done that, on the next request the process will be restarted and your Django application reloaded.

Also,

If you are not sure whether you are using embedded mode or daemon mode, then substitute your WSGI application entry point with:

def application(environ, start_response):
    status = '200 OK'

    if not environ['mod_wsgi.process_group']:
      output = 'EMBEDDED MODE'
    else:
      output = 'DAEMON MODE'

    response_headers = [('Content-Type', 'text/plain'),('Content-Length', str(len(output)))]

    start_response(status, response_headers)

    return [output]

If your WSGI application is running in embedded mode, this will output to the browser ‘EMBEDDED MODE’. If your WSGI application is running in daemon mode, this will output to the browser ‘DAEMON MODE’.

In order to use daemon mode, a “WSGIDaemonProcess” attribute must be specified in apache’s configuration referring to my previous post.

MORE...

My setup of a python3 django environment in Ubuntu 14.04

First setup a few essentail packages that is needed:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential apache2 python3-dev mysql-server
sudo apt-get install python-pip libapache2-mod-wsgi-py3
sudo pip install virtualenv
sudo pip install virtualenvwrapper

login to mysql server and

CREATE DATABASE django;

To make virtualenvwrapper starts automatically, add

source /usr/local/bin/virtualenvwrapper.sh

to .bashrc;
Make django virtual environment:

mkvirtualenv --python=/usr/bin/python3 django
workon django
pip install django
pip install mysqlclient
django-admin startproject django

create folder static and media under the project folder;
Modify settings.py to add STATIC_ROOT=os.path.join(BASE_DIR, ‘static’);
change the database settings:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'django',
        'USER': 'root',
        'PASSWORD': 'passwd',
        'HOST': 'localhost',
    }
}

migrate the admin static file:

./manage.py collectstatic

modify the apache conf:

vi /etc/apache2/sites-enabled/000-default.conf

make it looks like:

<VirtualHost *:80>
    ServerName www.example.com
    WSGIDaemonProcess django python-path=/home/zhouanbo/django/:/home/zhouanbo/.virtualenvs/django/lib/python3.4/site-packages
    WSGIProcessGroup django
    WSGIScriptAlias / /home/zhouanbo/django/django/wsgi.py
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
    Alias /robots.txt /home/zhouanbo/django/static/robots.txt
    Alias /favicon.ico /home/zhouanbo/django/static/favicon.ico
    Alias /media/ /home/zhouanbo/django/media/
    Alias /static/ /home/zhouanbo/django/static/
    <Directory /home/zhouanbo/django/static>
        Require all granted
    </Directory>
    <Directory /home/zhouanbo/django/media>
        Require all granted
    </Directory>
    <Directory /home/zhouanbo/django/ideas>
        <Files wsgi.py>
            Require all granted
        </Files>
    </Directory>
</VirtualHost>

restart the apache service:

service apache2 restart

and django should be good to go.

MORE...

Intersection and subtraction of two data files

The efficiency of file processing in Linux environment has a strong positive correlation to how large your brain capacity is to remember common commands. So I write down commands to release brain space.

Intersection of records:

grep -F -f a.txt b.txt | sort | uniq

Subtraction of records:

grep -F -v -f a.txt b.txt | sort | uniq
# or:
grep -F -v -f b.txt a.txt | sort | uniq

MORE...

Jquery Mouse Events

In jquery mouse events, “.mouseover” and “.mouseleave” can be accomplished equivalently by setting up the CSS “:hover” property.

However, this site involves using one element detects the mouse event, while another element is displaying the effect when the event is detected. This prevents the usage of pure CSS because it’s lack of logical operations. Therefore I achieved this in jquery by applying show and fade effects on on element and “.mouseover” and “.mouseleave” event on another, which allows separation of detection and effect displaying.

However, there is another event – “.mouseenter” in addition to “.mouseover”.
Using “.mouseover” will cause unpredictable latency in detection of the mouse leaving the object while using “.mouseenter” resolves that.

I speculate jquery rewrote the “.mouseenter” and “.mouseleave” events on top of the original javascript’s API, and used original javascript’s “.mouseover” event as-is, which caused the two methods’ event detection mechanisms to differ and lead to inconsistent results.

MORE...