Deploy Rocket.Chat Development Server

Using Lightsail on Ubuntu 20.04

Normally, 4 GiB instance size should be enough. During both initial and subsequent meteor npm start, 2 GiB instance size is not enough, so it needs at least 4 GiB for meteor npm start. After meteor npm start is successful, usage is 95% of 3.8 G, but can go down into 75%. With 4 GiB, Rocket.Chat frequently crashes the whole server when restarting due to file change, especially when VS Code Remote-SSH is used.

EC2 m6g.medium 4 GiB (ap-southeast-1) + 20 GiB is $37.44 (on-demand pricing). Lightsail 4 GiB is $20, 8 GiB is $40. EC2 r6g.medium 8 GiB (ap-southeast-1) + 20 GiB is $46.78 (on-demand pricing).

The recommendation is to create a temporary 8 GiB Lightsail instance just for initial build, then rsync /home/ubuntu into the normal 4 GiB instance. Although if you want to run code-server or gitpod on it for convenient editing experience, you’ll need more than 4 GiB.

# Update Ubuntu
sudo apt update && sudo apt full-upgrade -y
byobu-enable

Reference: https://docs.rocket.chat/guides/developer/quick-start

Login as ubuntu user
Install tools required: (pkg-config etc. is required since 3.8.0 for canvas)
sudo apt install -y g++ build-essential git curl python3-minimal pkg-config
sudo apt install -y build-essential libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev
Install meteor:
curl https://install.meteor.com/ | sh
(Under some circumstances, you may need to install a specific (older) release of Meteor instead of the latest, always check the .meteor/release file of the Github code repository to determine if you need to do this before you install meteor)
There is no need to install node or npm, as meteor already includes them. Verify by:
meteor node -v
meteor npm -v
Get rocket.chat code from soluvas’s soluvas-3.x branch:
git clone https://github.com/soluvas/Rocket.Chat.git -b soluvas-3.x
(you may want to fork the code on Github first, and then clone your fork)
This will clone into folder /home/ubuntu/Rocket.Chat
Install modules
cd Rocket.Chat
meteor npm install
Set environment variables: (save it as ~/setenv.sh — you need to source it, not execute it!)

export MONGO_OPLOG_URL="mongodb+srv://xxx.mongodb.net/local?retryWrites=true&w=majority"
export MONGO_OPTIONS="{\"ssl\":true}"
export MONGO_URL="mongodb+srv://xxx.mongodb.net/soluvas_chat_prod?retryWrites=true&w=majority"
export Push_enable_gateway=0
export Push_gcm_api_key="AAAA..."
export Push_gcm_project_number="420xxx"
export ROOT_URL="https://chat.soluvas.com"

Start building (the first build can take 10 or more minutes, and you may see various warnings or minor errors — please be patient; subsequent dev builds after the first will be 5 minutes or less)

source ~/setenv.sh
meteor npm start

During testing environment, Lightsail needs to open port 3000.

Add nginx Reverse Proxy

sudo apt -y install nginx

Create self-signed cert

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt
# set common name as *.soluvas.com

Edit /etc/nginx/sites-available/chat.soluvas.com.conf

# Upstreams
upstream backend {
    server 127.0.0.1:3000;
}

# HTTPS Server
server {
    listen 443;
    server_name chat.soluvas.com;

    # You can increase the limit if your need to.
    client_max_body_size 200M;

    error_log /var/log/nginx/rocketchat.access.log;

    ssl on;
    ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
    ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # don’t use SSLv3 ref: POODLE

    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Nginx-Proxy true;

        proxy_redirect off;
    }
}

sudo ln -sv /etc/nginx/sites-available/chat.soluvas.com.conf /etc/nginx/sites-enabled/chat.soluvas.com.conf
sudo systemctl restart nginx

In Lightsail, allow firewall on port HTTPS (TCP 443).

Add A record to Lightsail instance IP address, and proxied.

In Cloudflare, set SSL to Full (not strict).

Visual Studio Code Connect Using Remote – SSH

Note: With 4 GiB of RAM and Rocket.Chat running, VS Code Remote-SSH will take some memory on the server and crash it.

To enable VSCode to watch files, please apply the following system tweak to the server.

Meteor also recommends this: https://github.com/meteor/docs/blob/master/long-form/file-change-watcher-efficiency.md

Troubleshooting: UnhandledPromiseRejectionWarning: Error: ENOSPC: System limit for number of file watchers reached

Solution:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Swap File

Perhaps adding 1 GiB swap to 4 GiB VM may make it more resilient to out-of-memory crashes?

Reference: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-swap-file/

sudo dd if=/dev/zero of=/swapfile bs=128M count=8
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon -s

6. Enable the swap file at boot time by editing the /etc/fstab file.

Open the file in the editor:

$ sudo vi /etc/fstab

Add the following new line at the end of the file, save the file, and then exit:

/swapfile swap swap defaults 0 0

Updating Rocket.Chat Development Instance

Update Ubuntu:

sudo apt update && sudo apt -y full-upgrade

Stop meteor npm start first, otherwise it will reload and crash Ubuntu due to insufficient memory.

Update Meteor:

meteor update

Merge stable tag:

git pull
git fetch RocketChat
# get commit ID from GitHub tag page
git merge COMMIT
git commit

Install dependencies:

meteor npm start

In the previous step, make sure there are no errors.

Source environment variables configuration then start Rocket.Chat:

source ~/setenv.sh
meteor npm start

Troubleshooting: Error: `gyp` failed with exit code: 1

Install canvas development dependencies.

Try:

rm -rf ~/.node-gyp
rm -rf node_modules

Then try again.

Troubleshooting:

../src/Canvas.cc:542:85: error: no matching function for call to ‘v8::Value::ToObject()’
542 | Context2d *context2d = Nan::ObjectWrap::Unwrap(context->ToObject());

Reference: https://github.com/Automattic/node-canvas/issues/1490

Infrastructure