Engineering

⌘K
  2. Home
  4. Docs
  6. Engineering
  8. NestJS

NestJS

We use NestJS to create business logic services and workers (excluding repositories, which are implemented using Strapi).

Logging

Use winston. We only use limited feature set like setting log levels, because we follow The Twelve-Factor App methodology, the log routing should be handled by the deployment environment.

For format, when possible, using JSON structured logging is probably a good idea. See: https://medium.com/unomaly/logging-wisdom-how-to-log-5a19145e35ec.

The logs eventually need to go into Elasticsearch.

For container environment like AWS Fargate or Kubernetes, you can use Filebeat, or maybe pipe AWS CloudWatch Logs to Elasticsearch somehow.

For AWS Lambda, I think you need to configure AWS CloudWatch Logs to pipe to Elasticsearch, which will then be visible in Kibana.

Other Logging Libraries and Why You Should Know Them

Other alternatives: visionmedia/debug, which is very easy to get started and has sane and useful defaults, especially for client apps.

yarn add debug
yarn add --dev @types/debug

The problem with debug is, it only has one level. You can’t have error/info/trace log. A lot of people use debug with chalk to give the log messages a colorful visual hint, but I still think granular log level is critical. (source)

Some libraries, including bpmn-engine, uses debug. So you should know how to configure it, e.g. using environment variable DEBUG=bpmn-engine:*.

Another alternative is log4js (see also: IBM’s article). It has very good architecture, however, half as popular as winston, and rarely discussed by the community. Since winston is still flexible and has active development, and usable both for server-side and client-side apps, we use winston to simplify organization-level guidelines.

Creating NestJS Project with Nx

Reference: Building API’s with NestJS and Nrwl Nx (5 Part Series)

  1. Introduction to Building API’s with NestJS and Nrwl Nx
  2. Set up and configure a new Nx Workspace
  3. Add a NestJS API to a Nx Workspace
  4. Add GraphQL to a NestJS API in a Nx Workspace
  5. Deploy a NestJS API to Heroku from a Nx Workspace
yarn add -D @nrwl/nest
nx generate @nrwl/nest:app myapp

Start The Dev Server

nx serve myapp

How To Load Environment Variables from .env

The nx serve command has built-in support for apps/myapp/.env file, all you need to do is put that file. You do not have to put this in apps/myapp/src/main.ts:

import { config } from 'dotenv';
config({path: 'apps/myapp/.env', debug: true});

Build The App

Reference: https://stackoverflow.com/a/62157813/122441

# 'development' environment
nx build myapp
# 'production' environment
nx build --prod myapp

Debugging NestJS Projects with Auto Attach in VS Code

Enable Smart (opens in a new tab)” rel=”noreferrer noopener” class=”rank-math-link”>Auto Attach > Smart. You will get automatic debugging support when launching nx serve.

Implementing GraphQL Query & Mutation Resolvers

See NestJS Docs > GraphQL + TypeScript > Code first.

app.module.ts:

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { GraphQLModule } from '@nestjs/graphql';
import { UsersModule } from './users/users.module';
import { ConfigModule, ConfigService } from '@nestjs/config';
// import { TypeOrmModule } from '@nestjs/typeorm';
import { AuthModule } from './auth/auth.module';
import { TypegooseModule } from 'nestjs-typegoose';
import { Logger } from 'mongodb';
import { EmploymentsResolver, Employment } from './employments/employments.resolver';
import { CulturesResolver, Culture } from './cultures/cultures.resolver';
import { CommonModule } from './common/common.module';

@Module({
  imports: [
    ConfigModule.forRoot(),
    TypegooseModule.forRootAsync({
      imports: [ConfigModule.forRoot()],
      inject: [ConfigService],
      useFactory: async (configService: ConfigService) => ({
        uri: configService.get<string>('V4_MONGODB_URI'),
        dbName: configService.get<string>('V4_MONGODB_DATABASE'),
        useNewUrlParser: true,
        useUnifiedTopology: true,
      }),
    }),
    TypegooseModule.forFeature([Employment, Culture]),
    UsersModule,
    GraphQLModule.forRoot({
      installSubscriptionHandlers: false,
      // See: https://stackoverflow.com/a/61048144/122441
      autoSchemaFile: process.env.NODE_ENV === 'development' ? 'apps/miluv-profile/schema.gql' : true,
    }),
    AuthModule,
    CommonModule,
  ],
  controllers: [AppController],
  providers: [AppService, EmploymentsResolver, CulturesResolver],
})
export class AppModule {
  public async init() {
    Logger.setLevel('debug');
  }
}

Simple Mutation resolver:

import { Args, Field, InputType, Mutation, ObjectType, Resolver } from '@nestjs/graphql';

@InputType()
class AuthenticateUserInput {
  @Field({description: 'Email to be authenticed'})
  email: string;
  @Field({description: 'User\'s password'})
  password: string;
}

@ObjectType()
class AuthenticateUserPayload {
  @Field()
  id?: string;
}

@Resolver()
export class AuthResolver {

  @Mutation(returns => AuthenticateUserPayload, {
    description: 'Authenticates a user'})
  async authenticateUser(@Args('input') input: AuthenticateUserInput): Promise<AuthenticateUserPayload> {
    return { id: 'bismillah' };
  }

}

Sync-ing GraphQL Schema in Postman

For some reason, Postman needs GraphQL schema to be loaded manually instead of refreshing schema directly from GraphQL server.

In Postman, go to APIs sidebar, then create/update your GraphQL schema’s name, by copy-paste-ing the contents of apps/myapp/schema.gql that is generated by NestJS. Now you can use that GraphQL API/schema in a Request.

Generate GraphQL Client SDKs

See: Generate GraphQL Client SDKs

Fastify instead of Express

Warning: Fastify + GraphQL is currently unusable due to nestjs/graphql#1205.

Reference: Performance (Fastify) – NestJS

Add @nestjs/platform-fastify package:

yarn remove @nestjs/platform-express apollo-server-express @types/express
yarn add @nestjs/platform-fastify
# if you use GraphQL
yarn add apollo-server-fastify

Edit src/main.ts:

import { config } from 'dotenv';
config();

import { NestFactory } from '@nestjs/core';
import { FastifyAdapter, NestFastifyApplication } from '@nestjs/platform-fastify';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create<NestFastifyApplication>(
    AppModule, new FastifyAdapter(), {
    logger: ['debug', 'log', 'error', 'warn', 'verbose'],
  });
  // const appModule = app.get(AppModule);
  // await appModule.init();
  await app.listen(parseInt(process.env.PORT || '3001'));
}
bootstrap();

In Fastify, it’s called FastifyReply instead of Express’s Response:

import { FastifyReply } from 'fastify';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Post('generatePdf')
  async generatePdf(@Body() data: CertificateData, @Res() res: FastifyReply) {
    if (!data?.awardeeName) {
      throw new HttpException('data is required', 400);
    }
    const doc = await generateCertificate(data);
    res.header('Content-Type', 'application/pdf');
    // doc.pipe(res);
    const buffers = [];
    doc.on('data', data => buffers.push(data));
    doc.on('end', () => {
      const buf = Buffer.concat(buffers);
      console.info(`Rendered PDF size is ${buf.length} bytes`);
      res.send(buf);
    });
    doc.end();
  }

}

JWT Authentication with GraphQL & Subscriptions Support

Reference: nestjs/graphql#48

Additional dependency we need is @nestjs/jwt:

yarn add @nestjs/jwt

Create Auth module:

yarn nest g module auth

Define auth/fusionauth-payload.ts:

/**
 * Holds information about current user.
 */
export class FusionAuthPayload {
  aud: string;
  exp?: number;
  lat?: number;
  iss: string;
  sub: string;
  jti: string;
  authenticationType: string;
  email: string;
  email_verified: boolean;
  preferred_username: string;
  applicationId: string;
  roles: string[];
}

Create JwtAuth service:

yarn nest g service JwtAuth auth
import { Injectable, UnauthorizedException } from '@nestjs/common';
import { JwtService } from '@nestjs/jwt';
import { FusionAuthPayload } from '../fusionauth-payload';

@Injectable()
export class JwtAuthService {
  constructor(private jwtService: JwtService) {}

  // async validateUser(email: string, password: string): Promise<LoginResponse> {
  //   const user = await this.usersRepository.findOne({ email });
  //   if (!user) {
  //     throw 'user not found';
  //   }
  //   if (user.password === password) {
  //     const token = this.jwtService.sign({
  //       userId: user.id,
  //       email: user.email,
  //     });
  //     return { user, token };
  //   }
  // }

  async validateToken(token: string): Promise<FusionAuthPayload> {
    try {
      console.debug('Verifying JWT token:', token, '...');
      let payload: FusionAuthPayload;
      try {
        // First we check using public key
        payload = this.jwtService.verify(token);
      } catch (e) {
        // If fails, then check using shared secret
        payload = this.jwtService.verify(token, {secret: process.env.JWT_SECRET});
      }
      console.debug('JWT payload:', payload);
      return payload;
    } catch (e) {
      console.warn('Invalid JWT token:', e);
      throw new UnauthorizedException(`Invalid JWT token: ${e}`);
    }
  }
}

Create RolesAuth guard:

yarn nest g guard RolesAuth auth
import { CanActivate, ExecutionContext, Injectable, BadRequestException, UnauthorizedException } from '@nestjs/common';
import { Observable } from 'rxjs';
// import { GqlExecutionContext } from '@nestjs/graphql';
import { JwtAuthService } from './jwt-auth/jwt-auth.service';
import { Reflector } from '@nestjs/core';
import { FusionAuthPayload } from './fusionauth-payload';
import { FastifyRequest } from 'fastify';
import { GqlExecutionContext } from '@nestjs/graphql';

export const ROLE_SALES_MANAGER = 'Sales Manager';
export const ROLE_FINANCE_MANAGER = 'Finance Manager';

/**
 * Allow only specified roles:
 * @SetMetadata('roles', ['Sales Manager'])
 */
@Injectable()
export class RolesAuthGuard implements CanActivate {
  constructor(private reflector: Reflector,
    private readonly jwtAuthService: JwtAuthService) {}

  getRequest(context: ExecutionContext) {
    // const ctx = GqlExecutionContext.create(context);
    // return ctx.getContext().req;
    // console.debug('context:', context);
    // context.getType() can be 'http' or 'graphql'
    if ((context.getType() as string) === 'graphql') {
      return GqlExecutionContext.create(context).getContext().req;
    } else {
      return context.switchToHttp().getRequest();
    }
  }

  async canActivate(context: ExecutionContext): Promise<boolean> {
      const req: FastifyRequest = this.getRequest(context);
    // console.debug('req=', req);
    const authHeader = req.headers.authorization as string;
    const accessTokenQuery = (req.query as any).accessToken;
    let token: string;
    if (authHeader) {
      const [type, headerToken] = authHeader.split(' ');
      if (type !== 'Bearer') {
        throw new BadRequestException(`Authentication type \'Bearer\' required. Found \'${type}\'`);
      }
      token = headerToken;
    } else if (accessTokenQuery) {
      token = accessTokenQuery;
    } else {
      throw new BadRequestException('Authorization header or accessToken query parameter required.');
    }

    let user: FusionAuthPayload;
    try {
      user = (req as any).user = await this.jwtAuthService.validateToken(token);
    } catch (e) {
      throw new UnauthorizedException(`Cannot validate JWT token: ${e}`);
    }

    // get from 'roles' metadata in method (want to fallback to class, but how?)
    const allowedRoles = this.reflector.get<string[]>('roles', context.getHandler());
    if (allowedRoles) {
      const intersection = allowedRoles.filter(_ => (user.roles || []).includes(_));
      if (intersection.length >= 1) {
        return true;
      } else {
        throw new UnauthorizedException(`${context.getClass().name}.${context.getHandler().name} requires ${allowedRoles}, but user '${user.preferred_username}' only has these roles: ${user.roles}`);
      }
    } else {
      const msg = `No role is allowed by ${context.getClass().name}.${context.getHandler().name}`;
      console.warn(msg);
      throw new UnauthorizedException(msg);
    }
  }
}

Configure JwtModule in auth/auth.module.ts:

import { Module } from '@nestjs/common';
import { JwtModule } from '@nestjs/jwt';
import pemtools from 'pemtools';
import { JwtAuthService } from './jwt-auth/jwt-auth.service';

let publicKey: string | Buffer | null = process.env.JWT_PUBLIC_KEY;
if (process.env.JWT_PUBLIC_KEY && !process.env.JWT_PUBLIC_KEY.startsWith('-----')) {
  publicKey = Buffer.from(process.env.JWT_PUBLIC_KEY, 'base64');
  // See: https://github.com/nestjs/jwt/issues/447
  publicKey = pemtools(publicKey, 'PUBLIC KEY').toString();
}
// console.debug('publicKey=', publicKey);

@Module({
  imports: [
    JwtModule.register({
      publicKey,
      // only one of public key or secret can be used at a time:
      // secret: process.env.JWT_SECRET,
    })
  ],
  providers: [JwtAuthService],
  exports: [JwtAuthService],
})
export class AuthModule {}

You’ll need to provide JWT_PUBLIC_KEY (recommended for FusionAuth access token for server-side verification, as it can be verified without knowing a shared secret), or JWT_SECRET, or both (with tweak).

To use the guard, you must put 'roles' metadata on the method (not the class):

@UseGuards(RolesAuthGuard)
@Resolver(of => User)
export class UsersResolver {
...
  @Query(returns => [User])
  @SetMetadata('roles', [ROLE_SALES_MANAGER])
  async users(...

JWT Authentication using @nestjs/passport (not recommended)

See nestjs/graphql#48 for discussion why using @nestjs/passport is more trouble than it’s worth.

Reference: NestJS > Authentication > JWT functionality

Example implementation is in lovia-billing.

Add passport, passport-jwt, @nestjs/passport, @nestjs/jwt packages:

yarn add passport passport-jwt @nestjs/passport @nestjs/jwt

Generate module auth:

nest g module auth

Create auth/jwt-user.ts that holds information of current user (optional if you don’t need the current user, because you can just use the JWT payload):

/**
 * Holds information about current user.
 */
export class JwtUser {
  idV4: string;
  username: string;
  ssoId?: string;
  email?: string;
  roles: string[];
  userIsFinanceManager: boolean;
}

Create src/jwt.strategy.ts that will use ExtractJwt:

import { Injectable } from "@nestjs/common";
import { PassportStrategy } from "@nestjs/passport";
import { ExtractJwt, Strategy } from "passport-jwt";
import { JwtUser } from "./jwt-user";

@Injectable()
export class JwtStrategy extends PassportStrategy(Strategy) {
  constructor() {
    super({
      // Bearer header only, ignore URL query parameter:
      // jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(),
      jwtFromRequest: ExtractJwt.fromExtractors([
        ExtractJwt.fromAuthHeaderAsBearerToken(),
        ExtractJwt.fromUrlQueryParameter('accessToken')
      ]),
      ignoreExpiration: false,
      // either HMAC shared secret, or PEM-encoded RSA public key
      secretOrKey: process.env.JWT_SECRET || process.env.JWT_PUBLIC_KEY,
    });
  }

  /**
   * Convert JWT payload into our own structure... or you can just return the payload.
   * @param payload 
   */
  async validate(payload: any): Promise<JwtUser> {
    const user = {
      idV4: payload.sub,
      username: payload.username,
      //ssoId: 
      email: payload.email,
      roles: payload.roles || [],
      // helper
      userIsFinanceManager: (payload.roles || []).includes('Finance Manager'),
    };
    console.debug('JWT user:', user);
    return user;
  }

}

Now configure PassportModule, JwtModule with JWT_SECRET_OR_KEY environment variable, and use our JwtStrategy in auth/auth.module.ts:

import { Module } from '@nestjs/common';
import { PassportModule } from '@nestjs/passport';
import { JwtModule } from '@nestjs/jwt';
import { JwtStrategy } from './jwt.strategy';

@Module({
  imports: [
    PassportModule,
    JwtModule.register({
      secret: process.env.JWT_SECRET,
      publicKey: process.env.JWT_PUBLIC_KEY,
      signOptions: {},
    })
  ],
  providers: [JwtStrategy],
})
export class AuthModule {}

Create the JwtAuth guard inside auth module:

yarn nest g guard JwtAuth auth

Implement auth/jwt-auth.guard.ts by just extending AuthGuard:

import { Injectable } from '@nestjs/common';
import { AuthGuard } from '@nestjs/passport';

@Injectable()
export class JwtAuthGuard extends AuthGuard('jwt') {
}

Now the guard is ready to use.

To use the guard from your controller or controller methods, add @UseGuards(JwtAuthGuard) and access the payload (or returned value of JwtStrategy.validate()) using req.user:

import { FastifyRequest } from 'fastify';

...

  @UseGuards(JwtAuthGuard)
  @Get('profile')
  getProfile(@Request() req: FastifyRequest) {
    const user = (req as any).user as JwtUser;
    return user;
  }

Generate JWT Token

yarn global add njwt
# to enable node to require yarn global modules, add the following to ~/.bash_profile
export NODE_PATH=$(yarn global dir)/node_modules
node
const jwt = require('njwt');
let claims = { sub: 'finance', username: 'finance', roles: ["Finance Manager"] };
let token = jwt.create(claims, 'top-secret-phrase');
token.setExpiration(new Date().getTime() + 60*1000*60*24*365*100);
token.compact();

Now you can use generated token in Postman by adding Bearer token, as part of HTTP request header:

Authorization: Bearer YOUR_TOKEN_HERE

Reference: https://developer.okta.com/blog/2018/11/13/create-and-verify-jwts-with-node

Deployment

Deploy NestJS-Fastify with Serverless Framework on AWS Lambda

Before doing deploying Fastify to AWS Lambda, make sure you’ve converted your NestJS project to Fastify (above).

Prepare & Test Serverless Framework Locally

Reference:

yarn global add serverless

Add aws-lambda-fastify package:

yarn add @types/aws-lambda aws-lambda aws-lambda-fastify

Add to src/global.d.ts:

declare module 'aws-lambda-fastify';

Create src/lambda.ts:

import { Context } from 'aws-lambda';
import awsLambdaFastify from 'aws-lambda-fastify';
import { AppModule } from './app.module';
import { FastifyAdapter, NestFastifyApplication } from '@nestjs/platform-fastify';
import { fastify } from 'fastify';
import { NestFactory } from '@nestjs/core';

let cachedProxy: (event: any, context: Context) => Promise<Response>;

async function bootstrap(): Promise<(event: any, context: Context) => Promise<Response>> {
  const fastifyApp = fastify();

  const app = await NestFactory.create<NestFastifyApplication>(
    AppModule, new FastifyAdapter(fastifyApp));
  await app.init();

  return awsLambdaFastify(fastifyApp, { binaryMimeTypes: ['application/octet-stream', 'application/pdf'] });
}

export async function handler(event: any, context: Context): Promise<Response> {
  if (!cachedProxy) {
    const proxy = await bootstrap();
    cachedProxy = proxy;
  }

  return cachedProxy(event, context);
}

Serverless plugins: serverless-plugin-typescript.

yarn add --dev serverless-plugin-typescript serverless-offline

You’ll get error: messageText: “Option ‘–incremental’ can only be specified using tsconfig, emitting to single file or when option --tsBuildInfoFile is specified.”, so edit tsconfig.json and add:

    "tsBuildInfoFile": ".tsbuildinfo"

Create serverless.yml:

service:
  name: certificate2pdf

package:
  exclude:
    - dist/**
  include:
    - templates/**

plugins:
  - serverless-plugin-typescript
  # to transpile and minify your code
  # - serverless-plugin-opttimize
  # to be able to test your app offline
  - serverless-offline
  # - serverless-domain-manager
  # - serverless-plugin-warmup

provider:
  name: aws
  runtime: nodejs12.x
  stage: ${opt:stage, 'dev'} # Set the default stage used. Default is dev
  region: ${opt:region, 'ap-southeast-1'} # Overwrite the default region used. Default is us-east-1
  # RAM usage is only about 150 MiB, but need bigger RAM to speed up CPU to avoid timeout
  memorySize: 2048 # Overwrite the default memory size. Default is 1024
  timeout: 15 # The default is 6 seconds. Note: API Gateway current maximum is 30 seconds
  apiGateway:
    # minimumCompressionSize: 1024 # Compress response when larger than specified size in bytes (must be between 0 and 10485760)
    binaryMediaTypes: # Optional binary media types the API might return
      - application/octet-stream
      - application/pdf

functions:
  main: # The name of the lambda function
    # The module 'handler' is exported in the file 'src/lambda'
    handler: src/lambda.handler
    events:
      - http:
          method: any
          path: /{any+}

# custom:
#   # Enable warmup on all functions (only for production and staging)
#   warmup:
#     enabled: true

Try locally:

# You'll need to either export required environment variables, or use dotenv package
export JWT_SECRET=...
serverless offline start

Troubleshooting & Logging

serverless logs -f main --tail -s dev
serverless logs -f main --tail -s production

GraphQL schema.gql issue

Error: EROFS: read-only file system, open ‘schema.gql’

Solution: https://stackoverflow.com/a/61048144/122441

TL;DR: in src/app.module.ts:

    GraphQLModule.forRoot({
      installSubscriptionHandlers: false,
      // See: https://stackoverflow.com/a/61048144/122441
      autoSchemaFile: process.env.NODE_ENV === 'development' ? 'schema.gql' : true,
    }),
...

Deploy to AWS Lambda with Custom Domain & SSL Certificate

Reference: https://www.serverless.com/blog/serverless-api-gateway-domain

  1. Request SSL certificate using AWS ACM, make sure to use wildcard domain + naked domain.
yarn add --dev serverless-domain-manager

Add in plugins, and also custom.customDomain configuration in serverless.yml:

plugins:
  # ...
  - serverless-domain-manager

custom:
  customDomain:
    domainName: certificate2pdf.talentiva.net
    # certificateName: '*.talentiva.net'
    # Since this API is used only internally by ERPNext, save bandwidth & time by using Regional endpoint instead of Edge
    endpointType: regional
    # basePath: ''
    stage: ${self:provider.stage}
    createRoute53Record: false
#   # Enable warmup on all functions (only for production and staging)
#   warmup:
#     enabled: true
~/.yarn/bin/serverless create_domain -s production

Then deploy again:

~/.yarn/bin/serverless deploy -s production

Use Cloudflare/DNS Manager to add CNAME record for the custom domain.

Testing API Webhooks/Callbacks

It’s recommended to use Gitpod whenever possible.

Otherwise, when developing locally you can use ngrok to get a public HTTPS endpoint.

ngrok http --region=ap 3005

Articles

How can we help?