Section 7: MongoDB, Mongoose, and REST APIs (Todo API)


Install MongoDB and Robomongo

Download MongoDB from https://www.mongodb.com/download-center?jmp=nav#community

Location of Installation

During installation, click custom to get where the MongoDB is installed before going back to complete.

Alt text


Make the first connection to MongoDB

Create a directory in the USER\\[YOUR ACCOUNT] called mongo-data. This is where your database is located.

C:\Program Files\MongoDB\Server\4.0\bin is the path of MongoDB. Use cd command to reach it.

Run command mongod.exe --dbpath /Users/x85gao/mongo-data to make a connection.


MongoDB manager

Open another prompt. Go to the mongo bin directory: cd "\Program Files\MongoDB\Server\4.0\bin"

Run mongo.exe

Example of command interface of mongoDB

> db.Todos.insert({text: 'create a new todo'})
WriteResult({ "nInserted" : 1 })
> db.Todos.find()
{ "_id" : ObjectId("5b6633942830b398d6a86773"), "text" : "create a new todo" }

This is for ensuring the connection only. We will talk more about this later.


Install Robomongo

A GUI version of mongo.exe
Go to robomongo.org/download to download it. Install robomongo


Connection on robomongo

The default address and port do not need to be changed. You can edit the name as you wish.

Alt text


View Data on robomongo

Alt text



Building a NoSQL Vocabulary

SQL NO SQL
Database Database
Table Collection
Row/Record Document
Column Field



Connecting to Mongo and Writing Data

Help Node Connect MongoDB: https://github.com/mongodb/node-mongodb-native

API: http://mongodb.github.io/node-mongodb-native/3.1/api/
DOC: http://mongodb.github.io/node-mongodb-native/3.1/


Create New Project

Create a new directory called: node-todo-api
Run npm init
Create a sub-directory called: playground
Create a file called mongodb-connect.js inside playground

Install the mongodb native library npm install mongodb --save


Example in mongodb-connect.js

const MongoClient = require('mongodb').MongoClient;
MongoClient.connect("mongodb://localhost:27017", (err, client)=>{
if (err) {
return console.log('Unable to connect to MongoDB server');
}
console.log('Connected to MongoDB server');
client.close(); // close the connection with MongoDB server
})
$ node playground/mongodb-connect.js
(node:13608) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
Connected to MongoDB server

We will get the above message about deprecation. Just ignore it for now.


No TodoApp Database Shown

Now we go back to robomongo, but realize that TodoApp is not shown here.

Alt text

Mongo is not going to create a new database until we create data in it.


Example with Connection + Creating Document

MongoClient.connect("mongodb://localhost:27017", (err, client)=>{
if (err) {
return console.log('Unable to connect to MongoDB server');
}
console.log('Connected to MongoDB server');
const db = client.db('TodoApp'); // client -> database
db.collection('Todos').insertOne({ // database -> db
userName: 'Jude'
}, (err, res) => {
if (err) {
return console.log('Unable to insert todo', err);
}
console.log(JSON.stringify(res.ops, undefined, 2)); // ops stores the created document
})
client.close(); // close the connection with MongoDB server
})

Another example with Collection (USER)

db.collection('Users').insertOne({
name: 'Jude',
age: 19,
location: 'London'
}, (err, res) => {
if (err) return console.log('Unable to insert User', err);
console.log(JSON.stringify(res.ops, undefined, 2));
})

Verify whether actually added or not

Alt text



The ObjectId

Alt text

We are going to understand this _id.


[
{
"name": "Jude",
"age": 19,
"location": "London",
"_id": "5b663cb9ceceda376cb97b1f"
}
]
[
{
"name": "Jude",
"age": 19,
"location": "London",
"_id": "5b663ef99ec6d63624df5e17"
}
]

This _id is not incremented by 1.


Components of Object ID

12 byte value:


What happens if we provide _id field in an object

db.collection('Users').insertOne({
_id: 123,
name: 'Jude',
age: 19,
location: 'London'
}

This is perfectly legal! but …

[
{
"_id": 123,
"name": "Jude",
"age": 19,
"location": "London"
}
]

The synthesized ID will be overridden if you have a _id field in your object of creation.

Alt text


Extract Time Stamp from an ObjectId

Print out the _id: console.log(res.ops[0]._id);
Get the time stamp: res.ops[0]._id.getTimestamp()

Connected to MongoDB server
2018-08-05T00:13:45.000Z

Make an ObjectId Ourselves

Object Destructoring (ES6)

Object destructoring lets you pull out property of an object to make it as a variable.
Example:

var user = {name: 'Jude', age: 19};
var {name} = user; // name = user.name;

Example:

const MongoClient = require('mongodb').MongoClient;
const {MongoClient} = require('mongodb');

The above statements are identical.


const {MongoClient, ObjectID} = require('mongodb');

extracts multiple properties from require('mongodb').


Make an Object ID

var obj = new ObjectID();
console.log(obj);


Fetching Data

Make up at least two data in todos

Alt text


Query All

db.collection('Todos').find().toArray() // returns a PROMISE
.then((docs)=>{
console.log('Todos');
console.log(JSON.stringify(docs, undefined, 2));
}, (err)=>{
console.log('Unable to fetch todos', err);
})

We will get

Todos
[
{
"_id": "5b66431ba5a5f118dd195f13",
"text": "go to accomud8u",
"completed": false
},
{
"_id": "5b664332a5a5f118dd195f1c",
"text": "study m235",
"completed": false
}
]

Selective Query

Alt text

What to do if we want only to have documents with completed being true/ false?

db.collection('Todos').find({completed: false}) // provide a QUERY key-val pair
.toArray()
.then((docs)=>{
console.log('Todos');
console.log(JSON.stringify(docs, undefined, 2));
}, (err)=>{
console.log('Unable to fetch todos', err);
})

The output would be

Todos
[
{
"_id": "5b664332a5a5f118dd195f1c",
"text": "study m235",
"completed": false
}
]

containing only ones with completed : true.


Query by _id

Grab an ID: 5b664332a5a5f118dd195f1c, e.g.

.find({_id: new ObjectID("5b664332a5a5f118dd195f1c")})

ObjectID is not a String, so we need to use its constructor to construct one by a String.


Cursor

Cursor is what comes back from db.collection().find()
.toArray is an example to use the Cursor.

Count Cursor

Example:

db.collection('Todos').find()
.count()
.then((count)=>{
console.log(`Todos count: ${count}`);
}, (err)=>{
console.log('Unable to fetch todos', err);
})

We would get Todos count: 2.

There are more Cursors.


Exercise

Alt text

Given this collection, query only Jude.
Solution:

db
.collection('Users')
.find({name: 'Jude'})
.toArray()
.then((docs)=>console.log(docs), (err)=>console.log(err));


Deleting Documents


deleteMany

Alt text
What to do to delete all documents with text being play.

db
.collection('Todos')
.deleteMany({
text: "play"
})
.then((res)=>console.log(res), (err)=>console.log(err));

.deleteMany returns a promise.
The success case has a callback taking a res. Let’s see what is in that.

The most important information is at the very top: result: { n: 3, ok: 1 }.


deleteOne

It works exactly the same as deleteMany, but it only deletes the first one it sees satisfying the criteria.

Example:

db.collection('Todos').deleteOne({text: 'go to gym'}).then((res)=>console.log(res), (err)=>console.log(err));

We have the same result as well result: { n: 1, ok: 1 }.


findOneAndDelete

Example:

db.collection('Todos').findOneAndDelete({completed: false}).then((res)=>console.log(res), (err)=>console.log(err));

Output:

{ lastErrorObject: { n: 1 },
value:
{ _id: 5b664332a5a5f118dd195f1c,
text: 'study m235',
completed: false },
ok: 1 }


Updating Data

findOneAndUpdate

findOneAndUpdate(filter, update, options) returns a promise.

Get an ID, e.g. 5b664a17a5a5f118dd19628f.

MongoDB Update Operators

$set: Sets the value of a field in a document.
There are a lot more operators!
https://docs.mongodb.com/manual/reference/operator/update/

Options

returnOriginal: false allows you to obtain the new one after updating it.


Example:

db.collection('Todos').findOneAndUpdate({
_id: new ObjectID('5b664a17a5a5f118dd19628f')
}, {
$set: {
completed: true
}
}, {
returnOriginal: false
}).then((res)=>console.log(res), (err)=>console.log(err));

Output:

{ lastErrorObject: { n: 1, updatedExisting: true },
value:
{ _id: 5b664a17a5a5f118dd19628f,
text: 'finish node.js',
completed: true },
ok: 1 }

Incrementing a Field by 1

What to do to increment an integer field by 1?

db.collection('Users').findOneAndUpdate({
_id: new ObjectID('5b663cb9ceceda376cb97b1f')
}, {
$inc: {
age: 1
}
}, {
returnOriginal: false
}).then((res) => console.log(res), (err) => console.log(err));

CRUD is done! Creating. Reading. Updating. Deleting!



Mongoose

Check the website out: https://www.npmjs.com/package/mongoose


Install Mongoose

npm i mongoose --save


Connection to MongoDB

mongoose.connect("mongodb://localhost:27017/TodoApp");


Specify Promise Library

mongoose.Promise = global.Promise;


Create a Model

To organize documents in a collection, that is, to limit the flexibility of variety of fields one document can be in a collection, we define a Model.

Example:

var Todo = mongoose.model('Todo', {
text: {
type: String
},
completed: {
type: Boolean
},
completedAt: {
type: Number
}
});

Create an Instance

var newTodo = new Todo({
text: ‘Cook dinner’
});


Save to Database

newTodo.save().then((doc)=>{console.log("saved todo", doc)}, (e)=>console.log('unable to save todo'));


Finalize Everything

var newTodo = new Todo({
text: 'Cook dinner',
completed: true,
completedAt: 0 // talk about it later
});
newTodo.save().then((doc)=>{console.log("saved todo", doc)}, (e)=>console.log('unable to save todo'));


Validators, Types, and Defaults

Schemas: http://mongoosejs.com/docs/guide.html


Example:

var Todo = mongoose.model('Todo', {
text: {
type: String,
required: true, // avoid no string given
minlength: 1, // avoid empty string
trim: true // remove leading and trailing spaces
},
completed: {
type: Boolean,
default: false
},
completedAt: {
type: Number,
default: null
}
});

This whole big JSON is called a Schema.


Exercise: User Model

The only field of User is email. Set type to be a String. Set minlength to be 1. Create a new User.

Solution:

var User = mongoose.model('User', {
email: {
type: String,
required: true,
minlength: 1, // talk about it later
trim: true
}
});
var user = new User({email: 'x85gao@edu.uwaterloo.ca'})


Install Postman

Get downloaded from https://www.getpostman.com/apps
Install it!


First Try

GET https://google.com

Alt text

The body here is an html page but usually we will interact with JSON.


GET JSON

GET https://maps.googleapis.com/maps/api/geocode/json?address=uwaterloo

Alt text

Successfully, we get the JSON body back!



Resource Creation Endpoint - POST /todos

Install body-parser, express.

Final Code

const {mongoose} = require('./db/mongoose')
var {Todo} = require('./models/todo');
var {User} = require('./models/user');
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
app.use(bodyParser.json()); // middleware
app.post('/todos', (req, res) => {
console.log(req.body); // body is from body-parser
var todo = new Todo({
text: req.body.text
});
todo.save().then((doc)=>{
res.send(doc);
},(e)=>{
res.status(400).send(e);
})
})
app.listen(3000, () => console.log('started on port 3000'));


Testing POST /todos

npm i expect mocha nodemon supertest --save-dev


server.test.js

Export app
module.exports = {app};

Final Test Suite

const expect = require('expect');
const request = require('supertest');
const {app} = require('./../server');
const {Todo} = require('./../models/todo');
beforeEach((done)=>{
Todo.remove({}).then(() => done()); // remove everything before testing
}); // sth to do before tests run
describe('POST /todos', ()=>{
it('should create a new todo', (done) => {
var text = 'Test to do text';
request(app) // from supertest
.post('/todos') // supertest post request to app
.send({text}) // with this JSON
.expect(200) // expecting status code to be 200
.expect((res) => { // expect callback provides a response object
expect(res.body.text).toBe(text); // res.body is from body-parser and text is a field of that JSON
})
.end((err, res)=>{ // for testing Database
if (err) { // from above tests
return done(err);
}
Todo.find().then((todos)=>{ // Model.find() returns an array representing a collection
expect(todos.length).toBe(1);
expect(todos[0].text).toBe(text);
done();
}).catch((e) => done(e));// length & text not matching
});
});
});

Testing Boundary

it('should not create todo with invalid body data', (done) => {
request(app)
.post('/todos')
.send({})
.expect(400)
.end((err, res)=>{
if (err) {
return done(err);
}
Todo.find().then((todos)=>{
expect(todos.length).toBe(0);
done();
}).catch((e) => done(e));
});
});


List Resources - GET /todos

app.get('/todos', (req, res) => {
Todo.find().then((todos)=>{res.send({todos})},(e)=>res.status(400).send(e));
})


Testing GET /todos

It is problematic to have

beforeEach((done)=>{
Todo.remove({}).then(() => done()); // remove everything before testing
}); // sth to do before tests ru

to test GET /todos.


Add a fake todos array:

const todos = [{
text: 'first test todo'
}, {
text: 'second test todo'
}];

And a modified beforeEach

beforeEach((done) => {
Todo.remove({}).then(()=>{Todo.insertMany(todos).then(()=>done())})});

Now this is going to have a problem for the old tests.
Since the counter should be incorrect, it is based on an empty database.

In the first test, we change it to

Todo.find({text}).then((todos)=>{ // Model.find() returns an array representing a collection
expect(todos.length).toBe(1);
expect(todos[0].text).toBe(text);
done();

In the second,

Todo.find().then((todos)=>{
expect(todos.length).toBe(2);
done();
}).catch((e) => done(e));

Test Suite

describe('GET /todos', () => {
it('should get all todos', (done) => {
request(app)
.get('/todos')
.expect(200)
.expect((res) => {
expect(res.body.todos.length).toBe(2);
})
.end(done);
})
})


Mongoose Queries and ID Validation

Grab an ID. var id = '5b66723df4b00c30f4cbf12e'; We will query by this ID.
Mongoose enables us to have a String ID, not an Object ID.

Example:

Todo.find({
_id: id
}).then((todos)=>{
console.log('Todos', todos);
})

findOne returns only one document it sees first, or null.

If you are sure that the item is unique by your query, recommend to use findOne since it returns an object instead of an array.


findById returns one document, given an ID string.

Todo.findById(id).then((todo) => console.log('Todos', todo)).catch((e)=>console.log(e));


What happens if id not correct?
var id = '6b66723df4b00c30f4cbf12e'; // tweaked a bit

Usually, it just returns null.

Let’s handle it!

Todo.findById(id).then((todo) => {
if(!todo) {
return console.log('id not found', todo)
}
console.log('Todos', todo)
}).catch((e)=>console.log(e));

If the id is completely illegal or invalid, you will receive a big error message from e.

Validate an Object ID

const {ObjectID} = require('mongodb');
var id = '6b66723df4b00c30f4cbf12e';
ObjectID.isValid(id);

Example:

const {User} = require('../server/models/user')
var id = '5b665cc29bd88e362cd79577';
if (!ObjectID.isValid(id)) {
return console.log('Object ID is invalid');
}
User.findById("").then((user) => {
if (!user) {
return console.log('User not Found');
}
console.log('User:', user);
}).catch((e) => {
console.log(e);
})


Getting an Individual Resource - GET /todos/:id

How to fetch a variable passed as an URL?

From req.params.id;


Final Code

app.get('/todos/:id', (req, res) => {
var id = req.params.id;
if (!ObjectID.isValid(id)) {
return res.status(404).send();
}
Todo.findById(id).then((todo) => {
if (!todo) {
return res.status(404).send();
}
res.send({todo});
}).catch((e) => {
res.status(400).send();
})
})


Testing GET /todos/:id

Modifies todos array to include Object ids.

Before doing so, please do not forget to require an Object ID.

const todos = [{
_id: new ObjectID(),
text: 'first test todo'
}, {
_id: new ObjectID(),
text: 'second test todo'
}];

Writing a test case for success:

describe('GET /todos/:id', () => {
it('should return todo back', (done) => {
request(app)
.get(`/todos/${todos[0]._id.toHexString()}`)
.expect(200)
.expect((res) => {
expect(res.body.todo.text).toBe(todos[0].text);
})
.end(done);
});
});

Now, practice for tests:

Solution:

it('should return 404 if todo not found', (done) => {
request(app)
.get(`/todos/${new ObjectID().toHexString()}`)
.expect(404)
.end(done);
});
it('should return 404 for non-object IDs', (done) => {
request(app)
.get('/todos/123')
.expect(404)
.end(done);
})


Deploy API to Heroku

Things to change for Heroku:
As usual, we must change the port

const port = process.env.PORT || 8000;

Tell the Heroku how to start the project:

...
"engines": {
"node": "6.2.2"
},
...
"scripts": {
"start": "node server/server.js",

Now set up a Heroku project using heroku create

Set up a real MongoDB database from mLab.

Change your mongoDB URL to your own.

git heroku push publishes your project.

heroku logs displays your server logs.

heroku open opens up the application in your browser.

Note: when dealing with mLab, use User that is created on mLab, not the User of your mLab account.



Postman Environments

Create two environments.

Alt text

Alt text


Change Path to Support Dynamic Variable