Edit Page

.stream()

Stream records from your database one at a time or in batches, without first having to buffer the entire result set in memory.

await Something.stream(criteria)
.eachRecord(async (record, next)=>{
  return next();
});

Usage

	Argument	Type	Details
1	criteria		The Waterline criteria to use for matching records in the database.

Iteratee

Use one of the following:

.eachRecord(async (record, next)=>{ ... })
.eachBatch(async (batch, next)=>{ ... })

	Argument	Type	Details
1	record or batch	or	The current record, or the current batch of records. A batch will always contain at least one (and no more than thirty) records.
2	next		A callback function that the iteratee should invoke when it is finished processing the current record or batch. Like any Node callback, if your code in the iteratee calls `next()` with a truthy first argument (conventionally an Error instance), then Waterline understands that to mean an error occurred, and that it should stop processing records/batches. Otherwise, it is assumed that everything went according to plan.

Errors

Name	Type	When?
UsageError		Thrown if something invalid was passed in.
AdapterError		Thrown if something went wrong in the database adapter.
Error		Thrown if anything else unexpected happens.

See Concepts > Models and ORM > Errors for examples of negotiating errors in Sails and Waterline.

When should I use this?

The .stream() method is almost exactly like .find(), except that it does not actually provide a second argument to the .exec() callback nor does it provide it as a result. Instead, you use .eachRecord() or .eachBatch() to provide an iteratee function which receives one record or batch at a time.

This is useful for working with very large result sets; the kinds of result sets that might overflow your server's available RAM... at least, they would if you tried to hold the entire thing in memory at the same time. You can use Waterline's .stream() method to do the kinds of things you might already be familiar with from Mongo cursors: preparing reports, moving large amounts of data from one place to another, performing complex transformations, or even orchestrating map/reduce jobs.

Examples

There are 4 examples below.

Basic usage

An action that iterates over users named Finn in the database, one at a time:

await User.stream({name:'Finn'})
.eachRecord(async (user, next)=>{

  if (Math.random() > 0.5) {
    return next(new Error('Oops!  This is a simulated error.'));
  }

  sails.log(`Found a user ${user.id} named Finn.`);
  return next();
}

return res.ok();

Generating a dynamic sitemap

An action that responds with a dynamically-generated sitemap:

// e.g. in an action that handles `GET /sitemap.xml`:

var sitemapXml = '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';

var RENDER_SITEMAP_XML_URL_EL = _.template(
  '<url>\n'+
  '  <loc><%= url %></loc>\n'+
  '  <lastmod><%= updatedAt %></lastmod>\n'+
  '<changefreq>monthly</changefreq>\n'+
  '</url>'
);

await BlogPost.stream()
.limit(50000)
.sort('title ASC')
.eachRecord(async (blogPost, next)=>{

  sitemapXml += RENDER_SITEMAP_XML_URL_EL({
    url: 'https://blog.example.com/' + blogPost.slug,
    updatedAt: blogPost.updatedAt
  });

  return next();
});

sitemapXml += '</urlset>';

return res.send(sitemapXml);

With `.populate()`

A snippet of a command-line script that searches for creepy comments from someone named "Bailey Bitterbumps" and reports them to the authorities:

// e.g. in a shell script

var numReported = 0;

await Comment.stream({ author: 'Bailey Bitterbumps' })
.limit(1000)
.skip(40)
.sort('title ASC')
.populate('attachedFiles', {
  limit: 3,
  sort: 'updatedAt'
})
.populate('fromBlogPost')
.eachRecord(async (comment, next)=>{

  var isCreepyEnoughToWorryAbout = comment.rawMessage.match(/beanie weenies/) && comment.attachedFiles.length > 1;
  if (!isCreepyEnoughToWorryAbout) {
    return next();
  }

  await sails.helpers.sendTemplateEmail.with({
    template: 'email-creepy-comment-notification',
    templateData: {
      url: `https://blog.example.com/${comment.fromBlogPost.slug}/comments/${comment.slug}.`
    },
    to: '[email protected]',
    subject: 'Creepy comment alert'
  });

  numReported++;
  return next();
});

sails.log(`Successfully reported ${numReported} creepy comments.`);
return exits.success();

Batch-at-a-time

If we ran the code in the previous example, we'd be sending one email per creepy comment... which could be a lot! Not only would this be slow, it could mean sending thousands of individual API requests to our transactional email provider, quickly overwhelming our API rate limit.

Fortunately, there are a few easy changes we can make to our script to solve this. Let's try again; but this go-round, instead of processing individual records one at a time, we'll receive and process them as batches:

// e.g. in a shell script, batch at a time
var numReported = 0;
var numEmailsSent = 0;

await Comment.stream({
  author: 'Bailey Bitterbumps'
})
.limit(1000)
.skip(40)
.sort('title ASC')
.populate('attachedFiles', {
  limit: 3,
  sort: 'updatedAt'
})
.populate('fromBlogPost')
.eachBatch(async (someCreepyComments, next)=>{

  // If a comment contains the phrase "beanie weenies", AND it has
  // at least one attached file, then we'll consider it creepy.
  // Otherwise, it's not creepy enough to worry about, so we'll
  // remove it from the `someCreepyComments` array (effectively skipping it).
  _.remove(someCreepyComments, function (comment){
    var isCreepyEnoughToWorryAbout = comment.rawMessage.match(/beanie weenies/) && comment.attachedFiles.length > 1;
    if (!isCreepyEnoughToWorryAbout) { return true; }//<< not creepy enough, remove it.
    else { return false; }//<< this is creepy enough, keep it.
  });

  // If this batch doesn't contain any comments that are creepy enough,
  // then bail now and skip to the next batch, if any are left.
  if (someCreepyComments.length === 0) {
    return next();
  }//--•

  await sails.helpers.sendTemplateEmail.with({
    template: 'email-creepy-comment-digest',
    templateData: {
      urls: _.reduce(someCreepyComments, function (memo, creepyComment){
        memo += ' • ' + `https://blog.example.com/${creepyComment.fromBlogPost.slug}/comments/${creepyComment.slug}.` + '\n';
        return memo;
      }, '')
    },
    to: '[email protected]',
    subject: 'Creepy comment alert: daily digest',
  });

  numReported += someCreepyComments.length;
  numEmailsSent++;

  return next();

})//~∞%°

sails.log('Successfully reported '+numReported+' creepy comment(s)-- spread across '+numEmailsSent+' different emails.');
return exits.success();

Notes

This method can be used with await, promise chaining, or [traditional Node callbacks](https://sailsjs.com/documentation/reference/waterline-orm/queries/exe

Internally, regardless whether you're using .eachBatch() or .eachRecord(), Waterline grabs pages of 30 records at a time.

Just like async.eachSeries(), this method bails and throws an error (or calls its .exec() callback with an error) immediately upon receiving the first error from any iteratee.

.stream() runs the provided iteratee function on each record or batch, one at a time, in series.

Prior to Sails v1.0 / Waterline 0.13, this method had a lower-level interface, exposing a Readable "object stream". This was powerful, but tended to be error-prone. So the new, adapter-agnostic .stream() does not rely on emitters, or any particular flavor of Node streams. (Need to get it working the old way? Don't worry, with a little code, you can still easily build a streams2/streams3-compatible Readable "object stream" using the new interface.)

Read more about .stream() here, including additional examples, background information, and implementation details.

Is something missing?

If you notice something we've missed or could be improved on, please follow this link and submit a pull request to the sails-docs repo. Once we merge it, the changes will be reflected on the website the next time it is deployed.

Documentation

Reference

Built with Love

The Sails framework is built by a web & mobile shop in Austin, TX, with the help of our contributors. We created Sails in 2012 to assist us on Node.js projects. Naturally we open-sourced it. We hope it makes your life a little bit easier!

.stream()

Usage

Iteratee

Errors

When should I use this?

Examples

Basic usage

Generating a dynamic sitemap

With `.populate()`

Batch-at-a-time

Notes

Is something missing?

Documentation

Reference

Built with Love

Sails:

About:

Help:

.stream()

Usage

Iteratee

Errors

When should I use this?

Examples

Basic usage

Generating a dynamic sitemap

With .populate()

Batch-at-a-time

Notes

Is something missing?

Documentation

Reference

Built with Love

Sails:

About:

Help:

With `.populate()`