Category: javascript

Posted on

NPM Dependency Notation – idiots guide

Image from ChatGPT

Dependency notation in the package.json file influences how npm handles version installations, and this affects whether npm install may update the package-lock.json file.

Here’s how it works based on different notations:

Version Notations and Their Meanings:

  1. Exact Version (“4.19.2”):
  • Notation: “4.19.2”
  • Meaning: Install exactly version 4.19.2 of the package.
  • Effect on npm install:
  • When this version is specified, npm will always install version 4.19.2, regardless of whether newer minor or patch versions are available. The package-lock.json will not be updated unless you manually change the version in package.json.
  • Example: If version 4.19.3 is available, npm will not install it.


2. Caret (^) Notation (“^4.19.2”):

  • Notation: “^4.19.2”
  • Meaning: Install any compatible version according to semver rules, meaning any version >=4.19.2 and <5.0.0.
  • Effect on npm install:
  • With ^, npm allows updates to the minor and patch versions, but not the major version. This means if a newer patch version (like 4.19.3) or minor version (like 4.20.0) is available, npm will install it. If an update is installed, the package-lock.json will be updated to reflect the new version.
  • Example: If 4.20.1 is available, npm install will update package-lock.json to install 4.20.1.


3. Tilde (~) Notation (“~4.19.2”):

  • Notation: “~4.19.2”
  • Meaning: Install the most recent patch version that matches the specified minor version, meaning any version >=4.19.2 and <4.20.0.
  • Effect on npm install:
  • With ~, npm will allow updates to the patch version but not the minor version. If a newer patch version is available (e.g., 4.19.3), npm will install it, and the package-lock.json will be updated.
  • Example: If version 4.19.5 is available, npm will install that, but it will not install version 4.20.0.


4. Major Version (“4.19”):

  • Notation: “4.19”
  • Meaning: This implies “4.19.x”, which means install the latest available patch version within the 4.19.x range.
  • Effect on npm install:
  • This is similar to using ~4.19.0 but more permissive. It allows updates within the minor version and to any patch version (e.g., 4.19.2 → 4.19.5).
  • Example: If version 4.19.3 or 4.19.5 is available, npm will install it and update package-lock.json.


Impact on npm install and package-lock.json:

  • Exact version (“4.19.2”): No updates will occur unless you change the version manually in package.json.
  • Caret (^4.19.2) or Major version (“4.19”): Newer patch or minor versions will be installed automatically, and this will update package-lock.json with the exact version.
  • Tilde (~4.19.2): Only patch updates are allowed, and package-lock.json will reflect those updates when they occur.


How This Affects npm install:


If you use npm install (as opposed to npm ci), and the notation in your package.json allows for updates (like ^4.19.2), npm may install a newer version within the range, and as a result, the package-lock.json file will be updated with this new version.

On the other hand, if the package-lock.json specifies a version (say 4.19.2), and your package.json allows updates (^4.19.2), running npm install could still install a newer version (like 4.19.3), which would then update the lock file.

This flexibility is a double-edged sword: it allows for automatic updates of patches and minor versions, but if not managed well, it can lead to differences in installed versions across environments, which is why many teams prefer using npm ci in CI/CD pipelines for consistency.

Posted on

NPM install, ci and audit

Image created by ChatGPT

I needed to take a step back and fully understand this issue so I could explain it clearly to both new and experienced developers. The problem surfaced because our deployment pipelines run npm audit, which became a bottleneck in our process. We kept seeing the same vulnerabilities flagged repeatedly, even though they had been fixed multiple times.

Here we go, npm instal, ci and audit

Here’s an overview of the flow from installing a new package with npm to running npm install or npm ci in a pipeline, along with details on how vulnerabilities may resurface through npm audit.

Flow from Installing a Package to CI/CD

  1. Installing a Package:
  • When you install a new package locally (e.g., npm install package-name), npm adds the package to the node_modules directory and updates your package.json and package-lock.json files (or yarn.lock if you use Yarn).
  • package.json specifies the declared dependencies and their versions.
  • package-lock.json contains the exact versions of the installed packages and their entire dependency tree (including transitive dependencies). This ensures that everyone who installs your project gets the same versions of dependencies.

2. Pushing to Version Control:

  • Once you are satisfied with your code, including the new dependency, you push the changes to version control (e.g., Git). It’s important that both the package.json and package-lock.json files are committed to ensure consistency across environments.


3. Pipeline – npm install vs npm ci:

  • npm install:
    • During a build or deployment pipeline, running npm install will install dependencies based on the package.json and update the node_modules directory.
    • If a package-lock.json file exists, npm tries to install exact versions from the lock file, but if it detects any changes (e.g., new versions of dependencies or conflicts), it may update the lock file. (See dependency notation)
  • npm ci:
    • In a CI/CD pipeline, npm ci is preferred as it is faster and more deterministic.
    • It strictly adheres to the versions specified in package-lock.json. If any discrepancies (such as missing or extra dependencies) are found, the entire node_modules directory is deleted, and the exact dependencies from the package-lock.json are installed.
    • npm ci does not update package-lock.json, making it ideal for CI environments where reproducibility is critical.


4. npm audit:

  • During or after the install process, npm may run npm audit to check for security vulnerabilities in your dependencies. It compares the installed packages against a database of known vulnerabilities and flags any risks.
  • npm audit fix can automatically update vulnerable dependencies to the latest non-breaking versions (as defined by semver).


How Do npm audit Problems Reappear?

  1. Indirect Dependencies (Transitive Dependencies):
  • Most npm packages rely on other packages (dependencies of dependencies), and vulnerabilities often arise in these indirect dependencies.
  • Even if you’ve addressed an issue by updating your direct dependencies, some transitive dependencies may still have unresolved issues. This happens because they may not have yet released a fixed version.


2. New Vulnerabilities Discovered:

  • Sometimes, new vulnerabilities are discovered in packages that were previously considered safe. When npm’s vulnerability database is updated, a previously resolved issue may reappear if it’s related to a newly discovered flaw.


3. Out-of-Date Dependencies:

  • When the package-lock.json or a specific package hasn’t been updated for a while, and a vulnerability was later fixed in a newer version, your audit might flag the outdated dependency.
  • Running npm audit regularly (especially on pipelines) will catch such vulnerabilities, but sometimes an older transitive dependency may bring back the issue.


4. Partial Fixes:

  • Sometimes, packages release partial fixes, where only certain issues are resolved. If the fix doesn’t cover all security concerns, npm audit may still flag the package.


5. Conflicts Between Versions:

  • Certain updates may not be backward compatible with your project’s current environment or with other dependencies. This can lead to situations where you are unable to fully update vulnerable dependencies without breaking something else in your codebase.


Dealing with Persistent npm audit Problems:

  • Explicit Version Control: Sometimes you may have to manually control the versions in package-lock.json by using specific version ranges or resolutions (in tools like Yarn) to enforce the use of patched versions.
  • Selective Fixing: If you know a particular vulnerability doesn’t affect your project (e.g., it only impacts a feature you don’t use), you can audit it with exceptions.
  • Monitor Transitive Dependencies: Regularly check your dependency tree to monitor transitive dependencies and see if any have lagging versions. This can be done using tools like npm ls or through dependency-checking platforms.
Posted on

Fire and Forget

Yesterday was spent diagnosing what was reported as CORS issue in a Node service used for logging that was being called by front end apps.

CORS turned out to be a red herring, it ended up being the wrong format for a new database connection string, less than gracefully handling errors (it would just die and return 500), but what concerned me was the frontend request (sending a packet to a remote logging service) was not necessary to the successful sequence of editing and saving an object (plus it was sent prior to the event being successful) and would halt following requests.

The frontend fix was a “fire and forget” call to send a logging packet to the Node service, while not ideal if the service was down it would not effect frontend users experience.

const logEvent = (str) => {
  const send = async (str) => {
    // send the request, it might throw an error
    throw new Error(`thrown....${str}`)
  }

  send(str).catch(() => {
    // If you don't catch the error you can/will end up with a non handled exception error.
    console.log('It did not break, log error with service like Azure App Insights')
  })
}

console.log(
  logEvent("do nothing")
)
Posted on

NPM update

So, for a long time now our projects CI/CD pipelines have an audit step and will prevent merging if there and high risk issues. Nothing complicated just 

npm audit

Then a developer would need to sit down and follow their own approach to updating NPM packages. Mine was to install all the patch versions – test, minor versions and test. Then I’d take a stab at any major versions after reading any release notes.

See: https://docs.npmjs.com/about-semantic-versioning

It works, but it is a pain.

Then I found NPM GUI

https://www.npmjs.com/package/npm-gui

After installing it globally, I run the following command

npm-gui

This opens a browser tab and will first show all the global packages and versions. Navigate to your directory that has the package.json file, a nice table of installed node packages is displayed with required, current, compatible and latest versions. Clicking the versions will install them.

I still follow the same work flow of patches, minor and major versions but it takes me far less time and cognitive load to update projects.

It might not be perfect, but the rest of my team have started using this and its helped overcome the dread of package maintenance.

Posted on

More regEx in VSCode

I had a list of a few thousand words either 3, 4 or 5 letters, i needed to turn the imported list into an array.

const listFourLetterWords = [
ages
foot
good
hope
host
]

Using VSCodes finder with regEx it took moments to turn the 1000’s of words into

const listFourLetterWords = [
"ages",
"foot",
"good",
"hope",
"host",
]

Solution:

Find: \b[a-zA-Z]{4}\b

Replace: “$&”,

Posted on

Typescript and Mongoose (MongoDb) Discriminators

Many thanks to Manuel Maldonado and his excellent article https://hackernoon.com/how-to-link-mongoose-and-typescript-for-a-single-source-of-truth-94o3uqc. This post carries on from where he left off.

Discriminators – the missing step.

The current project at time of writing makes use of Mongoose’s schema inheritance mechanism. That enables you to have multiple models with overlapping schemas on top of the same underlying MongoDB collection. In other words a single collection that can have multiple models. Mostly this is a cost saving exercise. For more detail see.

https://mongoosejs.com/docs/discriminators.html

https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-mongoose#using-mongoose-discriminators-to-store-data-in-a-single-collection

How to add types to discriminators

Begin by reading Manuel’s article first!

You will see how to set up models, schemas, statics and methods that are strongly typed and play well with your IDE IntelliSense.

You are using Typescript?

You project is using Typescript and you have installed the following package?

npm i @types/mongoose

https://www.npmjs.com/package/@types/mongoose

Discriminators and Typescript

Here is the finished Base Model:

import { model, Schema } from "mongoose";

const collection = "GeneralPurpose";

const baseOptions = {
  discriminatorKey: "__type",
  collection,
  timestamps: true,
};

export const Base = model("Base", new Schema({}, baseOptions));

Here is the Base Model being used as the inherited model:

import { Document, Model, Types, Schema } from "mongoose";
import { Base } from "./Base.model";

export const MODEL_REF = "JournalBlock";

export interface JournalBlock {
  // _id?: string; // Note, _id is inherited from Document type
  blockTitle: string;
  icon: string;
  createdAt: string;
  updatedAt: string;
  parentId: string;
}

export interface JournalBlockDocument extends JournalBlock, Document {
  minify(): unknown;
}

export interface JournalBlockModel extends Model<JournalBlockDocument> {}

export const JournalBlockSchema: Schema = new Schema({
  blockTitle: { type: String, required: [true, "A title is required"] },
  icon: { type: String, required: [true, "A URL for the icon is required"]},
  introText: {
    type: String,
    required: [true, "A introduction text required"],
    },
  },
});

// Just to prove that hooks are still functioning as expected
JournalBlockSchema.pre("save", function () {
   console.log("PRE SAVE", this);
 }).post("delete", function () {
   console.log("post delete", this);
});

// Add a method. In this case change the returned object
JournalBlockSchema.methods.minify = async function (
  
  this: JournalBlockDocument

) {
  const response: JournalBlock & { _id: string } = {
    _id: this._id,
    icon: this.icon,
    introText: this.introText,
    createdAt: this.createdAt,
    updatedAt: this.updatedAt,
    parentId: this.parentId,
  };
  return response;
};


// This is the magic where we connect Typescript to the Mongoose inherited base model (discriminator)

export const JournalBlockModel = Base.discriminator<
  JournalBlockDocument,
  JournalBlockModel
>(MODEL_REF, JournalBlockSchema);

Use

Import the model into for code as normal. Your IDE will predict the returned Documents from queries like “.findById” including any additional methods or statics

Posted on

JS Sorting with .localCompare

I’ve been using Javascript for 2 years now and can not remember ever having to sort collections. 95% of what I am doing is backend services work and I try to off load as much data manipulation to the databases, but this week I have had 7 instances of having to sort data.

This caught me out:

For a given collection:

const collection = [ {name: "Dave"}, {name: "Donna"}, {name: "dave"}, {name: "Derek"}, {name: "Dave"},];

Sorted with:

collection.sort(
  (a,b) => { 
    if (a.name < b.name) return -1; 
    if (a.name > b.name) return 1; 
    return 0;
  })

Returns this:

[ { name: 'Dave' },  { name: 'Dave' },  { name: 'Derek' },  { name: 'Donna' },  { name: 'dave' } ]

Which is not what I need.

.localCompare to the rescue

collection.sort(
  (a, b) => a.name.localeCompare(b.name)
)

Returns this:

[ { name: 'dave' },  { name: 'Dave' },  { name: 'Dave' },  { name: 'Derek' },  { name: 'Donna' } ] 

But there is more

.localCompare method has additional options. For the same collection above, this 

collection.sort(
  (a, b) => a.name.localeCompare(b.name, 'en', { sensitivity: 'base' })
)

Returns this:

[ { name: 'Dave' },  { name: 'Dave' },  { name: 'dave' },  { name: 'Derek' },  { name: 'Donna' } ] 

Be aware that full implementation of .localCompare in Node.js is dependant on the version you are running.

Posted on

JSTypescript Debounce

Debounce simply controls how many times a function to be executed over time.

For example, when typing into a search box, you probably don’t want to be calling an external API every time the user presses a key, it is more likely you would wait until they have paused or have finished entering text. Then after second or 2 fetch the data.

If you run the example below f() is called multiple times

let i = 0;
const f = () => {
  console.log(`foo ${++i}`);
};
f(); // foo 1
f(); // foo 2
f(); // foo 3
f(); // foo 4
f(); // fo0 5

console.log(i); // 5

Now wrapping the function inside a debounce method only the last call is completed

const debounce = (
  fnc: (...params: any[]) => unknown,
  n: number,
  immediately: boolean = false
) => {
  let timer: number | undefined = undefined;
  return function (this: any, ...args: any[]) {
    if (timer === undefined && immediately) {
      fnc.apply(this, args);
    }
    clearTimeout(timer);
    timer = setTimeout(() => fnc.apply(this, args), n);
    return timer;
  };
};

let i = 0;
const f = debounce(() => console.log(`foo ${++i}`), 2000);
f(); 
f();
f();
f();
f(); // foo 1

console.log(i) // 0

To recap; If a user keeps clicking on a button it is only after there has been a pause will the last click be carried out.

Throttling, is very similar to debounce but this time it is the FIRST click that is actioned and subsequent clicks are ignored until a period of time has passed.

Posted on

Quick1: JS funky date requirement

The problem is:

Take a date object, set the time to 09:15 the following day. If that day is a Saturday, Sunday or Monday bump the date to the next Tuesday.

Solution:

const reminderTime = (date: Date): Date => {
  const response = new Date(date);
  
  const SUNDAY = 0;
  const MONDAY = 1;
  const TUESDAY = 2;
  const SATURDAY = 6;
  
  response.setDate(response.getDate() + 1);
  
  if ([SUNDAY, MONDAY, SATURDAY].some((x) => x === response.getDay())) {
    response.setDate(response.getDate() + ((TUESDAY + 7 - response.getDay()) % 7));
  }

  response.setHours(9);
  response.setMinutes(15);

  return response;
};

Is the date local or UTC? That depends on how the Date object passed into the method was created. In either case local or UTC is preserved.

Here are some proofs:

console.log(reminderTime(new Date(2020, 10, 2))); 
// Returns Tue Nov 03 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 10, 3)));
// Returns Wed Nov 04 2020 09:15:00 GMT+0000 (Greenwich Mean Time)

console.log(reminderTime(new Date(2020, 10, 4))); 
// Returns Thu Nov 05 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 10, 5)));
// Returns Fri Nov 06 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 10, 6)));
// Returns Tue Nov 10 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 10, 7)));
// Returns Tue Nov 10 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 10, 8)));
// Returns Tue Nov 10 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 10, 9)));
// Returns Tue Nov 10 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 11, 31)));
// Returns Fri Jan 01 2021 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 1, 29)));
// Returns Tue Mar 03 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 

console.log(reminderTime(new Date(2020, 1, 30)));
// Returns Tue Mar 03 2020 09:15:00 GMT+0000 (Greenwich Mean Time) 
Posted on

JS Dates

Dates in JS are difficult and it is easy to reach for an external library like Moment.

But is that true? No, but there are a few gotchas.

There are two “times” we need to concern ourselves with, these are.

  • Local Time and Coordinated Universal Time (UTC). Local time refers to the time zone your computer is operating on.
  • UTC is kinda like Greenwich Mean Time and from what dates and times relate to.

Unless you are doing something really funky the date object uses local time unless you explicitly request UTC. HOWEVER, there is one exception to this. That is creating a date with a string.

Create a date with a string

new Date("2020-11-04")

This is method uses a string to set the date. Pretty straight forward but creating a date with no time is a UTC date/time.

The problem is if your computer is operating in a time zone ahead or behind UTC the date could be out by a day.

SOLUTION 1: Always include hours and minutes as a minimum.

SOLUTION 2: Do not create date objects with strings, use a different method.

Create a date with arguments

new Date(2020, 7, 19, 8, 33)

Creating a date object by using arguments. The above has a value of :

Wed, 19 Aug 2020 08:33 // Local time!

If you look closely it is not really intuitive. August is the 8th month but is represented by the number 7. That is because in Javascript is zero indexed and January is 0 …. December is 11. Not much you can do about it so do not stress and just get on with it. Treat it as a feature, and you wont end up with a UTC time when you wanted a local time.

If you are wondering how to create a UTC date with arguments:

new Date(Date.UTC(2020, 11, 10)); // 10th November 2020 <whatever time it is now>

Create a date without arguments or string

new Date()

Will create a date object with the current local date and time.

Create a date with a timestamp

new Date(1604762586100); // Sat Nov 07 2020 15:23:06 GMT+0000

You can also create a date object using a timestamp. In JavaScript, a timestamp is the amount of milliseconds elapsed since 1 January 1970. I do not remember ever using this method.

Your can get the current timestamp with:

Date.now();

That should be pretty straight forward. The key takeaways are:

  • Know that you are creating a local or UTC date object
  • Maybe avoid creating a date object with a string
  • Remember that months start at 0. January = 0 , December = 11