Tag: javascript

Posted on

NPM install, ci and audit

Image created by ChatGPT

I needed to take a step back and fully understand this issue so I could explain it clearly to both new and experienced developers. The problem surfaced because our deployment pipelines run npm audit, which became a bottleneck in our process. We kept seeing the same vulnerabilities flagged repeatedly, even though they had been fixed multiple times.

Here we go, npm instal, ci and audit

Here’s an overview of the flow from installing a new package with npm to running npm install or npm ci in a pipeline, along with details on how vulnerabilities may resurface through npm audit.

Flow from Installing a Package to CI/CD

  1. Installing a Package:
  • When you install a new package locally (e.g., npm install package-name), npm adds the package to the node_modules directory and updates your package.json and package-lock.json files (or yarn.lock if you use Yarn).
  • package.json specifies the declared dependencies and their versions.
  • package-lock.json contains the exact versions of the installed packages and their entire dependency tree (including transitive dependencies). This ensures that everyone who installs your project gets the same versions of dependencies.

2. Pushing to Version Control:

  • Once you are satisfied with your code, including the new dependency, you push the changes to version control (e.g., Git). It’s important that both the package.json and package-lock.json files are committed to ensure consistency across environments.


3. Pipeline – npm install vs npm ci:

  • npm install:
    • During a build or deployment pipeline, running npm install will install dependencies based on the package.json and update the node_modules directory.
    • If a package-lock.json file exists, npm tries to install exact versions from the lock file, but if it detects any changes (e.g., new versions of dependencies or conflicts), it may update the lock file. (See dependency notation)
  • npm ci:
    • In a CI/CD pipeline, npm ci is preferred as it is faster and more deterministic.
    • It strictly adheres to the versions specified in package-lock.json. If any discrepancies (such as missing or extra dependencies) are found, the entire node_modules directory is deleted, and the exact dependencies from the package-lock.json are installed.
    • npm ci does not update package-lock.json, making it ideal for CI environments where reproducibility is critical.


4. npm audit:

  • During or after the install process, npm may run npm audit to check for security vulnerabilities in your dependencies. It compares the installed packages against a database of known vulnerabilities and flags any risks.
  • npm audit fix can automatically update vulnerable dependencies to the latest non-breaking versions (as defined by semver).


How Do npm audit Problems Reappear?

  1. Indirect Dependencies (Transitive Dependencies):
  • Most npm packages rely on other packages (dependencies of dependencies), and vulnerabilities often arise in these indirect dependencies.
  • Even if you’ve addressed an issue by updating your direct dependencies, some transitive dependencies may still have unresolved issues. This happens because they may not have yet released a fixed version.


2. New Vulnerabilities Discovered:

  • Sometimes, new vulnerabilities are discovered in packages that were previously considered safe. When npm’s vulnerability database is updated, a previously resolved issue may reappear if it’s related to a newly discovered flaw.


3. Out-of-Date Dependencies:

  • When the package-lock.json or a specific package hasn’t been updated for a while, and a vulnerability was later fixed in a newer version, your audit might flag the outdated dependency.
  • Running npm audit regularly (especially on pipelines) will catch such vulnerabilities, but sometimes an older transitive dependency may bring back the issue.


4. Partial Fixes:

  • Sometimes, packages release partial fixes, where only certain issues are resolved. If the fix doesn’t cover all security concerns, npm audit may still flag the package.


5. Conflicts Between Versions:

  • Certain updates may not be backward compatible with your project’s current environment or with other dependencies. This can lead to situations where you are unable to fully update vulnerable dependencies without breaking something else in your codebase.


Dealing with Persistent npm audit Problems:

  • Explicit Version Control: Sometimes you may have to manually control the versions in package-lock.json by using specific version ranges or resolutions (in tools like Yarn) to enforce the use of patched versions.
  • Selective Fixing: If you know a particular vulnerability doesn’t affect your project (e.g., it only impacts a feature you don’t use), you can audit it with exceptions.
  • Monitor Transitive Dependencies: Regularly check your dependency tree to monitor transitive dependencies and see if any have lagging versions. This can be done using tools like npm ls or through dependency-checking platforms.
Posted on

What they never tell you about GitHub

GitHub is brilliant…. There are loads of tutorials, video and articles out there, recently I’ve spent time filling gaps in my Git version control knowledge and something struck me as odd. Every tutorial should make it clear right from the start and reminders throughout their presentations that you

DO NOT STORE PASSWORDS, KEYS, USERNAMES, CREDENTIALS, ETC. ON GIT HUB

Why?

Because there are bastards out there who are scanning git repositories for anything that could help access secured areas or consume resources they should not.

Git does scan your repo’s and will warn you but its not in real time so be warned. Still not convinced? The image is of a free Amazon Web Service account that was set up as a learning exercise the keys where saved in a Git Repo and over the course of 3 days a free account had racked up nearly $10,000US .

What can you do, if you accidentally commit a password, key, etc? Delete the branch? Roll back?

No!

The best advice and this is also given by Git themselves is to consider your data compromised and your accounts that use the data hacked. Once again, rolling back and deleting does not work.

So reset your passwords/key etc and don’t just add a 1 at the end or swap out vowels for 12345.

How do you manage password and environment values?

Have a Google for git .gitignore or visit dotIgnore on GitHub

Posted on

JavaScript fetch()

In a previous post I was refreshing my memory with vanilla JavaScript  after years of jQuery-ing, especially AJAX request and the

XMLHttpRequest()

My eyes rolled when I was reading more about it then  I found fetch()  ( I know its been about since 2015 and now almost universally adopted).

Fetch does what XMLHttpRequest does but in a more elegant way and DOES NOT need additional libraries as it is bundled with JavaScript. It also makes use of “promises” .

const url = "https://randomuser.me/api/?results=10";

fetch(url)
.then(function(response) {
 return response.json();
})
.then(function(data) {
 console.log(data.results);
})
.catch(function() {
 console.log("Booo");
});

The above code queries the URL, then parses the string as JSON, then does something with the data.

It also catches errors.

Good eh?

Its looks to be easy to expand and do all manner of fetching. For example submitting a form

var form = new FormData(commentForm);
fetch( url, { method: "POST", body: form })

You can then chain .then() promises to get the desired outcomes.

For more information

A quick introduction video from Google

From Google Working with the Fetch API

Jake Archibald’s post Thats so Fetch

Scotch post The Fetch API

Posted on

Javascript AJAX without jQuery

I learnt this stuff years ago but with the rise of jQuery I like many others had become lazy. Add the fact that so many browsers handled things slightly different going down the library route had its benefits.

It is 2017 and things have improved many jQuery(ish) approaches have been adopted by Javascript and browsers have become more similar.

So here goes. First off a GET request to retrieve a Chuck Norris joke

<script>
var url = "http://api.icndb.com/jokes/random";
var request = new XMLHttpRequest();

request.open('GET', url, true);

request.onload = function() {
 if (request.status >= 200 && request.status < 400) {
 // Success!

printJokes(request.responseText);

} else {
 // We reached our target server, but it returned an error
 console.log('There was a problem. Status:' + request.status);

}
};

request.onerror = function() {
 console.log('There was a problem!');
};

request.send();


var printJokes = function(jokes){

JSON.parse(jokes, (key, value) => {
 key == 'joke' ?
 console.log(value) : ''
 });

}
</script>

 

 

 

Posted on

Javascript reminder #1 Spread syntax

Okay, a quick reminder on Spread Syntax

If you have a Javascript function that you call with

average(1, 2, 3);

function average(x, y, z){
// Average of the three numbers
}

You are limited to just three properties/arguments. Not very useful if you have more or less numbers you want to find an average. Using Spread Syntax we don’t have to worry about the number of values passed

function average(...args) {
  var sum = 0;
  for (let value of args) {
    sum += value;
  }
  return sum / args.length;
}

average(2, 3, 4, 5); // 3.5

The Spread Syntax ( the three dots{name}) includes all the unknown values.  It gets better still

var averageMethod = 'mode';
var roundTo = 2; /Decimal places
doThing( averageMethod, [2,4,3,6,10,19], roundTo);

function doThing(method, ...valuesToAverage, roundTo)
{
// Calculate averages
}

And Spread Syntax can be used when creating arrays

var parts = ['shoulders', 'knees']; 
var lyrics = ['head', ...parts, 'and', 'toes']; 
// ["head", "shoulders", "knees", "and", "toes"]

Good eh?