This commit is contained in:
gyq
2024-02-26 16:13:48 +08:00
parent eeac3e3541
commit 1f59f300d0
10000 changed files with 1689285 additions and 1 deletions

View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2017 Luke Childs
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -0,0 +1,206 @@
# cacheable-request
> Wrap native HTTP requests with RFC compliant cache support
[![Build Status](https://travis-ci.org/lukechilds/cacheable-request.svg?branch=master)](https://travis-ci.org/lukechilds/cacheable-request)
[![Coverage Status](https://coveralls.io/repos/github/lukechilds/cacheable-request/badge.svg?branch=master)](https://coveralls.io/github/lukechilds/cacheable-request?branch=master)
[![npm](https://img.shields.io/npm/dm/cacheable-request.svg)](https://www.npmjs.com/package/cacheable-request)
[![npm](https://img.shields.io/npm/v/cacheable-request.svg)](https://www.npmjs.com/package/cacheable-request)
[RFC 7234](http://httpwg.org/specs/rfc7234.html) compliant HTTP caching for native Node.js HTTP/HTTPS requests. Caching works out of the box in memory or is easily pluggable with a wide range of storage adapters.
**Note:** This is a low level wrapper around the core HTTP modules, it's not a high level request library.
## Features
- Only stores cacheable responses as defined by RFC 7234
- Fresh cache entries are served directly from cache
- Stale cache entries are revalidated with `If-None-Match`/`If-Modified-Since` headers
- 304 responses from revalidation requests use cached body
- Updates `Age` header on cached responses
- Can completely bypass cache on a per request basis
- In memory cache by default
- Official support for Redis, MongoDB, SQLite, PostgreSQL and MySQL storage adapters
- Easily plug in your own or third-party storage adapters
- If DB connection fails, cache is automatically bypassed ([disabled by default](#optsautomaticfailover))
- Adds cache support to any existing HTTP code with minimal changes
- Uses [http-cache-semantics](https://github.com/pornel/http-cache-semantics) internally for HTTP RFC 7234 compliance
## Install
```shell
npm install cacheable-request
```
## Usage
```js
const http = require('http');
const CacheableRequest = require('cacheable-request');
// Then instead of
const req = http.request('http://example.com', cb);
req.end();
// You can do
const cacheableRequest = new CacheableRequest(http.request);
const cacheReq = cacheableRequest('http://example.com', cb);
cacheReq.on('request', req => req.end());
// Future requests to 'example.com' will be returned from cache if still valid
// You pass in any other http.request API compatible method to be wrapped with cache support:
const cacheableRequest = new CacheableRequest(https.request);
const cacheableRequest = new CacheableRequest(electron.net);
```
## Storage Adapters
`cacheable-request` uses [Keyv](https://github.com/lukechilds/keyv) to support a wide range of storage adapters.
For example, to use Redis as a cache backend, you just need to install the official Redis Keyv storage adapter:
```
npm install @keyv/redis
```
And then you can pass `CacheableRequest` your connection string:
```js
const cacheableRequest = new CacheableRequest(http.request, 'redis://user:pass@localhost:6379');
```
[View all official Keyv storage adapters.](https://github.com/lukechilds/keyv#official-storage-adapters)
Keyv also supports anything that follows the Map API so it's easy to write your own storage adapter or use a third-party solution.
e.g The following are all valid storage adapters
```js
const storageAdapter = new Map();
// or
const storageAdapter = require('./my-storage-adapter');
// or
const QuickLRU = require('quick-lru');
const storageAdapter = new QuickLRU({ maxSize: 1000 });
const cacheableRequest = new CacheableRequest(http.request, storageAdapter);
```
View the [Keyv docs](https://github.com/lukechilds/keyv) for more information on how to use storage adapters.
## API
### new cacheableRequest(request, [storageAdapter])
Returns the provided request function wrapped with cache support.
#### request
Type: `function`
Request function to wrap with cache support. Should be [`http.request`](https://nodejs.org/api/http.html#http_http_request_options_callback) or a similar API compatible request function.
#### storageAdapter
Type: `Keyv storage adapter`<br>
Default: `new Map()`
A [Keyv](https://github.com/lukechilds/keyv) storage adapter instance, or connection string if using with an official Keyv storage adapter.
### Instance
#### cacheableRequest(opts, [cb])
Returns an event emitter.
##### opts
Type: `object`, `string`
- Any of the default request functions options.
- Any [`http-cache-semantics`](https://github.com/kornelski/http-cache-semantics#constructor-options) options.
- Any of the following:
###### opts.cache
Type: `boolean`<br>
Default: `true`
If the cache should be used. Setting this to false will completely bypass the cache for the current request.
###### opts.strictTtl
Type: `boolean`<br>
Default: `false`
If set to `true` once a cached resource has expired it is deleted and will have to be re-requested.
If set to `false` (default), after a cached resource's TTL expires it is kept in the cache and will be revalidated on the next request with `If-None-Match`/`If-Modified-Since` headers.
###### opts.maxTtl
Type: `number`<br>
Default: `undefined`
Limits TTL. The `number` represents milliseconds.
###### opts.automaticFailover
Type: `boolean`<br>
Default: `false`
When set to `true`, if the DB connection fails we will automatically fallback to a network request. DB errors will still be emitted to notify you of the problem even though the request callback may succeed.
###### opts.forceRefresh
Type: `boolean`<br>
Default: `false`
Forces refreshing the cache. If the response could be retrieved from the cache, it will perform a new request and override the cache instead.
##### cb
Type: `function`
The callback function which will receive the response as an argument.
The response can be either a [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage) or a [responselike object](https://github.com/lukechilds/responselike). The response will also have a `fromCache` property set with a boolean value.
##### .on('request', request)
`request` event to get the request object of the request.
**Note:** This event will only fire if an HTTP request is actually made, not when a response is retrieved from cache. However, you should always handle the `request` event to end the request and handle any potential request errors.
##### .on('response', response)
`response` event to get the response object from the HTTP request or cache.
##### .on('error', error)
`error` event emitted in case of an error with the cache.
Errors emitted here will be an instance of `CacheableRequest.RequestError` or `CacheableRequest.CacheError`. You will only ever receive a `RequestError` if the request function throws (normally caused by invalid user input). Normal request errors should be handled inside the `request` event.
To properly handle all error scenarios you should use the following pattern:
```js
cacheableRequest('example.com', cb)
.on('error', err => {
if (err instanceof CacheableRequest.CacheError) {
handleCacheError(err); // Cache error
} else if (err instanceof CacheableRequest.RequestError) {
handleRequestError(err); // Request function thrown
}
})
.on('request', req => {
req.on('error', handleRequestError); // Request error emitted
req.end();
});
```
**Note:** Database connection errors are emitted here, however `cacheable-request` will attempt to re-request the resource and bypass the cache on a connection error. Therefore a database connection error doesn't necessarily mean the request won't be fulfilled.
## License
MIT © Luke Childs

View File

@@ -0,0 +1,59 @@
{
"name": "cacheable-request",
"version": "7.0.4",
"description": "Wrap native HTTP requests with RFC compliant cache support",
"license": "MIT",
"repository": "lukechilds/cacheable-request",
"author": "Luke Childs <lukechilds123@gmail.com> (http://lukechilds.co.uk)",
"main": "src/index.js",
"engines": {
"node": ">=8"
},
"scripts": {
"test": "xo && nyc ava",
"coverage": "nyc report --reporter=text-lcov | coveralls"
},
"files": [
"src"
],
"keywords": [
"HTTP",
"HTTPS",
"cache",
"caching",
"layer",
"cacheable",
"RFC 7234",
"RFC",
"7234",
"compliant"
],
"dependencies": {
"clone-response": "^1.0.2",
"get-stream": "^5.1.0",
"http-cache-semantics": "^4.0.0",
"keyv": "^4.0.0",
"lowercase-keys": "^2.0.0",
"normalize-url": "^6.0.1",
"responselike": "^2.0.0"
},
"devDependencies": {
"@keyv/sqlite": "^2.0.0",
"ava": "^1.1.0",
"coveralls": "^3.0.0",
"create-test-server": "3.0.0",
"delay": "^4.0.0",
"eslint-config-xo-lukechilds": "^1.0.0",
"nyc": "^14.1.1",
"pify": "^4.0.0",
"sqlite3": "^4.0.2",
"this": "^1.0.2",
"xo": "^0.23.0"
},
"xo": {
"extends": "xo-lukechilds"
},
"__npminstall_done": true,
"_from": "cacheable-request@7.0.4",
"_resolved": "https://registry.npmmirror.com/cacheable-request/-/cacheable-request-7.0.4.tgz"
}

View File

@@ -0,0 +1,251 @@
'use strict';
const EventEmitter = require('events');
const urlLib = require('url');
const normalizeUrl = require('normalize-url');
const getStream = require('get-stream');
const CachePolicy = require('http-cache-semantics');
const Response = require('responselike');
const lowercaseKeys = require('lowercase-keys');
const cloneResponse = require('clone-response');
const Keyv = require('keyv');
class CacheableRequest {
constructor(request, cacheAdapter) {
if (typeof request !== 'function') {
throw new TypeError('Parameter `request` must be a function');
}
this.cache = new Keyv({
uri: typeof cacheAdapter === 'string' && cacheAdapter,
store: typeof cacheAdapter !== 'string' && cacheAdapter,
namespace: 'cacheable-request'
});
return this.createCacheableRequest(request);
}
createCacheableRequest(request) {
return (opts, cb) => {
let url;
if (typeof opts === 'string') {
url = normalizeUrlObject(urlLib.parse(opts));
opts = {};
} else if (opts instanceof urlLib.URL) {
url = normalizeUrlObject(urlLib.parse(opts.toString()));
opts = {};
} else {
const [pathname, ...searchParts] = (opts.path || '').split('?');
const search = searchParts.length > 0 ?
`?${searchParts.join('?')}` :
'';
url = normalizeUrlObject({ ...opts, pathname, search });
}
opts = {
headers: {},
method: 'GET',
cache: true,
strictTtl: false,
automaticFailover: false,
...opts,
...urlObjectToRequestOptions(url)
};
opts.headers = lowercaseKeys(opts.headers);
const ee = new EventEmitter();
const normalizedUrlString = normalizeUrl(
urlLib.format(url),
{
stripWWW: false,
removeTrailingSlash: false,
stripAuthentication: false
}
);
const key = `${opts.method}:${normalizedUrlString}`;
let revalidate = false;
let madeRequest = false;
const makeRequest = opts => {
madeRequest = true;
let requestErrored = false;
let requestErrorCallback;
const requestErrorPromise = new Promise(resolve => {
requestErrorCallback = () => {
if (!requestErrored) {
requestErrored = true;
resolve();
}
};
});
const handler = response => {
if (revalidate && !opts.forceRefresh) {
response.status = response.statusCode;
const revalidatedPolicy = CachePolicy.fromObject(revalidate.cachePolicy).revalidatedPolicy(opts, response);
if (!revalidatedPolicy.modified) {
const headers = revalidatedPolicy.policy.responseHeaders();
response = new Response(revalidate.statusCode, headers, revalidate.body, revalidate.url);
response.cachePolicy = revalidatedPolicy.policy;
response.fromCache = true;
}
}
if (!response.fromCache) {
response.cachePolicy = new CachePolicy(opts, response, opts);
response.fromCache = false;
}
let clonedResponse;
if (opts.cache && response.cachePolicy.storable()) {
clonedResponse = cloneResponse(response);
(async () => {
try {
const bodyPromise = getStream.buffer(response);
await Promise.race([
requestErrorPromise,
new Promise(resolve => response.once('end', resolve))
]);
if (requestErrored) {
return;
}
const body = await bodyPromise;
const value = {
cachePolicy: response.cachePolicy.toObject(),
url: response.url,
statusCode: response.fromCache ? revalidate.statusCode : response.statusCode,
body
};
let ttl = opts.strictTtl ? response.cachePolicy.timeToLive() : undefined;
if (opts.maxTtl) {
ttl = ttl ? Math.min(ttl, opts.maxTtl) : opts.maxTtl;
}
await this.cache.set(key, value, ttl);
} catch (error) {
ee.emit('error', new CacheableRequest.CacheError(error));
}
})();
} else if (opts.cache && revalidate) {
(async () => {
try {
await this.cache.delete(key);
} catch (error) {
ee.emit('error', new CacheableRequest.CacheError(error));
}
})();
}
ee.emit('response', clonedResponse || response);
if (typeof cb === 'function') {
cb(clonedResponse || response);
}
};
try {
const req = request(opts, handler);
req.once('error', requestErrorCallback);
req.once('abort', requestErrorCallback);
ee.emit('request', req);
} catch (error) {
ee.emit('error', new CacheableRequest.RequestError(error));
}
};
(async () => {
const get = async opts => {
await Promise.resolve();
const cacheEntry = opts.cache ? await this.cache.get(key) : undefined;
if (typeof cacheEntry === 'undefined') {
return makeRequest(opts);
}
const policy = CachePolicy.fromObject(cacheEntry.cachePolicy);
if (policy.satisfiesWithoutRevalidation(opts) && !opts.forceRefresh) {
const headers = policy.responseHeaders();
const response = new Response(cacheEntry.statusCode, headers, cacheEntry.body, cacheEntry.url);
response.cachePolicy = policy;
response.fromCache = true;
ee.emit('response', response);
if (typeof cb === 'function') {
cb(response);
}
} else {
revalidate = cacheEntry;
opts.headers = policy.revalidationHeaders(opts);
makeRequest(opts);
}
};
const errorHandler = error => ee.emit('error', new CacheableRequest.CacheError(error));
this.cache.once('error', errorHandler);
ee.on('response', () => this.cache.removeListener('error', errorHandler));
try {
await get(opts);
} catch (error) {
if (opts.automaticFailover && !madeRequest) {
makeRequest(opts);
}
ee.emit('error', new CacheableRequest.CacheError(error));
}
})();
return ee;
};
}
}
function urlObjectToRequestOptions(url) {
const options = { ...url };
options.path = `${url.pathname || '/'}${url.search || ''}`;
delete options.pathname;
delete options.search;
return options;
}
function normalizeUrlObject(url) {
// If url was parsed by url.parse or new URL:
// - hostname will be set
// - host will be hostname[:port]
// - port will be set if it was explicit in the parsed string
// Otherwise, url was from request options:
// - hostname or host may be set
// - host shall not have port encoded
return {
protocol: url.protocol,
auth: url.auth,
hostname: url.hostname || url.host || 'localhost',
port: url.port,
pathname: url.pathname,
search: url.search
};
}
CacheableRequest.RequestError = class extends Error {
constructor(error) {
super(error.message);
this.name = 'RequestError';
Object.assign(this, error);
}
};
CacheableRequest.CacheError = class extends Error {
constructor(error) {
super(error.message);
this.name = 'CacheError';
Object.assign(this, error);
}
};
module.exports = CacheableRequest;

View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2017 Luke Childs
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

View File

@@ -0,0 +1,62 @@
# clone-response
> Clone a Node.js HTTP response stream
[![Build Status](https://travis-ci.org/lukechilds/clone-response.svg?branch=master)](https://travis-ci.org/lukechilds/clone-response)
[![Coverage Status](https://coveralls.io/repos/github/lukechilds/clone-response/badge.svg?branch=master)](https://coveralls.io/github/lukechilds/clone-response?branch=master)
[![npm](https://img.shields.io/npm/dm/clone-response.svg)](https://www.npmjs.com/package/clone-response)
[![npm](https://img.shields.io/npm/v/clone-response.svg)](https://www.npmjs.com/package/clone-response)
Returns a new stream and copies over all properties and methods from the original response giving you a complete duplicate.
This is useful in situations where you need to consume the response stream but also want to pass an unconsumed stream somewhere else to be consumed later.
## Install
```shell
npm install --save clone-response
```
## Usage
```js
const http = require('http');
const cloneResponse = require('clone-response');
http.get('http://example.com', response => {
const clonedResponse = cloneResponse(response);
response.pipe(process.stdout);
setImmediate(() => {
// The response stream has already been consumed by the time this executes,
// however the cloned response stream is still available.
doSomethingWithResponse(clonedResponse);
});
});
```
Please bear in mind that the process of cloning a stream consumes it. However, you can consume a stream multiple times in the same tick, therefore allowing you to create multiple clones. e.g:
```js
const clone1 = cloneResponse(response);
const clone2 = cloneResponse(response);
// response can still be consumed in this tick but cannot be consumed if passed
// into any async callbacks. clone1 and clone2 can be passed around and be
// consumed in the future.
```
## API
### cloneResponse(response)
Returns a clone of the passed in response.
#### response
Type: `stream`
A [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage) to clone.
## License
MIT © Luke Childs

View File

@@ -0,0 +1,44 @@
{
"name": "clone-response",
"version": "1.0.3",
"description": "Clone a Node.js HTTP response stream",
"main": "src/index.js",
"scripts": {
"test": "xo && nyc ava",
"coverage": "nyc report --reporter=text-lcov | coveralls"
},
"funding": "https://github.com/sponsors/sindresorhus",
"xo": {
"extends": "xo-lukechilds"
},
"repository": {
"type": "git",
"url": "git+https://github.com/sindresorhus/clone-response.git"
},
"keywords": [
"clone",
"duplicate",
"copy",
"response",
"HTTP",
"stream"
],
"author": "Luke Childs <lukechilds123@gmail.com> (http://lukechilds.co.uk)",
"license": "MIT",
"dependencies": {
"mimic-response": "^1.0.0"
},
"devDependencies": {
"ava": "^0.22.0",
"coveralls": "^2.13.1",
"create-test-server": "^2.0.1",
"eslint-config-xo-lukechilds": "^1.0.0",
"get-stream": "^3.0.0",
"nyc": "^11.0.2",
"pify": "^3.0.0",
"xo": "^0.19.0"
},
"__npminstall_done": true,
"_from": "clone-response@1.0.3",
"_resolved": "https://registry.npmmirror.com/clone-response/-/clone-response-1.0.3.tgz"
}

View File

@@ -0,0 +1,17 @@
'use strict';
const PassThrough = require('stream').PassThrough;
const mimicResponse = require('mimic-response');
const cloneResponse = response => {
if (!(response && response.pipe)) {
throw new TypeError('Parameter `response` must be a response stream.');
}
const clone = new PassThrough();
mimicResponse(response, clone);
return response.pipe(clone);
};
module.exports = cloneResponse;

View File

@@ -0,0 +1,52 @@
'use strict';
const {PassThrough: PassThroughStream} = require('stream');
module.exports = options => {
options = {...options};
const {array} = options;
let {encoding} = options;
const isBuffer = encoding === 'buffer';
let objectMode = false;
if (array) {
objectMode = !(encoding || isBuffer);
} else {
encoding = encoding || 'utf8';
}
if (isBuffer) {
encoding = null;
}
const stream = new PassThroughStream({objectMode});
if (encoding) {
stream.setEncoding(encoding);
}
let length = 0;
const chunks = [];
stream.on('data', chunk => {
chunks.push(chunk);
if (objectMode) {
length = chunks.length;
} else {
length += chunk.length;
}
});
stream.getBufferedValue = () => {
if (array) {
return chunks;
}
return isBuffer ? Buffer.concat(chunks, length) : chunks.join('');
};
stream.getBufferedLength = () => length;
return stream;
};

View File

@@ -0,0 +1,108 @@
/// <reference types="node"/>
import {Stream} from 'stream';
declare class MaxBufferErrorClass extends Error {
readonly name: 'MaxBufferError';
constructor();
}
declare namespace getStream {
interface Options {
/**
Maximum length of the returned string. If it exceeds this value before the stream ends, the promise will be rejected with a `MaxBufferError` error.
@default Infinity
*/
readonly maxBuffer?: number;
}
interface OptionsWithEncoding<EncodingType = BufferEncoding> extends Options {
/**
[Encoding](https://nodejs.org/api/buffer.html#buffer_buffer) of the incoming stream.
@default 'utf8'
*/
readonly encoding?: EncodingType;
}
type MaxBufferError = MaxBufferErrorClass;
}
declare const getStream: {
/**
Get the `stream` as a string.
@returns A promise that resolves when the end event fires on the stream, indicating that there is no more data to be read. The stream is switched to flowing mode.
@example
```
import * as fs from 'fs';
import getStream = require('get-stream');
(async () => {
const stream = fs.createReadStream('unicorn.txt');
console.log(await getStream(stream));
// ,,))))))));,
// __)))))))))))))),
// \|/ -\(((((''''((((((((.
// -*-==//////(('' . `)))))),
// /|\ ))| o ;-. '((((( ,(,
// ( `| / ) ;))))' ,_))^;(~
// | | | ,))((((_ _____------~~~-. %,;(;(>';'~
// o_); ; )))(((` ~---~ `:: \ %%~~)(v;(`('~
// ; ''''```` `: `:::|\,__,%% );`'; ~
// | _ ) / `:|`----' `-'
// ______/\/~ | / /
// /~;;.____/;;' / ___--,-( `;;;/
// / // _;______;'------~~~~~ /;;/\ /
// // | | / ; \;;,\
// (<_ | ; /',/-----' _>
// \_| ||_ //~;~~~~~~~~~
// `\_| (,~~
// \~\
// ~~
})();
```
*/
(stream: Stream, options?: getStream.OptionsWithEncoding): Promise<string>;
/**
Get the `stream` as a buffer.
It honors the `maxBuffer` option as above, but it refers to byte length rather than string length.
*/
buffer(
stream: Stream,
options?: getStream.OptionsWithEncoding
): Promise<Buffer>;
/**
Get the `stream` as an array of values.
It honors both the `maxBuffer` and `encoding` options. The behavior changes slightly based on the encoding chosen:
- When `encoding` is unset, it assumes an [object mode stream](https://nodesource.com/blog/understanding-object-streams/) and collects values emitted from `stream` unmodified. In this case `maxBuffer` refers to the number of items in the array (not the sum of their sizes).
- When `encoding` is set to `buffer`, it collects an array of buffers. `maxBuffer` refers to the summed byte lengths of every buffer in the array.
- When `encoding` is set to anything else, it collects an array of strings. `maxBuffer` refers to the summed character lengths of every string in the array.
*/
array<StreamObjectModeType>(
stream: Stream,
options?: getStream.Options
): Promise<StreamObjectModeType[]>;
array(
stream: Stream,
options: getStream.OptionsWithEncoding<'buffer'>
): Promise<Buffer[]>;
array(
stream: Stream,
options: getStream.OptionsWithEncoding<BufferEncoding>
): Promise<string[]>;
MaxBufferError: typeof MaxBufferErrorClass;
// TODO: Remove this for the next major release
default: typeof getStream;
};
export = getStream;

View File

@@ -0,0 +1,60 @@
'use strict';
const {constants: BufferConstants} = require('buffer');
const pump = require('pump');
const bufferStream = require('./buffer-stream');
class MaxBufferError extends Error {
constructor() {
super('maxBuffer exceeded');
this.name = 'MaxBufferError';
}
}
async function getStream(inputStream, options) {
if (!inputStream) {
return Promise.reject(new Error('Expected a stream'));
}
options = {
maxBuffer: Infinity,
...options
};
const {maxBuffer} = options;
let stream;
await new Promise((resolve, reject) => {
const rejectPromise = error => {
// Don't retrieve an oversized buffer.
if (error && stream.getBufferedLength() <= BufferConstants.MAX_LENGTH) {
error.bufferedData = stream.getBufferedValue();
}
reject(error);
};
stream = pump(inputStream, bufferStream(options), error => {
if (error) {
rejectPromise(error);
return;
}
resolve();
});
stream.on('data', () => {
if (stream.getBufferedLength() > maxBuffer) {
rejectPromise(new MaxBufferError());
}
});
});
return stream.getBufferedValue();
}
module.exports = getStream;
// TODO: Remove this for the next major release
module.exports.default = getStream;
module.exports.buffer = (stream, options) => getStream(stream, {...options, encoding: 'buffer'});
module.exports.array = (stream, options) => getStream(stream, {...options, array: true});
module.exports.MaxBufferError = MaxBufferError;

View File

@@ -0,0 +1,9 @@
MIT License
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (https://sindresorhus.com)
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@@ -0,0 +1,53 @@
{
"name": "get-stream",
"version": "5.2.0",
"description": "Get a stream as a string, buffer, or array",
"license": "MIT",
"repository": "sindresorhus/get-stream",
"funding": "https://github.com/sponsors/sindresorhus",
"author": {
"name": "Sindre Sorhus",
"email": "sindresorhus@gmail.com",
"url": "https://sindresorhus.com"
},
"engines": {
"node": ">=8"
},
"scripts": {
"test": "xo && ava && tsd"
},
"files": [
"index.js",
"index.d.ts",
"buffer-stream.js"
],
"keywords": [
"get",
"stream",
"promise",
"concat",
"string",
"text",
"buffer",
"read",
"data",
"consume",
"readable",
"readablestream",
"array",
"object"
],
"dependencies": {
"pump": "^3.0.0"
},
"devDependencies": {
"@types/node": "^12.0.7",
"ava": "^2.0.0",
"into-stream": "^5.0.0",
"tsd": "^0.7.2",
"xo": "^0.24.0"
},
"__npminstall_done": true,
"_from": "get-stream@5.2.0",
"_resolved": "https://registry.npmmirror.com/get-stream/-/get-stream-5.2.0.tgz"
}

View File

@@ -0,0 +1,124 @@
# get-stream [![Build Status](https://travis-ci.com/sindresorhus/get-stream.svg?branch=master)](https://travis-ci.com/github/sindresorhus/get-stream)
> Get a stream as a string, buffer, or array
## Install
```
$ npm install get-stream
```
## Usage
```js
const fs = require('fs');
const getStream = require('get-stream');
(async () => {
const stream = fs.createReadStream('unicorn.txt');
console.log(await getStream(stream));
/*
,,))))))));,
__)))))))))))))),
\|/ -\(((((''''((((((((.
-*-==//////(('' . `)))))),
/|\ ))| o ;-. '((((( ,(,
( `| / ) ;))))' ,_))^;(~
| | | ,))((((_ _____------~~~-. %,;(;(>';'~
o_); ; )))(((` ~---~ `:: \ %%~~)(v;(`('~
; ''''```` `: `:::|\,__,%% );`'; ~
| _ ) / `:|`----' `-'
______/\/~ | / /
/~;;.____/;;' / ___--,-( `;;;/
/ // _;______;'------~~~~~ /;;/\ /
// | | / ; \;;,\
(<_ | ; /',/-----' _>
\_| ||_ //~;~~~~~~~~~
`\_| (,~~
\~\
~~
*/
})();
```
## API
The methods returns a promise that resolves when the `end` event fires on the stream, indicating that there is no more data to be read. The stream is switched to flowing mode.
### getStream(stream, options?)
Get the `stream` as a string.
#### options
Type: `object`
##### encoding
Type: `string`\
Default: `'utf8'`
[Encoding](https://nodejs.org/api/buffer.html#buffer_buffer) of the incoming stream.
##### maxBuffer
Type: `number`\
Default: `Infinity`
Maximum length of the returned string. If it exceeds this value before the stream ends, the promise will be rejected with a `getStream.MaxBufferError` error.
### getStream.buffer(stream, options?)
Get the `stream` as a buffer.
It honors the `maxBuffer` option as above, but it refers to byte length rather than string length.
### getStream.array(stream, options?)
Get the `stream` as an array of values.
It honors both the `maxBuffer` and `encoding` options. The behavior changes slightly based on the encoding chosen:
- When `encoding` is unset, it assumes an [object mode stream](https://nodesource.com/blog/understanding-object-streams/) and collects values emitted from `stream` unmodified. In this case `maxBuffer` refers to the number of items in the array (not the sum of their sizes).
- When `encoding` is set to `buffer`, it collects an array of buffers. `maxBuffer` refers to the summed byte lengths of every buffer in the array.
- When `encoding` is set to anything else, it collects an array of strings. `maxBuffer` refers to the summed character lengths of every string in the array.
## Errors
If the input stream emits an `error` event, the promise will be rejected with the error. The buffered data will be attached to the `bufferedData` property of the error.
```js
(async () => {
try {
await getStream(streamThatErrorsAtTheEnd('unicorn'));
} catch (error) {
console.log(error.bufferedData);
//=> 'unicorn'
}
})()
```
## FAQ
### How is this different from [`concat-stream`](https://github.com/maxogden/concat-stream)?
This module accepts a stream instead of being one and returns a promise instead of using a callback. The API is simpler and it only supports returning a string, buffer, or array. It doesn't have a fragile type inference. You explicitly choose what you want. And it doesn't depend on the huge `readable-stream` package.
## Related
- [get-stdin](https://github.com/sindresorhus/get-stdin) - Get stdin as a string or buffer
---
<div align="center">
<b>
<a href="https://tidelift.com/subscription/pkg/npm-get-stream?utm_source=npm-get-stream&utm_medium=referral&utm_campaign=readme">Get professional support for this package with a Tidelift subscription</a>
</b>
<br>
<sub>
Tidelift helps make open source sustainable for maintainers while giving companies<br>assurances about security, maintenance, and licensing for their dependencies.
</sub>
</div>

View File

@@ -0,0 +1,9 @@
Copyright 2016-2018 Kornel Lesiński
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@@ -0,0 +1,203 @@
# Can I cache this? [![Build Status](https://travis-ci.org/kornelski/http-cache-semantics.svg?branch=master)](https://travis-ci.org/kornelski/http-cache-semantics)
`CachePolicy` tells when responses can be reused from a cache, taking into account [HTTP RFC 7234](http://httpwg.org/specs/rfc7234.html) rules for user agents and shared caches.
It also implements [RFC 5861](https://tools.ietf.org/html/rfc5861), implementing `stale-if-error` and `stale-while-revalidate`.
It's aware of many tricky details such as the `Vary` header, proxy revalidation, and authenticated responses.
## Usage
Cacheability of an HTTP response depends on how it was requested, so both `request` and `response` are required to create the policy.
```js
const policy = new CachePolicy(request, response, options);
if (!policy.storable()) {
// throw the response away, it's not usable at all
return;
}
// Cache the data AND the policy object in your cache
// (this is pseudocode, roll your own cache (lru-cache package works))
letsPretendThisIsSomeCache.set(
request.url,
{ policy, response },
policy.timeToLive()
);
```
```js
// And later, when you receive a new request:
const { policy, response } = letsPretendThisIsSomeCache.get(newRequest.url);
// It's not enough that it exists in the cache, it has to match the new request, too:
if (policy && policy.satisfiesWithoutRevalidation(newRequest)) {
// OK, the previous response can be used to respond to the `newRequest`.
// Response headers have to be updated, e.g. to add Age and remove uncacheable headers.
response.headers = policy.responseHeaders();
return response;
}
```
It may be surprising, but it's not enough for an HTTP response to be [fresh](#yo-fresh) to satisfy a request. It may need to match request headers specified in `Vary`. Even a matching fresh response may still not be usable if the new request restricted cacheability, etc.
The key method is `satisfiesWithoutRevalidation(newRequest)`, which checks whether the `newRequest` is compatible with the original request and whether all caching conditions are met.
### Constructor options
Request and response must have a `headers` property with all header names in lower case. `url`, `status` and `method` are optional (defaults are any URL, status `200`, and `GET` method).
```js
const request = {
url: '/',
method: 'GET',
headers: {
accept: '*/*',
},
};
const response = {
status: 200,
headers: {
'cache-control': 'public, max-age=7234',
},
};
const options = {
shared: true,
cacheHeuristic: 0.1,
immutableMinTimeToLive: 24 * 3600 * 1000, // 24h
ignoreCargoCult: false,
};
```
If `options.shared` is `true` (default), then the response is evaluated from a perspective of a shared cache (i.e. `private` is not cacheable and `s-maxage` is respected). If `options.shared` is `false`, then the response is evaluated from a perspective of a single-user cache (i.e. `private` is cacheable and `s-maxage` is ignored). `shared: true` is recommended for HTTP clients.
`options.cacheHeuristic` is a fraction of response's age that is used as a fallback cache duration. The default is 0.1 (10%), e.g. if a file hasn't been modified for 100 days, it'll be cached for 100\*0.1 = 10 days.
`options.immutableMinTimeToLive` is a number of milliseconds to assume as the default time to cache responses with `Cache-Control: immutable`. Note that [per RFC](http://httpwg.org/http-extensions/immutable.html) these can become stale, so `max-age` still overrides the default.
If `options.ignoreCargoCult` is true, common anti-cache directives will be completely ignored if the non-standard `pre-check` and `post-check` directives are present. These two useless directives are most commonly found in bad StackOverflow answers and PHP's "session limiter" defaults.
### `storable()`
Returns `true` if the response can be stored in a cache. If it's `false` then you MUST NOT store either the request or the response.
### `satisfiesWithoutRevalidation(newRequest)`
This is the most important method. Use this method to check whether the cached response is still fresh in the context of the new request.
If it returns `true`, then the given `request` matches the original response this cache policy has been created with, and the response can be reused without contacting the server. Note that the old response can't be returned without being updated, see `responseHeaders()`.
If it returns `false`, then the response may not be matching at all (e.g. it's for a different URL or method), or may require to be refreshed first (see `revalidationHeaders()`).
### `responseHeaders()`
Returns updated, filtered set of response headers to return to clients receiving the cached response. This function is necessary, because proxies MUST always remove hop-by-hop headers (such as `TE` and `Connection`) and update response's `Age` to avoid doubling cache time.
```js
cachedResponse.headers = cachePolicy.responseHeaders(cachedResponse);
```
### `timeToLive()`
Returns approximate time in _milliseconds_ until the response becomes stale (i.e. not fresh).
After that time (when `timeToLive() <= 0`) the response might not be usable without revalidation. However, there are exceptions, e.g. a client can explicitly allow stale responses, so always check with `satisfiesWithoutRevalidation()`.
`stale-if-error` and `stale-while-revalidate` extend the time to live of the cache, that can still be used if stale.
### `toObject()`/`fromObject(json)`
Chances are you'll want to store the `CachePolicy` object along with the cached response. `obj = policy.toObject()` gives a plain JSON-serializable object. `policy = CachePolicy.fromObject(obj)` creates an instance from it.
### Refreshing stale cache (revalidation)
When a cached response has expired, it can be made fresh again by making a request to the origin server. The server may respond with status 304 (Not Modified) without sending the response body again, saving bandwidth.
The following methods help perform the update efficiently and correctly.
#### `revalidationHeaders(newRequest)`
Returns updated, filtered set of request headers to send to the origin server to check if the cached response can be reused. These headers allow the origin server to return status 304 indicating the response is still fresh. All headers unrelated to caching are passed through as-is.
Use this method when updating cache from the origin server.
```js
updateRequest.headers = cachePolicy.revalidationHeaders(updateRequest);
```
#### `revalidatedPolicy(revalidationRequest, revalidationResponse)`
Use this method to update the cache after receiving a new response from the origin server. It returns an object with two keys:
- `policy` — A new `CachePolicy` with HTTP headers updated from `revalidationResponse`. You can always replace the old cached `CachePolicy` with the new one.
- `modified` — Boolean indicating whether the response body has changed.
- If `false`, then a valid 304 Not Modified response has been received, and you can reuse the old cached response body. This is also affected by `stale-if-error`.
- If `true`, you should use new response's body (if present), or make another request to the origin server without any conditional headers (i.e. don't use `revalidationHeaders()` this time) to get the new resource.
```js
// When serving requests from cache:
const { oldPolicy, oldResponse } = letsPretendThisIsSomeCache.get(
newRequest.url
);
if (!oldPolicy.satisfiesWithoutRevalidation(newRequest)) {
// Change the request to ask the origin server if the cached response can be used
newRequest.headers = oldPolicy.revalidationHeaders(newRequest);
// Send request to the origin server. The server may respond with status 304
const newResponse = await makeRequest(newRequest);
// Create updated policy and combined response from the old and new data
const { policy, modified } = oldPolicy.revalidatedPolicy(
newRequest,
newResponse
);
const response = modified ? newResponse : oldResponse;
// Update the cache with the newer/fresher response
letsPretendThisIsSomeCache.set(
newRequest.url,
{ policy, response },
policy.timeToLive()
);
// And proceed returning cached response as usual
response.headers = policy.responseHeaders();
return response;
}
```
# Yo, FRESH
![satisfiesWithoutRevalidation](fresh.jpg)
## Used by
- [ImageOptim API](https://imageoptim.com/api), [make-fetch-happen](https://github.com/zkat/make-fetch-happen), [cacheable-request](https://www.npmjs.com/package/cacheable-request) ([got](https://www.npmjs.com/package/got)), [npm/registry-fetch](https://github.com/npm/registry-fetch), [etc.](https://github.com/kornelski/http-cache-semantics/network/dependents)
## Implemented
- `Cache-Control` response header with all the quirks.
- `Expires` with check for bad clocks.
- `Pragma` response header.
- `Age` response header.
- `Vary` response header.
- Default cacheability of statuses and methods.
- Requests for stale data.
- Filtering of hop-by-hop headers.
- Basic revalidation request
- `stale-if-error`
## Unimplemented
- Merging of range requests, `If-Range` (but correctly supports them as non-cacheable)
- Revalidation of multiple representations
### Trusting server `Date`
Per the RFC, the cache should take into account the time between server-supplied `Date` and the time it received the response. The RFC-mandated behavior creates two problems:
* Servers with incorrectly set timezone may add several hours to cache age (or more, if the clock is completely wrong).
* Even reasonably correct clocks may be off by a couple of seconds, breaking `max-age=1` trick (which is useful for reverse proxies on high-traffic servers).
Previous versions of this library had an option to ignore the server date if it was "too inaccurate". To support the `max-age=1` trick the library also has to ignore dates that pretty accurate. There's no point of having an option to trust dates that are only a bit inaccurate, so this library won't trust any server dates. `max-age` will be interpreted from the time the response has been received, not from when it has been sent. This will affect only [RFC 1149 networks](https://tools.ietf.org/html/rfc1149).

View File

@@ -0,0 +1,674 @@
'use strict';
// rfc7231 6.1
const statusCodeCacheableByDefault = new Set([
200,
203,
204,
206,
300,
301,
308,
404,
405,
410,
414,
501,
]);
// This implementation does not understand partial responses (206)
const understoodStatuses = new Set([
200,
203,
204,
300,
301,
302,
303,
307,
308,
404,
405,
410,
414,
501,
]);
const errorStatusCodes = new Set([
500,
502,
503,
504,
]);
const hopByHopHeaders = {
date: true, // included, because we add Age update Date
connection: true,
'keep-alive': true,
'proxy-authenticate': true,
'proxy-authorization': true,
te: true,
trailer: true,
'transfer-encoding': true,
upgrade: true,
};
const excludedFromRevalidationUpdate = {
// Since the old body is reused, it doesn't make sense to change properties of the body
'content-length': true,
'content-encoding': true,
'transfer-encoding': true,
'content-range': true,
};
function toNumberOrZero(s) {
const n = parseInt(s, 10);
return isFinite(n) ? n : 0;
}
// RFC 5861
function isErrorResponse(response) {
// consider undefined response as faulty
if(!response) {
return true
}
return errorStatusCodes.has(response.status);
}
function parseCacheControl(header) {
const cc = {};
if (!header) return cc;
// TODO: When there is more than one value present for a given directive (e.g., two Expires header fields, multiple Cache-Control: max-age directives),
// the directive's value is considered invalid. Caches are encouraged to consider responses that have invalid freshness information to be stale
const parts = header.trim().split(/,/);
for (const part of parts) {
const [k, v] = part.split(/=/, 2);
cc[k.trim()] = v === undefined ? true : v.trim().replace(/^"|"$/g, '');
}
return cc;
}
function formatCacheControl(cc) {
let parts = [];
for (const k in cc) {
const v = cc[k];
parts.push(v === true ? k : k + '=' + v);
}
if (!parts.length) {
return undefined;
}
return parts.join(', ');
}
module.exports = class CachePolicy {
constructor(
req,
res,
{
shared,
cacheHeuristic,
immutableMinTimeToLive,
ignoreCargoCult,
_fromObject,
} = {}
) {
if (_fromObject) {
this._fromObject(_fromObject);
return;
}
if (!res || !res.headers) {
throw Error('Response headers missing');
}
this._assertRequestHasHeaders(req);
this._responseTime = this.now();
this._isShared = shared !== false;
this._cacheHeuristic =
undefined !== cacheHeuristic ? cacheHeuristic : 0.1; // 10% matches IE
this._immutableMinTtl =
undefined !== immutableMinTimeToLive
? immutableMinTimeToLive
: 24 * 3600 * 1000;
this._status = 'status' in res ? res.status : 200;
this._resHeaders = res.headers;
this._rescc = parseCacheControl(res.headers['cache-control']);
this._method = 'method' in req ? req.method : 'GET';
this._url = req.url;
this._host = req.headers.host;
this._noAuthorization = !req.headers.authorization;
this._reqHeaders = res.headers.vary ? req.headers : null; // Don't keep all request headers if they won't be used
this._reqcc = parseCacheControl(req.headers['cache-control']);
// Assume that if someone uses legacy, non-standard uncecessary options they don't understand caching,
// so there's no point stricly adhering to the blindly copy&pasted directives.
if (
ignoreCargoCult &&
'pre-check' in this._rescc &&
'post-check' in this._rescc
) {
delete this._rescc['pre-check'];
delete this._rescc['post-check'];
delete this._rescc['no-cache'];
delete this._rescc['no-store'];
delete this._rescc['must-revalidate'];
this._resHeaders = Object.assign({}, this._resHeaders, {
'cache-control': formatCacheControl(this._rescc),
});
delete this._resHeaders.expires;
delete this._resHeaders.pragma;
}
// When the Cache-Control header field is not present in a request, caches MUST consider the no-cache request pragma-directive
// as having the same effect as if "Cache-Control: no-cache" were present (see Section 5.2.1).
if (
res.headers['cache-control'] == null &&
/no-cache/.test(res.headers.pragma)
) {
this._rescc['no-cache'] = true;
}
}
now() {
return Date.now();
}
storable() {
// The "no-store" request directive indicates that a cache MUST NOT store any part of either this request or any response to it.
return !!(
!this._reqcc['no-store'] &&
// A cache MUST NOT store a response to any request, unless:
// The request method is understood by the cache and defined as being cacheable, and
('GET' === this._method ||
'HEAD' === this._method ||
('POST' === this._method && this._hasExplicitExpiration())) &&
// the response status code is understood by the cache, and
understoodStatuses.has(this._status) &&
// the "no-store" cache directive does not appear in request or response header fields, and
!this._rescc['no-store'] &&
// the "private" response directive does not appear in the response, if the cache is shared, and
(!this._isShared || !this._rescc.private) &&
// the Authorization header field does not appear in the request, if the cache is shared,
(!this._isShared ||
this._noAuthorization ||
this._allowsStoringAuthenticated()) &&
// the response either:
// contains an Expires header field, or
(this._resHeaders.expires ||
// contains a max-age response directive, or
// contains a s-maxage response directive and the cache is shared, or
// contains a public response directive.
this._rescc['max-age'] ||
(this._isShared && this._rescc['s-maxage']) ||
this._rescc.public ||
// has a status code that is defined as cacheable by default
statusCodeCacheableByDefault.has(this._status))
);
}
_hasExplicitExpiration() {
// 4.2.1 Calculating Freshness Lifetime
return (
(this._isShared && this._rescc['s-maxage']) ||
this._rescc['max-age'] ||
this._resHeaders.expires
);
}
_assertRequestHasHeaders(req) {
if (!req || !req.headers) {
throw Error('Request headers missing');
}
}
satisfiesWithoutRevalidation(req) {
this._assertRequestHasHeaders(req);
// When presented with a request, a cache MUST NOT reuse a stored response, unless:
// the presented request does not contain the no-cache pragma (Section 5.4), nor the no-cache cache directive,
// unless the stored response is successfully validated (Section 4.3), and
const requestCC = parseCacheControl(req.headers['cache-control']);
if (requestCC['no-cache'] || /no-cache/.test(req.headers.pragma)) {
return false;
}
if (requestCC['max-age'] && this.age() > requestCC['max-age']) {
return false;
}
if (
requestCC['min-fresh'] &&
this.timeToLive() < 1000 * requestCC['min-fresh']
) {
return false;
}
// the stored response is either:
// fresh, or allowed to be served stale
if (this.stale()) {
const allowsStale =
requestCC['max-stale'] &&
!this._rescc['must-revalidate'] &&
(true === requestCC['max-stale'] ||
requestCC['max-stale'] > this.age() - this.maxAge());
if (!allowsStale) {
return false;
}
}
return this._requestMatches(req, false);
}
_requestMatches(req, allowHeadMethod) {
// The presented effective request URI and that of the stored response match, and
return (
(!this._url || this._url === req.url) &&
this._host === req.headers.host &&
// the request method associated with the stored response allows it to be used for the presented request, and
(!req.method ||
this._method === req.method ||
(allowHeadMethod && 'HEAD' === req.method)) &&
// selecting header fields nominated by the stored response (if any) match those presented, and
this._varyMatches(req)
);
}
_allowsStoringAuthenticated() {
// following Cache-Control response directives (Section 5.2.2) have such an effect: must-revalidate, public, and s-maxage.
return (
this._rescc['must-revalidate'] ||
this._rescc.public ||
this._rescc['s-maxage']
);
}
_varyMatches(req) {
if (!this._resHeaders.vary) {
return true;
}
// A Vary header field-value of "*" always fails to match
if (this._resHeaders.vary === '*') {
return false;
}
const fields = this._resHeaders.vary
.trim()
.toLowerCase()
.split(/\s*,\s*/);
for (const name of fields) {
if (req.headers[name] !== this._reqHeaders[name]) return false;
}
return true;
}
_copyWithoutHopByHopHeaders(inHeaders) {
const headers = {};
for (const name in inHeaders) {
if (hopByHopHeaders[name]) continue;
headers[name] = inHeaders[name];
}
// 9.1. Connection
if (inHeaders.connection) {
const tokens = inHeaders.connection.trim().split(/\s*,\s*/);
for (const name of tokens) {
delete headers[name];
}
}
if (headers.warning) {
const warnings = headers.warning.split(/,/).filter(warning => {
return !/^\s*1[0-9][0-9]/.test(warning);
});
if (!warnings.length) {
delete headers.warning;
} else {
headers.warning = warnings.join(',').trim();
}
}
return headers;
}
responseHeaders() {
const headers = this._copyWithoutHopByHopHeaders(this._resHeaders);
const age = this.age();
// A cache SHOULD generate 113 warning if it heuristically chose a freshness
// lifetime greater than 24 hours and the response's age is greater than 24 hours.
if (
age > 3600 * 24 &&
!this._hasExplicitExpiration() &&
this.maxAge() > 3600 * 24
) {
headers.warning =
(headers.warning ? `${headers.warning}, ` : '') +
'113 - "rfc7234 5.5.4"';
}
headers.age = `${Math.round(age)}`;
headers.date = new Date(this.now()).toUTCString();
return headers;
}
/**
* Value of the Date response header or current time if Date was invalid
* @return timestamp
*/
date() {
const serverDate = Date.parse(this._resHeaders.date);
if (isFinite(serverDate)) {
return serverDate;
}
return this._responseTime;
}
/**
* Value of the Age header, in seconds, updated for the current time.
* May be fractional.
*
* @return Number
*/
age() {
let age = this._ageValue();
const residentTime = (this.now() - this._responseTime) / 1000;
return age + residentTime;
}
_ageValue() {
return toNumberOrZero(this._resHeaders.age);
}
/**
* Value of applicable max-age (or heuristic equivalent) in seconds. This counts since response's `Date`.
*
* For an up-to-date value, see `timeToLive()`.
*
* @return Number
*/
maxAge() {
if (!this.storable() || this._rescc['no-cache']) {
return 0;
}
// Shared responses with cookies are cacheable according to the RFC, but IMHO it'd be unwise to do so by default
// so this implementation requires explicit opt-in via public header
if (
this._isShared &&
(this._resHeaders['set-cookie'] &&
!this._rescc.public &&
!this._rescc.immutable)
) {
return 0;
}
if (this._resHeaders.vary === '*') {
return 0;
}
if (this._isShared) {
if (this._rescc['proxy-revalidate']) {
return 0;
}
// if a response includes the s-maxage directive, a shared cache recipient MUST ignore the Expires field.
if (this._rescc['s-maxage']) {
return toNumberOrZero(this._rescc['s-maxage']);
}
}
// If a response includes a Cache-Control field with the max-age directive, a recipient MUST ignore the Expires field.
if (this._rescc['max-age']) {
return toNumberOrZero(this._rescc['max-age']);
}
const defaultMinTtl = this._rescc.immutable ? this._immutableMinTtl : 0;
const serverDate = this.date();
if (this._resHeaders.expires) {
const expires = Date.parse(this._resHeaders.expires);
// A cache recipient MUST interpret invalid date formats, especially the value "0", as representing a time in the past (i.e., "already expired").
if (Number.isNaN(expires) || expires < serverDate) {
return 0;
}
return Math.max(defaultMinTtl, (expires - serverDate) / 1000);
}
if (this._resHeaders['last-modified']) {
const lastModified = Date.parse(this._resHeaders['last-modified']);
if (isFinite(lastModified) && serverDate > lastModified) {
return Math.max(
defaultMinTtl,
((serverDate - lastModified) / 1000) * this._cacheHeuristic
);
}
}
return defaultMinTtl;
}
timeToLive() {
const age = this.maxAge() - this.age();
const staleIfErrorAge = age + toNumberOrZero(this._rescc['stale-if-error']);
const staleWhileRevalidateAge = age + toNumberOrZero(this._rescc['stale-while-revalidate']);
return Math.max(0, age, staleIfErrorAge, staleWhileRevalidateAge) * 1000;
}
stale() {
return this.maxAge() <= this.age();
}
_useStaleIfError() {
return this.maxAge() + toNumberOrZero(this._rescc['stale-if-error']) > this.age();
}
useStaleWhileRevalidate() {
return this.maxAge() + toNumberOrZero(this._rescc['stale-while-revalidate']) > this.age();
}
static fromObject(obj) {
return new this(undefined, undefined, { _fromObject: obj });
}
_fromObject(obj) {
if (this._responseTime) throw Error('Reinitialized');
if (!obj || obj.v !== 1) throw Error('Invalid serialization');
this._responseTime = obj.t;
this._isShared = obj.sh;
this._cacheHeuristic = obj.ch;
this._immutableMinTtl =
obj.imm !== undefined ? obj.imm : 24 * 3600 * 1000;
this._status = obj.st;
this._resHeaders = obj.resh;
this._rescc = obj.rescc;
this._method = obj.m;
this._url = obj.u;
this._host = obj.h;
this._noAuthorization = obj.a;
this._reqHeaders = obj.reqh;
this._reqcc = obj.reqcc;
}
toObject() {
return {
v: 1,
t: this._responseTime,
sh: this._isShared,
ch: this._cacheHeuristic,
imm: this._immutableMinTtl,
st: this._status,
resh: this._resHeaders,
rescc: this._rescc,
m: this._method,
u: this._url,
h: this._host,
a: this._noAuthorization,
reqh: this._reqHeaders,
reqcc: this._reqcc,
};
}
/**
* Headers for sending to the origin server to revalidate stale response.
* Allows server to return 304 to allow reuse of the previous response.
*
* Hop by hop headers are always stripped.
* Revalidation headers may be added or removed, depending on request.
*/
revalidationHeaders(incomingReq) {
this._assertRequestHasHeaders(incomingReq);
const headers = this._copyWithoutHopByHopHeaders(incomingReq.headers);
// This implementation does not understand range requests
delete headers['if-range'];
if (!this._requestMatches(incomingReq, true) || !this.storable()) {
// revalidation allowed via HEAD
// not for the same resource, or wasn't allowed to be cached anyway
delete headers['if-none-match'];
delete headers['if-modified-since'];
return headers;
}
/* MUST send that entity-tag in any cache validation request (using If-Match or If-None-Match) if an entity-tag has been provided by the origin server. */
if (this._resHeaders.etag) {
headers['if-none-match'] = headers['if-none-match']
? `${headers['if-none-match']}, ${this._resHeaders.etag}`
: this._resHeaders.etag;
}
// Clients MAY issue simple (non-subrange) GET requests with either weak validators or strong validators. Clients MUST NOT use weak validators in other forms of request.
const forbidsWeakValidators =
headers['accept-ranges'] ||
headers['if-match'] ||
headers['if-unmodified-since'] ||
(this._method && this._method != 'GET');
/* SHOULD send the Last-Modified value in non-subrange cache validation requests (using If-Modified-Since) if only a Last-Modified value has been provided by the origin server.
Note: This implementation does not understand partial responses (206) */
if (forbidsWeakValidators) {
delete headers['if-modified-since'];
if (headers['if-none-match']) {
const etags = headers['if-none-match']
.split(/,/)
.filter(etag => {
return !/^\s*W\//.test(etag);
});
if (!etags.length) {
delete headers['if-none-match'];
} else {
headers['if-none-match'] = etags.join(',').trim();
}
}
} else if (
this._resHeaders['last-modified'] &&
!headers['if-modified-since']
) {
headers['if-modified-since'] = this._resHeaders['last-modified'];
}
return headers;
}
/**
* Creates new CachePolicy with information combined from the previews response,
* and the new revalidation response.
*
* Returns {policy, modified} where modified is a boolean indicating
* whether the response body has been modified, and old cached body can't be used.
*
* @return {Object} {policy: CachePolicy, modified: Boolean}
*/
revalidatedPolicy(request, response) {
this._assertRequestHasHeaders(request);
if(this._useStaleIfError() && isErrorResponse(response)) { // I consider the revalidation request unsuccessful
return {
modified: false,
matches: false,
policy: this,
};
}
if (!response || !response.headers) {
throw Error('Response headers missing');
}
// These aren't going to be supported exactly, since one CachePolicy object
// doesn't know about all the other cached objects.
let matches = false;
if (response.status !== undefined && response.status != 304) {
matches = false;
} else if (
response.headers.etag &&
!/^\s*W\//.test(response.headers.etag)
) {
// "All of the stored responses with the same strong validator are selected.
// If none of the stored responses contain the same strong validator,
// then the cache MUST NOT use the new response to update any stored responses."
matches =
this._resHeaders.etag &&
this._resHeaders.etag.replace(/^\s*W\//, '') ===
response.headers.etag;
} else if (this._resHeaders.etag && response.headers.etag) {
// "If the new response contains a weak validator and that validator corresponds
// to one of the cache's stored responses,
// then the most recent of those matching stored responses is selected for update."
matches =
this._resHeaders.etag.replace(/^\s*W\//, '') ===
response.headers.etag.replace(/^\s*W\//, '');
} else if (this._resHeaders['last-modified']) {
matches =
this._resHeaders['last-modified'] ===
response.headers['last-modified'];
} else {
// If the new response does not include any form of validator (such as in the case where
// a client generates an If-Modified-Since request from a source other than the Last-Modified
// response header field), and there is only one stored response, and that stored response also
// lacks a validator, then that stored response is selected for update.
if (
!this._resHeaders.etag &&
!this._resHeaders['last-modified'] &&
!response.headers.etag &&
!response.headers['last-modified']
) {
matches = true;
}
}
if (!matches) {
return {
policy: new this.constructor(request, response),
// Client receiving 304 without body, even if it's invalid/mismatched has no option
// but to reuse a cached body. We don't have a good way to tell clients to do
// error recovery in such case.
modified: response.status != 304,
matches: false,
};
}
// use other header fields provided in the 304 (Not Modified) response to replace all instances
// of the corresponding header fields in the stored response.
const headers = {};
for (const k in this._resHeaders) {
headers[k] =
k in response.headers && !excludedFromRevalidationUpdate[k]
? response.headers[k]
: this._resHeaders[k];
}
const newResponse = Object.assign({}, response, {
status: this._status,
method: this._method,
headers,
});
return {
policy: new this.constructor(request, newResponse, {
shared: this._isShared,
cacheHeuristic: this._cacheHeuristic,
immutableMinTimeToLive: this._immutableMinTtl,
}),
modified: false,
matches: true,
};
}
};

View File

@@ -0,0 +1,21 @@
{
"name": "http-cache-semantics",
"version": "4.1.1",
"description": "Parses Cache-Control and other headers. Helps building correct HTTP caches and proxies",
"repository": "https://github.com/kornelski/http-cache-semantics.git",
"main": "index.js",
"scripts": {
"test": "mocha"
},
"files": [
"index.js"
],
"author": "Kornel Lesiński <kornel@geekhood.net> (https://kornel.ski/)",
"license": "BSD-2-Clause",
"devDependencies": {
"mocha": "^10.0"
},
"__npminstall_done": true,
"_from": "http-cache-semantics@4.1.1",
"_resolved": "https://registry.npmmirror.com/http-cache-semantics/-/http-cache-semantics-4.1.1.tgz"
}

View File

@@ -0,0 +1,429 @@
<h1 align="center">
<img width="250" src="https://jaredwray.com/images/keyv.svg" alt="keyv">
<br>
<br>
</h1>
> Simple key-value storage with support for multiple backends
[![build](https://github.com/jaredwray/keyv/actions/workflows/tests.yaml/badge.svg)](https://github.com/jaredwray/keyv/actions/workflows/tests.yaml)
[![codecov](https://codecov.io/gh/jaredwray/keyv/branch/main/graph/badge.svg?token=bRzR3RyOXZ)](https://codecov.io/gh/jaredwray/keyv)
[![npm](https://img.shields.io/npm/dm/keyv.svg)](https://www.npmjs.com/package/keyv)
[![npm](https://img.shields.io/npm/v/keyv.svg)](https://www.npmjs.com/package/keyv)
Keyv provides a consistent interface for key-value storage across multiple backends via storage adapters. It supports TTL based expiry, making it suitable as a cache or a persistent key-value store.
## Features
There are a few existing modules similar to Keyv, however Keyv is different because it:
- Isn't bloated
- Has a simple Promise based API
- Suitable as a TTL based cache or persistent key-value store
- [Easily embeddable](#add-cache-support-to-your-module) inside another module
- Works with any storage that implements the [`Map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) API
- Handles all JSON types plus `Buffer`
- Supports namespaces
- Wide range of [**efficient, well tested**](#official-storage-adapters) storage adapters
- Connection errors are passed through (db failures won't kill your app)
- Supports the current active LTS version of Node.js or higher
## Usage
Install Keyv.
```
npm install --save keyv
```
By default everything is stored in memory, you can optionally also install a storage adapter.
```
npm install --save @keyv/redis
npm install --save @keyv/mongo
npm install --save @keyv/sqlite
npm install --save @keyv/postgres
npm install --save @keyv/mysql
npm install --save @keyv/etcd
```
Create a new Keyv instance, passing your connection string if applicable. Keyv will automatically load the correct storage adapter.
```js
const Keyv = require('keyv');
// One of the following
const keyv = new Keyv();
const keyv = new Keyv('redis://user:pass@localhost:6379');
const keyv = new Keyv('mongodb://user:pass@localhost:27017/dbname');
const keyv = new Keyv('sqlite://path/to/database.sqlite');
const keyv = new Keyv('postgresql://user:pass@localhost:5432/dbname');
const keyv = new Keyv('mysql://user:pass@localhost:3306/dbname');
const keyv = new Keyv('etcd://localhost:2379');
// Handle DB connection errors
keyv.on('error', err => console.log('Connection Error', err));
await keyv.set('foo', 'expires in 1 second', 1000); // true
await keyv.set('foo', 'never expires'); // true
await keyv.get('foo'); // 'never expires'
await keyv.delete('foo'); // true
await keyv.clear(); // undefined
```
### Namespaces
You can namespace your Keyv instance to avoid key collisions and allow you to clear only a certain namespace while using the same database.
```js
const users = new Keyv('redis://user:pass@localhost:6379', { namespace: 'users' });
const cache = new Keyv('redis://user:pass@localhost:6379', { namespace: 'cache' });
await users.set('foo', 'users'); // true
await cache.set('foo', 'cache'); // true
await users.get('foo'); // 'users'
await cache.get('foo'); // 'cache'
await users.clear(); // undefined
await users.get('foo'); // undefined
await cache.get('foo'); // 'cache'
```
### Custom Serializers
Keyv uses [`json-buffer`](https://github.com/dominictarr/json-buffer) for data serialization to ensure consistency across different backends.
You can optionally provide your own serialization functions to support extra data types or to serialize to something other than JSON.
```js
const keyv = new Keyv({ serialize: JSON.stringify, deserialize: JSON.parse });
```
**Warning:** Using custom serializers means you lose any guarantee of data consistency. You should do extensive testing with your serialisation functions and chosen storage engine.
## Official Storage Adapters
The official storage adapters are covered by [over 150 integration tests](https://github.com/jaredwray/keyv/actions/workflows/tests.yaml) to guarantee consistent behaviour. They are lightweight, efficient wrappers over the DB clients making use of indexes and native TTLs where available.
Database | Adapter | Native TTL
---|---|---
Redis | [@keyv/redis](https://github.com/jaredwray/keyv/tree/master/packages/redis) | Yes
MongoDB | [@keyv/mongo](https://github.com/jaredwray/keyv/tree/master/packages/mongo) | Yes
SQLite | [@keyv/sqlite](https://github.com/jaredwray/keyv/tree/master/packages/sqlite) | No
PostgreSQL | [@keyv/postgres](https://github.com/jaredwray/keyv/tree/master/packages/postgres) | No
MySQL | [@keyv/mysql](https://github.com/jaredwray/keyv/tree/master/packages/mysql) | No
Etcd | [@keyv/etcd](https://github.com/jaredwray/keyv/tree/master/packages/etcd) | Yes
Memcache | [@keyv/memcache](https://github.com/jaredwray/keyv/tree/master/packages/memcache) | Yes
## Third-party Storage Adapters
You can also use third-party storage adapters or build your own. Keyv will wrap these storage adapters in TTL functionality and handle complex types internally.
```js
const Keyv = require('keyv');
const myAdapter = require('./my-storage-adapter');
const keyv = new Keyv({ store: myAdapter });
```
Any store that follows the [`Map`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map) api will work.
```js
new Keyv({ store: new Map() });
```
For example, [`quick-lru`](https://github.com/sindresorhus/quick-lru) is a completely unrelated module that implements the Map API.
```js
const Keyv = require('keyv');
const QuickLRU = require('quick-lru');
const lru = new QuickLRU({ maxSize: 1000 });
const keyv = new Keyv({ store: lru });
```
The following are third-party storage adapters compatible with Keyv:
- [quick-lru](https://github.com/sindresorhus/quick-lru) - Simple "Least Recently Used" (LRU) cache
- [keyv-file](https://github.com/zaaack/keyv-file) - File system storage adapter for Keyv
- [keyv-dynamodb](https://www.npmjs.com/package/keyv-dynamodb) - DynamoDB storage adapter for Keyv
- [keyv-lru](https://www.npmjs.com/package/keyv-lru) - LRU storage adapter for Keyv
- [keyv-null](https://www.npmjs.com/package/keyv-null) - Null storage adapter for Keyv
- [keyv-firestore ](https://github.com/goto-bus-stop/keyv-firestore) Firebase Cloud Firestore adapter for Keyv
- [keyv-mssql](https://github.com/pmorgan3/keyv-mssql) - Microsoft Sql Server adapter for Keyv
- [keyv-azuretable](https://github.com/howlowck/keyv-azuretable) - Azure Table Storage/API adapter for Keyv
- [keyv-arango](https://github.com/TimMikeladze/keyv-arango) - ArangoDB storage adapter for Keyv
- [keyv-momento](https://github.com/momentohq/node-keyv-adaptor/) - Momento storage adapter for Keyv
## Add Cache Support to your Module
Keyv is designed to be easily embedded into other modules to add cache support. The recommended pattern is to expose a `cache` option in your modules options which is passed through to Keyv. Caching will work in memory by default and users have the option to also install a Keyv storage adapter and pass in a connection string, or any other storage that implements the `Map` API.
You should also set a namespace for your module so you can safely call `.clear()` without clearing unrelated app data.
Inside your module:
```js
class AwesomeModule {
constructor(opts) {
this.cache = new Keyv({
uri: typeof opts.cache === 'string' && opts.cache,
store: typeof opts.cache !== 'string' && opts.cache,
namespace: 'awesome-module'
});
}
}
```
Now it can be consumed like this:
```js
const AwesomeModule = require('awesome-module');
// Caches stuff in memory by default
const awesomeModule = new AwesomeModule();
// After npm install --save keyv-redis
const awesomeModule = new AwesomeModule({ cache: 'redis://localhost' });
// Some third-party module that implements the Map API
const awesomeModule = new AwesomeModule({ cache: some3rdPartyStore });
```
## Compression
Keyv supports `gzip` and `brotli` compression. To enable compression, pass the `compress` option to the constructor.
```js
const KeyvGzip = require('@keyv/compress-gzip');
const Keyv = require('keyv');
const keyvGzip = new KeyvGzip();
const keyv = new Keyv({ compression: KeyvGzip });
```
You can also pass a custom compression function to the `compression` option. Following the pattern of the official compression adapters.
### Want to build your own?
Great! Keyv is designed to be easily extended. You can build your own compression adapter by following the pattern of the official compression adapters based on this interface:
```typescript
interface CompressionAdapter {
async compress(value: any, options?: any);
async decompress(value: any, options?: any);
async serialize(value: any);
async deserialize(value: any);
}
```
In addition to the interface, you can test it with our compression test suite using @keyv/test-suite:
```js
const {keyvCompresstionTests} = require('@keyv/test-suite');
const KeyvGzip = require('@keyv/compress-gzip');
keyvCompresstionTests(test, new KeyvGzip());
```
## API
### new Keyv([uri], [options])
Returns a new Keyv instance.
The Keyv instance is also an `EventEmitter` that will emit an `'error'` event if the storage adapter connection fails.
### uri
Type: `String`<br>
Default: `undefined`
The connection string URI.
Merged into the options object as options.uri.
### options
Type: `Object`
The options object is also passed through to the storage adapter. Check your storage adapter docs for any extra options.
#### options.namespace
Type: `String`<br>
Default: `'keyv'`
Namespace for the current instance.
#### options.ttl
Type: `Number`<br>
Default: `undefined`
Default TTL. Can be overridden by specififying a TTL on `.set()`.
#### options.compression
Type: `@keyv/compress-<compression_package_name>`<br>
Default: `undefined`
Compression package to use. See [Compression](#compression) for more details.
#### options.serialize
Type: `Function`<br>
Default: `JSONB.stringify`
A custom serialization function.
#### options.deserialize
Type: `Function`<br>
Default: `JSONB.parse`
A custom deserialization function.
#### options.store
Type: `Storage adapter instance`<br>
Default: `new Map()`
The storage adapter instance to be used by Keyv.
#### options.adapter
Type: `String`<br>
Default: `undefined`
Specify an adapter to use. e.g `'redis'` or `'mongodb'`.
### Instance
Keys must always be strings. Values can be of any type.
#### .set(key, value, [ttl])
Set a value.
By default keys are persistent. You can set an expiry TTL in milliseconds.
Returns a promise which resolves to `true`.
#### .get(key, [options])
Returns a promise which resolves to the retrieved value.
##### options.raw
Type: `Boolean`<br>
Default: `false`
If set to true the raw DB object Keyv stores internally will be returned instead of just the value.
This contains the TTL timestamp.
#### .delete(key)
Deletes an entry.
Returns a promise which resolves to `true` if the key existed, `false` if not.
#### .clear()
Delete all entries in the current namespace.
Returns a promise which is resolved when the entries have been cleared.
#### .iterator()
Iterate over all entries of the current namespace.
Returns a iterable that can be iterated by for-of loops. For example:
```js
// please note that the "await" keyword should be used here
for await (const [key, value] of this.keyv.iterator()) {
console.log(key, value);
};
```
# How to Contribute
In this section of the documentation we will cover:
1) How to set up this repository locally
2) How to get started with running commands
3) How to contribute changes using Pull Requests
## Dependencies
This package requires the following dependencies to run:
1) [Yarn V1](https://yarnpkg.com/getting-started/install)
3) [Docker](https://docs.docker.com/get-docker/)
## Setting up your workspace
To contribute to this repository, start by setting up this project locally:
1) Fork this repository into your Git account
2) Clone the forked repository to your local directory using `git clone`
3) Install any of the above missing dependencies
## Launching the project
Once the project is installed locally, you are ready to start up its services:
1) Ensure that your Docker service is running.
2) From the root directory of your project, run the `yarn` command in the command prompt to install yarn.
3) Run the `yarn bootstrap` command to install any necessary dependencies.
4) Run `yarn test:services:start` to start up this project's Docker container. The container will launch all services within your workspace.
## Available Commands
Once the project is running, you can execute a variety of commands. The root workspace and each subpackage contain a `package.json` file with a `scripts` field listing all the commands that can be executed from that directory. This project also supports native `yarn`, and `docker` commands.
Here, we'll cover the primary commands that can be executed from the root directory. Unless otherwise noted, these commands can also be executed from a subpackage. If executed from a subpackage, they will only affect that subpackage, rather than the entire workspace.
### `yarn`
The `yarn` command installs yarn in the workspace.
### `yarn bootstrap`
The `yarn bootstrap` command installs all dependencies in the workspace.
### `yarn test:services:start`
The `yarn test:services:start` command starts up the project's Docker container, launching all services in the workspace. This command must be executed from the root directory.
### `yarn test:services:stop`
The `yarn test:services:stop` command brings down the project's Docker container, halting all services. This command must be executed from the root directory.
### `yarn test`
The `yarn test` command runs all tests in the workspace.
### `yarn clean`
The `yarn clean` command removes yarn and all dependencies installed by yarn. After executing this command, you must repeat the steps in *Setting up your workspace* to rebuild your workspace.
## Contributing Changes
Now that you've set up your workspace, you're ready to contribute changes to the `keyv` repository.
1) Make any changes that you would like to contribute in your local workspace.
2) After making these changes, ensure that the project's tests still pass by executing the `yarn test` command in the root directory.
3) Commit your changes and push them to your forked repository.
4) Navigate to the original `keyv` repository and go the *Pull Requests* tab.
5) Click the *New pull request* button, and open a pull request for the branch in your repository that contains your changes.
6) Once your pull request is created, ensure that all checks have passed and that your branch has no conflicts with the base branch. If there are any issues, resolve these changes in your local repository, and then commit and push them to git.
7) Similarly, respond to any reviewer comments or requests for changes by making edits to your local repository and pushing them to Git.
8) Once the pull request has been reviewed, those with write access to the branch will be able to merge your changes into the `keyv` repository.
If you need more information on the steps to create a pull request, you can find a detailed walkthrough in the [Github documentation](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork)
## License
MIT © Jared Wray

View File

@@ -0,0 +1,60 @@
{
"name": "keyv",
"version": "4.5.4",
"description": "Simple key-value storage with support for multiple backends",
"main": "src/index.js",
"scripts": {
"build": "echo 'No build step required.'",
"prepare": "yarn build",
"test": "xo && c8 ava --serial",
"test:ci": "xo && ava --serial",
"clean": "rm -rf node_modules && rm -rf ./coverage && rm -rf ./test/testdb.sqlite"
},
"xo": {
"rules": {
"unicorn/prefer-module": 0,
"unicorn/prefer-node-protocol": 0,
"@typescript-eslint/consistent-type-definitions": 0,
"unicorn/no-typeof-undefined": 0,
"unicorn/prefer-event-target": 0
}
},
"repository": {
"type": "git",
"url": "git+https://github.com/jaredwray/keyv.git"
},
"keywords": [
"key",
"value",
"store",
"cache",
"ttl"
],
"author": "Jared Wray <me@jaredwray.com> (http://jaredwray.com)",
"license": "MIT",
"bugs": {
"url": "https://github.com/jaredwray/keyv/issues"
},
"homepage": "https://github.com/jaredwray/keyv",
"dependencies": {
"json-buffer": "3.0.1"
},
"devDependencies": {
"@keyv/test-suite": "*",
"eslint": "^8.51.0",
"eslint-plugin-promise": "^6.1.1",
"pify": "^5.0.0",
"timekeeper": "^2.3.1",
"tsd": "^0.29.0"
},
"tsd": {
"directory": "test"
},
"types": "./src/index.d.ts",
"files": [
"src"
],
"__npminstall_done": true,
"_from": "keyv@4.5.4",
"_resolved": "https://registry.npmmirror.com/keyv/-/keyv-4.5.4.tgz"
}

View File

@@ -0,0 +1,112 @@
import {EventEmitter} from 'events';
type WithRequiredProperties<T, K extends keyof T> = T & Required<Pick<T, K>>;
declare class Keyv<Value = any, Options extends Record<string, any> = Record<string, unknown>> extends EventEmitter {
/**
* `this.opts` is an object containing at least the properties listed
* below. However, `Keyv.Options` allows arbitrary properties as well.
* These properties can be specified as the second type parameter to `Keyv`.
*/
opts: WithRequiredProperties<
Keyv.Options<Value>,
'deserialize' | 'namespace' | 'serialize' | 'store' | 'uri'
> &
Options;
/**
* @param opts The options object is also passed through to the storage adapter. Check your storage adapter docs for any extra options.
*/
constructor(options?: Keyv.Options<Value> & Options);
/**
* @param uri The connection string URI.
*
* Merged into the options object as options.uri.
* @param opts The options object is also passed through to the storage adapter. Check your storage adapter docs for any extra options.
*/
constructor(uri?: string, options?: Keyv.Options<Value> & Options);
/** Returns the value. */
get(key: string, options?: {raw?: false}): Promise<Value | undefined>;
/** Returns the raw value. */
get(key: string, options: {raw: true}): Promise<Keyv.DeserializedData<Value> | undefined>;
/** Returns an array of values. Uses `store.getMany` if it exists, otherwise uses parallel calls to `store.get`. */
get(key: string[], options?: {raw?: false}): Promise<Array<Value | undefined>>;
/** Returns an array of raw values. Uses `store.getMany` if it exists, otherwise uses parallel calls to `store.get`. */
get(key: string[], options: {raw: true}): Promise<Array<Keyv.DeserializedData<Value> | undefined>>;
/**
* Set a value.
*
* By default keys are persistent. You can set an expiry TTL in milliseconds.
*/
set(key: string, value: Value, ttl?: number): Promise<true>;
/**
* Deletes an entry.
*
* Returns `true` if the key existed, `false` if not.
*/
delete(key: string | string[]): Promise<boolean>;
/** Delete all entries in the current namespace. */
clear(): Promise<void>;
/** Check if key exists in current namespace. */
has(key: string): Promise<boolean>;
/** Iterator */
iterator(namespace?: string): AsyncGenerator<any, void, any>;
/**
* Closes the connection.
*
* Returns `undefined` when the connection closes.
*/
disconnect(): Promise<void>;
}
declare namespace Keyv {
interface Options<Value> {
[key: string]: any;
/** Namespace for the current instance. */
namespace?: string | undefined;
/** A custom serialization function. */
serialize?: ((data: DeserializedData<Value>) => string) | undefined;
/** A custom deserialization function. */
deserialize?: ((data: string) => DeserializedData<Value> | undefined) | undefined;
/** The connection string URI. */
uri?: string | undefined;
/** The storage adapter instance to be used by Keyv. */
store?: Store<string | undefined> | undefined;
/** Default TTL. Can be overridden by specififying a TTL on `.set()`. */
ttl?: number | undefined;
/** Specify an adapter to use. e.g `'redis'` or `'mongodb'`. */
adapter?: 'redis' | 'mongodb' | 'mongo' | 'sqlite' | 'postgresql' | 'postgres' | 'mysql' | undefined;
/** Enable compression option **/
compression?: CompressionAdapter | undefined;
}
interface CompressionAdapter {
compress(value: any, options?: any): Promise<any>;
decompress(value: any, options?: any): Promise<any>;
serialize(value: any): Promise<any>;
deserialize(value: any): Promise<any>;
}
interface DeserializedData<Value> {
value: Value; expires: number | undefined;
}
type StoredData<Value> = DeserializedData<Value> | string | undefined;
interface Store<Value> {
get(key: string): Value | Promise<Value | undefined> | undefined;
set(key: string, value: Value, ttl?: number): any;
delete(key: string): boolean | Promise<boolean>;
clear(): void | Promise<void>;
has?(key: string): boolean | Promise<boolean>;
getMany?(
keys: string[]
): Array<StoredData<Value>> | Promise<Array<StoredData<Value>>> | undefined;
}
}
export = Keyv;

View File

@@ -0,0 +1,259 @@
'use strict';
const EventEmitter = require('events');
const JSONB = require('json-buffer');
const loadStore = options => {
const adapters = {
redis: '@keyv/redis',
rediss: '@keyv/redis',
mongodb: '@keyv/mongo',
mongo: '@keyv/mongo',
sqlite: '@keyv/sqlite',
postgresql: '@keyv/postgres',
postgres: '@keyv/postgres',
mysql: '@keyv/mysql',
etcd: '@keyv/etcd',
offline: '@keyv/offline',
tiered: '@keyv/tiered',
};
if (options.adapter || options.uri) {
const adapter = options.adapter || /^[^:+]*/.exec(options.uri)[0];
return new (require(adapters[adapter]))(options);
}
return new Map();
};
const iterableAdapters = [
'sqlite',
'postgres',
'mysql',
'mongo',
'redis',
'tiered',
];
class Keyv extends EventEmitter {
constructor(uri, {emitErrors = true, ...options} = {}) {
super();
this.opts = {
namespace: 'keyv',
serialize: JSONB.stringify,
deserialize: JSONB.parse,
...((typeof uri === 'string') ? {uri} : uri),
...options,
};
if (!this.opts.store) {
const adapterOptions = {...this.opts};
this.opts.store = loadStore(adapterOptions);
}
if (this.opts.compression) {
const compression = this.opts.compression;
this.opts.serialize = compression.serialize.bind(compression);
this.opts.deserialize = compression.deserialize.bind(compression);
}
if (typeof this.opts.store.on === 'function' && emitErrors) {
this.opts.store.on('error', error => this.emit('error', error));
}
this.opts.store.namespace = this.opts.namespace;
const generateIterator = iterator => async function * () {
for await (const [key, raw] of typeof iterator === 'function'
? iterator(this.opts.store.namespace)
: iterator) {
const data = await this.opts.deserialize(raw);
if (this.opts.store.namespace && !key.includes(this.opts.store.namespace)) {
continue;
}
if (typeof data.expires === 'number' && Date.now() > data.expires) {
this.delete(key);
continue;
}
yield [this._getKeyUnprefix(key), data.value];
}
};
// Attach iterators
if (typeof this.opts.store[Symbol.iterator] === 'function' && this.opts.store instanceof Map) {
this.iterator = generateIterator(this.opts.store);
} else if (typeof this.opts.store.iterator === 'function' && this.opts.store.opts
&& this._checkIterableAdaptar()) {
this.iterator = generateIterator(this.opts.store.iterator.bind(this.opts.store));
}
}
_checkIterableAdaptar() {
return iterableAdapters.includes(this.opts.store.opts.dialect)
|| iterableAdapters.findIndex(element => this.opts.store.opts.url.includes(element)) >= 0;
}
_getKeyPrefix(key) {
return `${this.opts.namespace}:${key}`;
}
_getKeyPrefixArray(keys) {
return keys.map(key => `${this.opts.namespace}:${key}`);
}
_getKeyUnprefix(key) {
return key
.split(':')
.splice(1)
.join(':');
}
get(key, options) {
const {store} = this.opts;
const isArray = Array.isArray(key);
const keyPrefixed = isArray ? this._getKeyPrefixArray(key) : this._getKeyPrefix(key);
if (isArray && store.getMany === undefined) {
const promises = [];
for (const key of keyPrefixed) {
promises.push(Promise.resolve()
.then(() => store.get(key))
.then(data => (typeof data === 'string') ? this.opts.deserialize(data) : (this.opts.compression ? this.opts.deserialize(data) : data))
.then(data => {
if (data === undefined || data === null) {
return undefined;
}
if (typeof data.expires === 'number' && Date.now() > data.expires) {
return this.delete(key).then(() => undefined);
}
return (options && options.raw) ? data : data.value;
}),
);
}
return Promise.allSettled(promises)
.then(values => {
const data = [];
for (const value of values) {
data.push(value.value);
}
return data;
});
}
return Promise.resolve()
.then(() => isArray ? store.getMany(keyPrefixed) : store.get(keyPrefixed))
.then(data => (typeof data === 'string') ? this.opts.deserialize(data) : (this.opts.compression ? this.opts.deserialize(data) : data))
.then(data => {
if (data === undefined || data === null) {
return undefined;
}
if (isArray) {
return data.map((row, index) => {
if ((typeof row === 'string')) {
row = this.opts.deserialize(row);
}
if (row === undefined || row === null) {
return undefined;
}
if (typeof row.expires === 'number' && Date.now() > row.expires) {
this.delete(key[index]).then(() => undefined);
return undefined;
}
return (options && options.raw) ? row : row.value;
});
}
if (typeof data.expires === 'number' && Date.now() > data.expires) {
return this.delete(key).then(() => undefined);
}
return (options && options.raw) ? data : data.value;
});
}
set(key, value, ttl) {
const keyPrefixed = this._getKeyPrefix(key);
if (typeof ttl === 'undefined') {
ttl = this.opts.ttl;
}
if (ttl === 0) {
ttl = undefined;
}
const {store} = this.opts;
return Promise.resolve()
.then(() => {
const expires = (typeof ttl === 'number') ? (Date.now() + ttl) : null;
if (typeof value === 'symbol') {
this.emit('error', 'symbol cannot be serialized');
}
value = {value, expires};
return this.opts.serialize(value);
})
.then(value => store.set(keyPrefixed, value, ttl))
.then(() => true);
}
delete(key) {
const {store} = this.opts;
if (Array.isArray(key)) {
const keyPrefixed = this._getKeyPrefixArray(key);
if (store.deleteMany === undefined) {
const promises = [];
for (const key of keyPrefixed) {
promises.push(store.delete(key));
}
return Promise.allSettled(promises)
.then(values => values.every(x => x.value === true));
}
return Promise.resolve()
.then(() => store.deleteMany(keyPrefixed));
}
const keyPrefixed = this._getKeyPrefix(key);
return Promise.resolve()
.then(() => store.delete(keyPrefixed));
}
clear() {
const {store} = this.opts;
return Promise.resolve()
.then(() => store.clear());
}
has(key) {
const keyPrefixed = this._getKeyPrefix(key);
const {store} = this.opts;
return Promise.resolve()
.then(async () => {
if (typeof store.has === 'function') {
return store.has(keyPrefixed);
}
const value = await store.get(keyPrefixed);
return value !== undefined;
});
}
disconnect() {
const {store} = this.opts;
if (typeof store.disconnect === 'function') {
return store.disconnect();
}
}
}
module.exports = Keyv;

View File

@@ -0,0 +1,16 @@
/**
Lowercase the keys of an object.
@returns A new object with the keys lowercased.
@example
```
import lowercaseKeys = require('lowercase-keys');
lowercaseKeys({FOO: true, bAr: false});
//=> {foo: true, bar: false}
```
*/
declare function lowercaseKeys<T extends unknown>(object: {[key: string]: T}): {[key: string]: T};
export = lowercaseKeys;

View File

@@ -0,0 +1,10 @@
'use strict';
module.exports = object => {
const result = {};
for (const [key, value] of Object.entries(object)) {
result[key.toLowerCase()] = value;
}
return result;
};

View File

@@ -0,0 +1,9 @@
MIT License
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@@ -0,0 +1,41 @@
{
"name": "lowercase-keys",
"version": "2.0.0",
"description": "Lowercase the keys of an object",
"license": "MIT",
"repository": "sindresorhus/lowercase-keys",
"author": {
"name": "Sindre Sorhus",
"email": "sindresorhus@gmail.com",
"url": "sindresorhus.com"
},
"engines": {
"node": ">=8"
},
"scripts": {
"test": "xo && ava && tsd"
},
"files": [
"index.js",
"index.d.ts"
],
"keywords": [
"object",
"assign",
"extend",
"properties",
"lowercase",
"lower-case",
"case",
"keys",
"key"
],
"devDependencies": {
"ava": "^1.4.1",
"tsd": "^0.7.2",
"xo": "^0.24.0"
},
"__npminstall_done": true,
"_from": "lowercase-keys@2.0.0",
"_resolved": "https://registry.npmmirror.com/lowercase-keys/-/lowercase-keys-2.0.0.tgz"
}

View File

@@ -0,0 +1,32 @@
# lowercase-keys [![Build Status](https://travis-ci.org/sindresorhus/lowercase-keys.svg?branch=master)](https://travis-ci.org/sindresorhus/lowercase-keys)
> Lowercase the keys of an object
## Install
```
$ npm install lowercase-keys
```
## Usage
```js
const lowercaseKeys = require('lowercase-keys');
lowercaseKeys({FOO: true, bAr: false});
//=> {foo: true, bar: false}
```
## API
### lowercaseKeys(object)
Returns a new object with the keys lowercased.
## License
MIT © [Sindre Sorhus](https://sindresorhus.com)

View File

@@ -0,0 +1,267 @@
declare namespace normalizeUrl {
interface Options {
/**
@default 'http:'
*/
readonly defaultProtocol?: string;
/**
Prepends `defaultProtocol` to the URL if it's protocol-relative.
@default true
@example
```
normalizeUrl('//sindresorhus.com:80/');
//=> 'http://sindresorhus.com'
normalizeUrl('//sindresorhus.com:80/', {normalizeProtocol: false});
//=> '//sindresorhus.com'
```
*/
readonly normalizeProtocol?: boolean;
/**
Normalizes `https:` URLs to `http:`.
@default false
@example
```
normalizeUrl('https://sindresorhus.com:80/');
//=> 'https://sindresorhus.com'
normalizeUrl('https://sindresorhus.com:80/', {forceHttp: true});
//=> 'http://sindresorhus.com'
```
*/
readonly forceHttp?: boolean;
/**
Normalizes `http:` URLs to `https:`.
This option can't be used with the `forceHttp` option at the same time.
@default false
@example
```
normalizeUrl('https://sindresorhus.com:80/');
//=> 'https://sindresorhus.com'
normalizeUrl('http://sindresorhus.com:80/', {forceHttps: true});
//=> 'https://sindresorhus.com'
```
*/
readonly forceHttps?: boolean;
/**
Strip the [authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) part of a URL.
@default true
@example
```
normalizeUrl('user:password@sindresorhus.com');
//=> 'https://sindresorhus.com'
normalizeUrl('user:password@sindresorhus.com', {stripAuthentication: false});
//=> 'https://user:password@sindresorhus.com'
```
*/
readonly stripAuthentication?: boolean;
/**
Removes hash from the URL.
@default false
@example
```
normalizeUrl('sindresorhus.com/about.html#contact');
//=> 'http://sindresorhus.com/about.html#contact'
normalizeUrl('sindresorhus.com/about.html#contact', {stripHash: true});
//=> 'http://sindresorhus.com/about.html'
```
*/
readonly stripHash?: boolean;
/**
Removes HTTP(S) protocol from an URL `http://sindresorhus.com` → `sindresorhus.com`.
@default false
@example
```
normalizeUrl('https://sindresorhus.com');
//=> 'https://sindresorhus.com'
normalizeUrl('sindresorhus.com', {stripProtocol: true});
//=> 'sindresorhus.com'
```
*/
readonly stripProtocol?: boolean;
/**
Strip the [text fragment](https://web.dev/text-fragments/) part of the URL
__Note:__ The text fragment will always be removed if the `stripHash` option is set to `true`, as the hash contains the text fragment.
@default true
@example
```
normalizeUrl('http://sindresorhus.com/about.html#:~:text=hello');
//=> 'http://sindresorhus.com/about.html#'
normalizeUrl('http://sindresorhus.com/about.html#section:~:text=hello');
//=> 'http://sindresorhus.com/about.html#section'
normalizeUrl('http://sindresorhus.com/about.html#:~:text=hello', {stripTextFragment: false});
//=> 'http://sindresorhus.com/about.html#:~:text=hello'
normalizeUrl('http://sindresorhus.com/about.html#section:~:text=hello', {stripTextFragment: false});
//=> 'http://sindresorhus.com/about.html#section:~:text=hello'
```
*/
readonly stripTextFragment?: boolean;
/**
Removes `www.` from the URL.
@default true
@example
```
normalizeUrl('http://www.sindresorhus.com');
//=> 'http://sindresorhus.com'
normalizeUrl('http://www.sindresorhus.com', {stripWWW: false});
//=> 'http://www.sindresorhus.com'
```
*/
readonly stripWWW?: boolean;
/**
Removes query parameters that matches any of the provided strings or regexes.
@default [/^utm_\w+/i]
@example
```
normalizeUrl('www.sindresorhus.com?foo=bar&ref=test_ref', {
removeQueryParameters: ['ref']
});
//=> 'http://sindresorhus.com/?foo=bar'
```
If a boolean is provided, `true` will remove all the query parameters.
```
normalizeUrl('www.sindresorhus.com?foo=bar', {
removeQueryParameters: true
});
//=> 'http://sindresorhus.com'
```
`false` will not remove any query parameter.
```
normalizeUrl('www.sindresorhus.com?foo=bar&utm_medium=test&ref=test_ref', {
removeQueryParameters: false
});
//=> 'http://www.sindresorhus.com/?foo=bar&ref=test_ref&utm_medium=test'
```
*/
readonly removeQueryParameters?: ReadonlyArray<RegExp | string> | boolean;
/**
Removes trailing slash.
__Note__: Trailing slash is always removed if the URL doesn't have a pathname unless the `removeSingleSlash` option is set to `false`.
@default true
@example
```
normalizeUrl('http://sindresorhus.com/redirect/');
//=> 'http://sindresorhus.com/redirect'
normalizeUrl('http://sindresorhus.com/redirect/', {removeTrailingSlash: false});
//=> 'http://sindresorhus.com/redirect/'
normalizeUrl('http://sindresorhus.com/', {removeTrailingSlash: false});
//=> 'http://sindresorhus.com'
```
*/
readonly removeTrailingSlash?: boolean;
/**
Remove a sole `/` pathname in the output. This option is independant of `removeTrailingSlash`.
@default true
@example
```
normalizeUrl('https://sindresorhus.com/');
//=> 'https://sindresorhus.com'
normalizeUrl('https://sindresorhus.com/', {removeSingleSlash: false});
//=> 'https://sindresorhus.com/'
```
*/
readonly removeSingleSlash?: boolean;
/**
Removes the default directory index file from path that matches any of the provided strings or regexes.
When `true`, the regex `/^index\.[a-z]+$/` is used.
@default false
@example
```
normalizeUrl('www.sindresorhus.com/foo/default.php', {
removeDirectoryIndex: [/^default\.[a-z]+$/]
});
//=> 'http://sindresorhus.com/foo'
```
*/
readonly removeDirectoryIndex?: ReadonlyArray<RegExp | string>;
/**
Sorts the query parameters alphabetically by key.
@default true
@example
```
normalizeUrl('www.sindresorhus.com?b=two&a=one&c=three', {
sortQueryParameters: false
});
//=> 'http://sindresorhus.com/?b=two&a=one&c=three'
```
*/
readonly sortQueryParameters?: boolean;
}
}
/**
[Normalize](https://en.wikipedia.org/wiki/URL_normalization) a URL.
@param url - URL to normalize, including [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs).
@example
```
import normalizeUrl = require('normalize-url');
normalizeUrl('sindresorhus.com');
//=> 'http://sindresorhus.com'
normalizeUrl('//www.sindresorhus.com:80/../baz?b=bar&a=foo');
//=> 'http://sindresorhus.com/baz?a=foo&b=bar'
```
*/
declare function normalizeUrl(url: string, options?: normalizeUrl.Options): string;
export = normalizeUrl;

View File

@@ -0,0 +1,216 @@
'use strict';
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs
const DATA_URL_DEFAULT_MIME_TYPE = 'text/plain';
const DATA_URL_DEFAULT_CHARSET = 'us-ascii';
const testParameter = (name, filters) => {
return filters.some(filter => filter instanceof RegExp ? filter.test(name) : filter === name);
};
const normalizeDataURL = (urlString, {stripHash}) => {
const match = /^data:(?<type>[^,]*?),(?<data>[^#]*?)(?:#(?<hash>.*))?$/.exec(urlString);
if (!match) {
throw new Error(`Invalid URL: ${urlString}`);
}
let {type, data, hash} = match.groups;
const mediaType = type.split(';');
hash = stripHash ? '' : hash;
let isBase64 = false;
if (mediaType[mediaType.length - 1] === 'base64') {
mediaType.pop();
isBase64 = true;
}
// Lowercase MIME type
const mimeType = (mediaType.shift() || '').toLowerCase();
const attributes = mediaType
.map(attribute => {
let [key, value = ''] = attribute.split('=').map(string => string.trim());
// Lowercase `charset`
if (key === 'charset') {
value = value.toLowerCase();
if (value === DATA_URL_DEFAULT_CHARSET) {
return '';
}
}
return `${key}${value ? `=${value}` : ''}`;
})
.filter(Boolean);
const normalizedMediaType = [
...attributes
];
if (isBase64) {
normalizedMediaType.push('base64');
}
if (normalizedMediaType.length !== 0 || (mimeType && mimeType !== DATA_URL_DEFAULT_MIME_TYPE)) {
normalizedMediaType.unshift(mimeType);
}
return `data:${normalizedMediaType.join(';')},${isBase64 ? data.trim() : data}${hash ? `#${hash}` : ''}`;
};
const normalizeUrl = (urlString, options) => {
options = {
defaultProtocol: 'http:',
normalizeProtocol: true,
forceHttp: false,
forceHttps: false,
stripAuthentication: true,
stripHash: false,
stripTextFragment: true,
stripWWW: true,
removeQueryParameters: [/^utm_\w+/i],
removeTrailingSlash: true,
removeSingleSlash: true,
removeDirectoryIndex: false,
sortQueryParameters: true,
...options
};
urlString = urlString.trim();
// Data URL
if (/^data:/i.test(urlString)) {
return normalizeDataURL(urlString, options);
}
if (/^view-source:/i.test(urlString)) {
throw new Error('`view-source:` is not supported as it is a non-standard protocol');
}
const hasRelativeProtocol = urlString.startsWith('//');
const isRelativeUrl = !hasRelativeProtocol && /^\.*\//.test(urlString);
// Prepend protocol
if (!isRelativeUrl) {
urlString = urlString.replace(/^(?!(?:\w+:)?\/\/)|^\/\//, options.defaultProtocol);
}
const urlObj = new URL(urlString);
if (options.forceHttp && options.forceHttps) {
throw new Error('The `forceHttp` and `forceHttps` options cannot be used together');
}
if (options.forceHttp && urlObj.protocol === 'https:') {
urlObj.protocol = 'http:';
}
if (options.forceHttps && urlObj.protocol === 'http:') {
urlObj.protocol = 'https:';
}
// Remove auth
if (options.stripAuthentication) {
urlObj.username = '';
urlObj.password = '';
}
// Remove hash
if (options.stripHash) {
urlObj.hash = '';
} else if (options.stripTextFragment) {
urlObj.hash = urlObj.hash.replace(/#?:~:text.*?$/i, '');
}
// Remove duplicate slashes if not preceded by a protocol
if (urlObj.pathname) {
urlObj.pathname = urlObj.pathname.replace(/(?<!\b(?:[a-z][a-z\d+\-.]{1,50}:))\/{2,}/g, '/');
}
// Decode URI octets
if (urlObj.pathname) {
try {
urlObj.pathname = decodeURI(urlObj.pathname);
} catch (_) {}
}
// Remove directory index
if (options.removeDirectoryIndex === true) {
options.removeDirectoryIndex = [/^index\.[a-z]+$/];
}
if (Array.isArray(options.removeDirectoryIndex) && options.removeDirectoryIndex.length > 0) {
let pathComponents = urlObj.pathname.split('/');
const lastComponent = pathComponents[pathComponents.length - 1];
if (testParameter(lastComponent, options.removeDirectoryIndex)) {
pathComponents = pathComponents.slice(0, pathComponents.length - 1);
urlObj.pathname = pathComponents.slice(1).join('/') + '/';
}
}
if (urlObj.hostname) {
// Remove trailing dot
urlObj.hostname = urlObj.hostname.replace(/\.$/, '');
// Remove `www.`
if (options.stripWWW && /^www\.(?!www\.)(?:[a-z\-\d]{1,63})\.(?:[a-z.\-\d]{2,63})$/.test(urlObj.hostname)) {
// Each label should be max 63 at length (min: 1).
// Source: https://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names
// Each TLD should be up to 63 characters long (min: 2).
// It is technically possible to have a single character TLD, but none currently exist.
urlObj.hostname = urlObj.hostname.replace(/^www\./, '');
}
}
// Remove query unwanted parameters
if (Array.isArray(options.removeQueryParameters)) {
for (const key of [...urlObj.searchParams.keys()]) {
if (testParameter(key, options.removeQueryParameters)) {
urlObj.searchParams.delete(key);
}
}
}
if (options.removeQueryParameters === true) {
urlObj.search = '';
}
// Sort query parameters
if (options.sortQueryParameters) {
urlObj.searchParams.sort();
}
if (options.removeTrailingSlash) {
urlObj.pathname = urlObj.pathname.replace(/\/$/, '');
}
const oldUrlString = urlString;
// Take advantage of many of the Node `url` normalizations
urlString = urlObj.toString();
if (!options.removeSingleSlash && urlObj.pathname === '/' && !oldUrlString.endsWith('/') && urlObj.hash === '') {
urlString = urlString.replace(/\/$/, '');
}
// Remove ending `/` unless removeSingleSlash is false
if ((options.removeTrailingSlash || urlObj.pathname === '/') && urlObj.hash === '' && options.removeSingleSlash) {
urlString = urlString.replace(/\/$/, '');
}
// Restore relative protocol, if applicable
if (hasRelativeProtocol && !options.normalizeProtocol) {
urlString = urlString.replace(/^http:\/\//, '//');
}
// Remove http/https
if (options.stripProtocol) {
urlString = urlString.replace(/^(?:https?:)?\/\//, '');
}
return urlString;
};
module.exports = normalizeUrl;

View File

@@ -0,0 +1,9 @@
MIT License
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@@ -0,0 +1,53 @@
{
"name": "normalize-url",
"version": "6.1.0",
"description": "Normalize a URL",
"license": "MIT",
"repository": "sindresorhus/normalize-url",
"funding": "https://github.com/sponsors/sindresorhus",
"author": {
"name": "Sindre Sorhus",
"email": "sindresorhus@gmail.com",
"url": "https://sindresorhus.com"
},
"engines": {
"node": ">=10"
},
"scripts": {
"test": "xo && nyc ava && tsd"
},
"files": [
"index.js",
"index.d.ts"
],
"keywords": [
"normalize",
"url",
"uri",
"address",
"string",
"normalization",
"normalisation",
"query",
"querystring",
"simplify",
"strip",
"trim",
"canonical"
],
"devDependencies": {
"ava": "^2.4.0",
"nyc": "^15.0.0",
"tsd": "^0.11.0",
"xo": "^0.25.3"
},
"nyc": {
"reporter": [
"text",
"lcov"
]
},
"__npminstall_done": true,
"_from": "normalize-url@6.1.0",
"_resolved": "https://registry.npmmirror.com/normalize-url/-/normalize-url-6.1.0.tgz"
}

View File

@@ -0,0 +1,286 @@
# normalize-url [![Coverage Status](https://codecov.io/gh/sindresorhus/normalize-url/branch/main/graph/badge.svg)](https://codecov.io/gh/sindresorhus/normalize-url)
> [Normalize](https://en.wikipedia.org/wiki/URL_normalization) a URL
Useful when you need to display, store, deduplicate, sort, compare, etc, URLs.
## Install
```
$ npm install normalize-url
```
*If you need to use this in the browser, use version 4: `npm i normalize-url@4`*
## Usage
```js
const normalizeUrl = require('normalize-url');
normalizeUrl('sindresorhus.com');
//=> 'http://sindresorhus.com'
normalizeUrl('//www.sindresorhus.com:80/../baz?b=bar&a=foo');
//=> 'http://sindresorhus.com/baz?a=foo&b=bar'
```
## API
### normalizeUrl(url, options?)
#### url
Type: `string`
URL to normalize, including [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs).
#### options
Type: `object`
##### defaultProtocol
Type: `string`\
Default: `http:`
##### normalizeProtocol
Type: `boolean`\
Default: `true`
Prepend `defaultProtocol` to the URL if it's protocol-relative.
```js
normalizeUrl('//sindresorhus.com:80/');
//=> 'http://sindresorhus.com'
normalizeUrl('//sindresorhus.com:80/', {normalizeProtocol: false});
//=> '//sindresorhus.com'
```
##### forceHttp
Type: `boolean`\
Default: `false`
Normalize `https:` to `http:`.
```js
normalizeUrl('https://sindresorhus.com:80/');
//=> 'https://sindresorhus.com'
normalizeUrl('https://sindresorhus.com:80/', {forceHttp: true});
//=> 'http://sindresorhus.com'
```
##### forceHttps
Type: `boolean`\
Default: `false`
Normalize `http:` to `https:`.
```js
normalizeUrl('https://sindresorhus.com:80/');
//=> 'https://sindresorhus.com'
normalizeUrl('http://sindresorhus.com:80/', {forceHttps: true});
//=> 'https://sindresorhus.com'
```
This option can't be used with the `forceHttp` option at the same time.
##### stripAuthentication
Type: `boolean`\
Default: `true`
Strip the [authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) part of the URL.
```js
normalizeUrl('user:password@sindresorhus.com');
//=> 'https://sindresorhus.com'
normalizeUrl('user:password@sindresorhus.com', {stripAuthentication: false});
//=> 'https://user:password@sindresorhus.com'
```
##### stripHash
Type: `boolean`\
Default: `false`
Strip the hash part of the URL.
```js
normalizeUrl('sindresorhus.com/about.html#contact');
//=> 'http://sindresorhus.com/about.html#contact'
normalizeUrl('sindresorhus.com/about.html#contact', {stripHash: true});
//=> 'http://sindresorhus.com/about.html'
```
##### stripProtocol
Type: `boolean`\
Default: `false`
Remove HTTP(S) protocol from the URL: `http://sindresorhus.com``sindresorhus.com`.
```js
normalizeUrl('https://sindresorhus.com');
//=> 'https://sindresorhus.com'
normalizeUrl('https://sindresorhus.com', {stripProtocol: true});
//=> 'sindresorhus.com'
```
##### stripTextFragment
Type: `boolean`\
Default: `true`
Strip the [text fragment](https://web.dev/text-fragments/) part of the URL.
**Note:** The text fragment will always be removed if the `stripHash` option is set to `true`, as the hash contains the text fragment.
```js
normalizeUrl('http://sindresorhus.com/about.html#:~:text=hello');
//=> 'http://sindresorhus.com/about.html#'
normalizeUrl('http://sindresorhus.com/about.html#section:~:text=hello');
//=> 'http://sindresorhus.com/about.html#section'
normalizeUrl('http://sindresorhus.com/about.html#:~:text=hello', {stripTextFragment: false});
//=> 'http://sindresorhus.com/about.html#:~:text=hello'
normalizeUrl('http://sindresorhus.com/about.html#section:~:text=hello', {stripTextFragment: false});
//=> 'http://sindresorhus.com/about.html#section:~:text=hello'
```
##### stripWWW
Type: `boolean`\
Default: `true`
Remove `www.` from the URL.
```js
normalizeUrl('http://www.sindresorhus.com');
//=> 'http://sindresorhus.com'
normalizeUrl('http://www.sindresorhus.com', {stripWWW: false});
//=> 'http://www.sindresorhus.com'
```
##### removeQueryParameters
Type: `Array<RegExp | string> | boolean`\
Default: `[/^utm_\w+/i]`
Remove query parameters that matches any of the provided strings or regexes.
```js
normalizeUrl('www.sindresorhus.com?foo=bar&ref=test_ref', {
removeQueryParameters: ['ref']
});
//=> 'http://sindresorhus.com/?foo=bar'
```
If a boolean is provided, `true` will remove all the query parameters.
```js
normalizeUrl('www.sindresorhus.com?foo=bar', {
removeQueryParameters: true
});
//=> 'http://sindresorhus.com'
```
`false` will not remove any query parameter.
```js
normalizeUrl('www.sindresorhus.com?foo=bar&utm_medium=test&ref=test_ref', {
removeQueryParameters: false
});
//=> 'http://www.sindresorhus.com/?foo=bar&ref=test_ref&utm_medium=test'
```
##### removeTrailingSlash
Type: `boolean`\
Default: `true`
Remove trailing slash.
**Note:** Trailing slash is always removed if the URL doesn't have a pathname unless the `removeSingleSlash` option is set to `false`.
```js
normalizeUrl('http://sindresorhus.com/redirect/');
//=> 'http://sindresorhus.com/redirect'
normalizeUrl('http://sindresorhus.com/redirect/', {removeTrailingSlash: false});
//=> 'http://sindresorhus.com/redirect/'
normalizeUrl('http://sindresorhus.com/', {removeTrailingSlash: false});
//=> 'http://sindresorhus.com'
```
##### removeSingleSlash
Type: `boolean`\
Default: `true`
Remove a sole `/` pathname in the output. This option is independant of `removeTrailingSlash`.
```js
normalizeUrl('https://sindresorhus.com/');
//=> 'https://sindresorhus.com'
normalizeUrl('https://sindresorhus.com/', {removeSingleSlash: false});
//=> 'https://sindresorhus.com/'
```
##### removeDirectoryIndex
Type: `boolean | Array<RegExp | string>`\
Default: `false`
Removes the default directory index file from path that matches any of the provided strings or regexes. When `true`, the regex `/^index\.[a-z]+$/` is used.
```js
normalizeUrl('www.sindresorhus.com/foo/default.php', {
removeDirectoryIndex: [/^default\.[a-z]+$/]
});
//=> 'http://sindresorhus.com/foo'
```
##### sortQueryParameters
Type: `boolean`\
Default: `true`
Sorts the query parameters alphabetically by key.
```js
normalizeUrl('www.sindresorhus.com?b=two&a=one&c=three', {
sortQueryParameters: false
});
//=> 'http://sindresorhus.com/?b=two&a=one&c=three'
```
## Related
- [compare-urls](https://github.com/sindresorhus/compare-urls) - Compare URLs by first normalizing them
---
<div align="center">
<b>
<a href="https://tidelift.com/subscription/pkg/npm-normalize-url?utm_source=npm-normalize-url&utm_medium=referral&utm_campaign=readme">Get professional support for this package with a Tidelift subscription</a>
</b>
<br>
<sub>
Tidelift helps make open source sustainable for maintainers while giving companies<br>assurances about security, maintenance, and licensing for their dependencies.
</sub>
</div>

View File

@@ -0,0 +1,19 @@
Copyright (c) 2017 Luke Childs
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

View File

@@ -0,0 +1,77 @@
# responselike
> A response-like object for mocking a Node.js HTTP response stream
[![Build Status](https://travis-ci.org/lukechilds/responselike.svg?branch=master)](https://travis-ci.org/lukechilds/responselike)
[![Coverage Status](https://coveralls.io/repos/github/lukechilds/responselike/badge.svg?branch=master)](https://coveralls.io/github/lukechilds/responselike?branch=master)
[![npm](https://img.shields.io/npm/dm/responselike.svg)](https://www.npmjs.com/package/responselike)
[![npm](https://img.shields.io/npm/v/responselike.svg)](https://www.npmjs.com/package/responselike)
Returns a streamable response object similar to a [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage). Useful for formatting cached responses so they can be consumed by code expecting a real response.
## Install
```shell
npm install --save responselike
```
Or if you're just using for testing you'll want:
```shell
npm install --save-dev responselike
```
## Usage
```js
const Response = require('responselike');
const response = new Response(200, { foo: 'bar' }, Buffer.from('Hi!'), 'https://example.com');
response.statusCode;
// 200
response.headers;
// { foo: 'bar' }
response.body;
// <Buffer 48 69 21>
response.url;
// 'https://example.com'
response.pipe(process.stdout);
// Hi!
```
## API
### new Response(statusCode, headers, body, url)
Returns a streamable response object similar to a [Node.js HTTP response stream](https://nodejs.org/api/http.html#http_class_http_incomingmessage).
#### statusCode
Type: `number`
HTTP response status code.
#### headers
Type: `object`
HTTP headers object. Keys will be automatically lowercased.
#### body
Type: `buffer`
A Buffer containing the response body. The Buffer contents will be streamable but is also exposed directly as `response.body`.
#### url
Type: `string`
Request URL string.
## License
MIT © Luke Childs

View File

@@ -0,0 +1,42 @@
{
"name": "responselike",
"version": "2.0.1",
"description": "A response-like object for mocking a Node.js HTTP response stream",
"funding": "https://github.com/sponsors/sindresorhus",
"main": "src/index.js",
"scripts": {
"test": "xo && nyc ava",
"coverage": "nyc report --reporter=text-lcov | coveralls"
},
"xo": {
"extends": "xo-lukechilds"
},
"keywords": [
"http",
"https",
"response",
"mock",
"request",
"responselike"
],
"repository": {
"type": "git",
"url": "https://github.com/sindresorhus/responselike.git"
},
"author": "lukechilds",
"license": "MIT",
"devDependencies": {
"ava": "^0.25.0",
"coveralls": "^3.0.0",
"eslint-config-xo-lukechilds": "^1.0.0",
"get-stream": "^3.0.0",
"nyc": "^11.8.0",
"xo": "^0.19.0"
},
"dependencies": {
"lowercase-keys": "^2.0.0"
},
"__npminstall_done": true,
"_from": "responselike@2.0.1",
"_resolved": "https://registry.npmmirror.com/responselike/-/responselike-2.0.1.tgz"
}

View File

@@ -0,0 +1,34 @@
'use strict';
const Readable = require('stream').Readable;
const lowercaseKeys = require('lowercase-keys');
class Response extends Readable {
constructor(statusCode, headers, body, url) {
if (typeof statusCode !== 'number') {
throw new TypeError('Argument `statusCode` should be a number');
}
if (typeof headers !== 'object') {
throw new TypeError('Argument `headers` should be an object');
}
if (!(body instanceof Buffer)) {
throw new TypeError('Argument `body` should be a buffer');
}
if (typeof url !== 'string') {
throw new TypeError('Argument `url` should be a string');
}
super();
this.statusCode = statusCode;
this.headers = lowercaseKeys(headers);
this.body = body;
this.url = url;
}
_read() {
this.push(this.body);
this.push(null);
}
}
module.exports = Response;